http://arxiv.org/abs/2509.24359
DRIFT: Divergent Response in Filtered Transformations for Robust Adversarial Defense. (99%)
Amira Guesmi; Muhammad Shafique
Deep neural networks remain highly vulnerable to adversarial examples, and most defenses collapse once gradients can be reliably estimated. We identify \emph{gradient consensus} -- the tendency of randomized transformations to yield aligned gradients -- as a key driver of adversarial transferability. Attackers exploit this consensus to construct perturbations that remain effective across transformations. We introduce \textbf{DRIFT} (Divergent Response in Filtered Transformations), a stochastic ensemble of lightweight, learnable filters trained to actively disrupt gradient consensus. Unlike prior randomized defenses that rely on gradient masking, DRIFT enforces \emph{gradient dissonance} by maximizing divergence in Jacobian- and logit-space responses while preserving natural predictions. Our contributions are threefold: (i) we formalize gradient consensus and provide a theoretical analysis linking consensus to transferability; (ii) we propose a consensus-divergence training strategy combining prediction consistency, Jacobian separation, logit-space separation, and adversarial robustness; and (iii) we show that DRIFT achieves substantial robustness gains on ImageNet across CNNs and Vision Transformers, outperforming state-of-the-art preprocessing, adversarial training, and diffusion-based defenses under adaptive white-box, transfer-based, and gradient-free attacks. DRIFT delivers these improvements with negligible runtime and memory cost, establishing gradient divergence as a practical and generalizable principle for adversarial defense.

http://arxiv.org/abs/2410.05694
DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing. (98%)
June Suk Choi; Kyungmin Lee; Jongheon Jeong; Saining Xie; Jinwoo Shin; Kimin Lee
Recent advances in diffusion models have introduced a new era of text-guided image manipulation, enabling users to create realistic edited images with simple textual prompts. However, there is significant concern about the potential misuse of these methods, especially in creating misleading or harmful content. Although recent defense strategies, which introduce imperceptible adversarial noise to induce model failure, have shown promise, they remain ineffective against more sophisticated manipulations, such as editing with a mask. In this work, we propose DiffusionGuard, a robust and effective defense method against unauthorized edits by diffusion-based image editing models, even in challenging setups. Through a detailed analysis of these models, we introduce a novel objective that generates adversarial noise targeting the early stage of the diffusion process. This approach significantly improves the efficiency and effectiveness of adversarial noises. We also introduce a mask-augmentation technique to enhance robustness against various masks during test time. Finally, we introduce a comprehensive benchmark designed to evaluate the effectiveness and robustness of methods in protecting against privacy threats in realistic scenarios. Through extensive experiments, we show that our method achieves stronger protection and improved mask robustness with lower computational costs compared to the strongest baseline. Additionally, our method exhibits superior transferability and better resilience to noise removal techniques compared to all baseline methods. Our source code is publicly available at https://github.com/choi403/DiffusionGuard.

http://arxiv.org/abs/2509.19994
Improving Generalizability and Undetectability for Targeted Adversarial Attacks on Multimodal Pre-trained Models. (98%)
Zhifang Zhang; Jiahan Zhang; Shengjie Zhou; Qi Wei; Shuo He; Feng Liu; Lei Feng
Multimodal pre-trained models (e.g., ImageBind), which align distinct data modalities into a shared embedding space, have shown remarkable success across downstream tasks. However, their increasing adoption raises serious security concerns, especially regarding targeted adversarial attacks. In this paper, we show that existing targeted adversarial attacks on multimodal pre-trained models still have limitations in two aspects: generalizability and undetectability. Specifically, the crafted targeted adversarial examples (AEs) exhibit limited generalization to partially known or semantically similar targets in cross-modal alignment tasks (i.e., limited generalizability) and can be easily detected by simple anomaly detection methods (i.e., limited undetectability). To address these limitations, we propose a novel method called Proxy Targeted Attack (PTA), which leverages multiple source-modal and target-modal proxies to optimize targeted AEs, ensuring they remain evasive to defenses while aligning with multiple potential targets. We also provide theoretical analyses to highlight the relationship between generalizability and undetectability and to ensure optimal generalizability while meeting the specified requirements for undetectability. Furthermore, experimental results demonstrate that our PTA can achieve a high success rate across various related targets and remain undetectable against multiple anomaly detection methods.

http://arxiv.org/abs/2507.09993
3DGAA: Realistic and Robust 3D Gaussian-based Adversarial Attack for Autonomous Driving. (98%)
Yixun Zhang; Lizhi Wang; Junjun Zhao; Wending Zhao; Feng Zhou; Yonghao Dang; Jianqin Yin
Camera-based object detection systems play a vital role in autonomous driving, yet they remain vulnerable to adversarial threats in real-world environments. Existing 2D and 3D physical attacks, due to their focus on texture optimization, often struggle to balance physical realism and attack robustness. In this work, we propose 3D Gaussian-based Adversarial Attack (3DGAA), a novel adversarial object generation framework that leverages the full 14-dimensional parameterization of 3D Gaussian Splatting (3DGS) to jointly optimize geometry and appearance in physically realizable ways. Unlike prior works that rely on patches or texture optimization, 3DGAA jointly perturbs both geometric attributes (shape, scale, rotation) and appearance attributes (color, opacity) to produce physically realistic and transferable adversarial objects. We further introduce a physical filtering module that filters outliers to preserve geometric fidelity, and a physical augmentation module that simulates complex physical scenarios to enhance attack generalization under real-world conditions. We evaluate 3DGAA on both virtual benchmarks and physical-world setups using miniature vehicle models. Experimental results show that 3DGAA achieves to reduce the detection mAP from 87.21\% to 7.38\%, significantly outperforming existing 3D physical attacks. Moreover, our method maintains high transferability across different physical conditions, demonstrating a new state-of-the-art in physically realizable adversarial attacks.

http://arxiv.org/abs/2509.24891
VAGUEGAN: Stealthy Poisoning and Backdoor Attacks on Image Generative Pipelines. (81%)
Mostafa Mohaimen Akand Faisal; Rabeya Amin Jhuma
Generative models such as GANs and diffusion models are widely used to synthesize photorealistic images and to support downstream creative and editing tasks. While adversarial attacks on discriminative models are well studied, attacks targeting generative pipelines where small, stealthy perturbations in inputs lead to controlled changes in outputs are less explored. This study introduces VagueGAN, an attack pipeline combining a modular perturbation network PoisonerNet with a Generator Discriminator pair to craft stealthy triggers that cause targeted changes in generated images. Attack efficacy is evaluated using a custom proxy metric, while stealth is analyzed through perceptual and frequency domain measures. The transferability of the method to a modern diffusion based pipeline is further examined through ControlNet guided editing. Interestingly, the experiments show that poisoned outputs can display higher visual quality compared to clean counterparts, challenging the assumption that poisoning necessarily reduces fidelity. Unlike conventional pixel level perturbations, latent space poisoning in GANs and diffusion pipelines can retain or even enhance output aesthetics, exposing a blind spot in pixel level defenses. Moreover, carefully optimized perturbations can produce consistent, stealthy effects on generator outputs while remaining visually inconspicuous, raising concerns for the integrity of image generation pipelines.

http://arxiv.org/abs/2509.25082
MANI-Pure: Magnitude-Adaptive Noise Injection for Adversarial Purification. (74%)
Xiaoyi Huang; Junwei Wu; Kejia Zhang; Carl Yang; Zhiming Luo
Adversarial purification with diffusion models has emerged as a promising defense strategy, but existing methods typically rely on uniform noise injection, which indiscriminately perturbs all frequencies, corrupting semantic structures and undermining robustness. Our empirical study reveals that adversarial perturbations are not uniformly distributed: they are predominantly concentrated in high-frequency regions, with heterogeneous magnitude intensity patterns that vary across frequencies and attack types. Motivated by this observation, we introduce MANI-Pure, a magnitude-adaptive purification framework that leverages the magnitude spectrum of inputs to guide the purification process. Instead of injecting homogeneous noise, MANI-Pure adaptively applies heterogeneous, frequency-targeted noise, effectively suppressing adversarial perturbations in fragile high-frequency, low-magnitude bands while preserving semantically critical low-frequency content. Extensive experiments on CIFAR-10 and ImageNet-1K validate the effectiveness of MANI-Pure. It narrows the clean accuracy gap to within 0.59 of the original classifier, while boosting robust accuracy by 2.15, and achieves the top-1 robust accuracy on the RobustBench leaderboard, surpassing the previous state-of-the-art method.

http://arxiv.org/abs/2509.20463
Efficiently Attacking Memorization Scores. (70%)
Tue Do; Varun Chandrasekaran; Daniel Alabi
Influence estimation tools -- such as memorization scores -- are widely used to understand model behavior, attribute training data, and inform dataset curation. However, recent applications in data valuation and responsible machine learning raise the question: can these scores themselves be adversarially manipulated? In this work, we present a systematic study of the feasibility of attacking memorization-based influence estimators. We characterize attacks for producing highly memorized samples as highly sensitive queries in the regime where a trained algorithm is accurate. Our attack (calculating the pseudoinverse of the input) is practical, requiring only black-box access to model outputs and incur modest computational overhead. We empirically validate our attack across a wide suite of image classification tasks, showing that even state-of-the-art proxies are vulnerable to targeted score manipulations. In addition, we provide a theoretical analysis of the stability of memorization scores under adversarial perturbations, revealing conditions under which influence estimates are inherently fragile. Our findings highlight critical vulnerabilities in influence-based attribution and suggest the need for robust defenses. All code can be found at https://github.com/tuedo2/MemAttack

http://arxiv.org/abs/2509.24662
Community detection robustness of graph neural networks. (61%)
Jaidev Goel; Pablo Moriano; Ramakrishnan Kannan; Yulia R. Gel
Graph neural networks (GNNs) are increasingly widely used for community detection in attributed networks. They combine structural topology with node attributes through message passing and pooling. However, their robustness or lack of thereof with respect to different perturbations and targeted attacks in conjunction with community detection tasks is not well understood. To shed light into latent mechanisms behind GNN sensitivity on community detection tasks, we conduct a systematic computational evaluation of six widely adopted GNN architectures: GCN, GAT, Graph-SAGE, DiffPool, MinCUT, and DMoN. The analysis covers three perturbation categories: node attribute manipulations, edge topology distortions, and adversarial attacks. We use element-centric similarity as the evaluation metric on synthetic benchmarks and real-world citation networks. Our findings indicate that supervised GNNs tend to achieve higher baseline accuracy, while unsupervised methods, particularly DMoN, maintain stronger resilience under targeted and adversarial perturbations. Furthermore, robustness appears to be strongly influenced by community strength, with well-defined communities reducing performance loss. Across all models, node attribute perturbations associated with targeted edge deletions and shift in attribute distributions tend to cause the largest degradation in community recovery. These findings highlight important trade-offs between accuracy and robustness in GNN-based community detection and offer new insights into selecting architectures resilient to noise and adversarial attacks.

http://arxiv.org/abs/2509.24492
Guided Uncertainty Learning Using a Post-Hoc Evidential Meta-Model. (50%)
Charmaine Barker; Daniel Bethell; Simos Gerasimou
Reliable uncertainty quantification remains a major obstacle to the deployment of deep learning models under distributional shift. Existing post-hoc approaches that retrofit pretrained models either inherit misplaced confidence or merely reshape predictions, without teaching the model when to be uncertain. We introduce GUIDE, a lightweight evidential learning meta-model approach that attaches to a frozen deep learning model and explicitly learns how and when to be uncertain. GUIDE identifies salient internal features via a calibration stage, and then employs these features to construct a noise-driven curriculum that teaches the model how and when to express uncertainty. GUIDE requires no retraining, no architectural modifications, and no manual intermediate-layer selection to the base deep learning model, thus ensuring broad applicability and minimal user intervention. The resulting model avoids distilling overconfidence from the base model, improves out-of-distribution detection by ~77% and adversarial attack detection by ~80%, while preserving in-distribution performance. Across diverse benchmarks, GUIDE consistently outperforms state-of-the-art approaches, evidencing the need for actively guiding uncertainty to close the gap between predictive confidence and reliability.

http://arxiv.org/abs/2509.24566
TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models. (33%)
Zhifang Zhang; Qiqi Tao; Jiaqi Lv; Na Zhao; Lei Feng; Joey Tianyi Zhou
Large vision-language models (LVLMs) have achieved impressive performance across a wide range of vision-language tasks, while they remain vulnerable to backdoor attacks. Existing backdoor attacks on LVLMs aim to force the victim model to generate a predefined target pattern, which is either inserted into or replaces the original content. We find that these fixed-pattern attacks are relatively easy to detect, because the attacked LVLM tends to memorize such frequent patterns in the training dataset, thereby exhibiting overconfidence on these targets given poisoned inputs. To address these limitations, we introduce TokenSwap, a more evasive and stealthy backdoor attack that focuses on the compositional understanding capabilities of LVLMs. Instead of enforcing a fixed targeted content, TokenSwap subtly disrupts the understanding of object relationships in text. Specifically, it causes the backdoored model to generate outputs that mention the correct objects in the image but misrepresent their relationships (i.e., bags-of-words behavior). During training, TokenSwap injects a visual trigger into selected samples and simultaneously swaps the grammatical roles of key tokens in the corresponding textual answers. However, the poisoned samples exhibit only subtle differences from the original ones, making it challenging for the model to learn the backdoor behavior. To address this, TokenSwap employs an adaptive token-weighted loss that explicitly emphasizes the learning of swapped tokens, such that the visual triggers and bags-of-words behavior are associated. Extensive experiments demonstrate that TokenSwap achieves high attack success rates while maintaining superior evasiveness and stealthiness across multiple benchmarks and various LVLM architectures.

http://arxiv.org/abs/2508.02995
The Geometry of Cortical Computation: Manifold Disentanglement and Predictive Dynamics in VCNet. (26%)
Brennen A. Hill; Zhang Xinyu; Timothy Putra Prasetio
Despite their success, modern convolutional neural networks (CNNs) exhibit fundamental limitations, including data inefficiency, poor out-of-distribution generalization, and vulnerability to adversarial perturbations. These shortcomings can be traced to a lack of inductive biases that reflect the inherent geometric structure of the visual world. The primate visual system, in contrast, demonstrates superior efficiency and robustness, suggesting that its architectural and computational principles,which evolved to internalize these structures,may offer a blueprint for more capable artificial vision. This paper introduces Visual Cortex Network (VCNet), a novel neural network architecture whose design is informed by the macro-scale organization of the primate visual cortex. VCNet is framed as a geometric framework that emulates key biological mechanisms, including hierarchical processing across distinct cortical areas, dual-stream information segregation for learning disentangled representations, and top-down predictive feedback for representation refinement. We interpret these mechanisms through the lens of geometry and dynamical systems, positing that they guide the learning of structured, low-dimensional neural manifolds. We evaluate VCNet on two specialized benchmarks: the Spots-10 animal pattern dataset, which probes sensitivity to natural textures, and a light field image classification task, which requires processing higher-dimensional visual data. Our results show that VCNet achieves state-of-the-art accuracy of 92.1\% on Spots-10 and 74.4\% on the light field dataset, surpassing contemporary models of comparable size. This work demonstrates that integrating high-level neuroscientific principles, viewed through a geometric lens, can lead to more efficient and robust models, providing a promising direction for addressing long-standing challenges in machine learning.

http://arxiv.org/abs/2505.16670
BitHydra: Towards Bit-flip Inference Cost Attack against Large Language Models. (16%)
Xiaobei Yan; Yiming Li; Hao Wang; Han Qiu; Tianwei Zhang
Large language models (LLMs) are widely deployed, but their growing compute demands expose them to inference cost attacks that maximize output length. We reveal that prior attacks are fundamentally self-targeting because they rely on crafted inputs, so the added cost accrues to the attacker's own queries and scales poorly in practice. In this work, we introduce the first bit-flip inference cost attack that directly modifies model weights to induce persistent overhead for all users of a compromised LLM. Such attacks are stealthy yet realistic in practice: for instance, in shared MLaaS environments, co-located tenants can exploit hardware-level faults (e.g., Rowhammer) to flip memory bits storing model parameters. We instantiate this attack paradigm with BitHydra, which (1) minimizes a loss that suppresses the end-of-sequence token (i.e., EOS) and (2) employs an efficient yet effective critical-bit search focused on the EOS embedding vector, sharply reducing the search space while preserving benign-looking outputs. We evaluate across 11 LLMs (1.5B-14B) under int8 and float16, demonstrating that our method efficiently achieves scalable cost inflation with only a few bit flips, while remaining effective even against potential defenses.

http://arxiv.org/abs/2509.24215
Metamorphic Testing for Audio Content Moderation Software. (11%)
Wenxuan Wang; Yongjiang Wu; Junyuan Zhang; Shuqing Li; Yun Peng; Wenting Chen; Shuai Wang; Michael R. Lyu
The rapid growth of audio-centric platforms and applications such as WhatsApp and Twitter has transformed the way people communicate and share audio content in modern society. However, these platforms are increasingly misused to disseminate harmful audio content, such as hate speech, deceptive advertisements, and explicit material, which can have significant negative consequences (e.g., detrimental effects on mental health). In response, researchers and practitioners have been actively developing and deploying audio content moderation tools to tackle this issue. Despite these efforts, malicious actors can bypass moderation systems by making subtle alterations to audio content, such as modifying pitch or inserting noise. Moreover, the effectiveness of modern audio moderation tools against such adversarial inputs remains insufficiently studied. To address these challenges, we propose MTAM, a Metamorphic Testing framework for Audio content Moderation software. Specifically, we conduct a pilot study on 2000 audio clips and define 14 metamorphic relations across two perturbation categories: Audio Features-Based and Heuristic perturbations. MTAM applies these metamorphic relations to toxic audio content to generate test cases that remain harmful while being more likely to evade detection. In our evaluation, we employ MTAM to test five commercial textual content moderation software and an academic model against three kinds of toxic content. The results show that MTAM achieves up to 38.6%, 18.3%, 35.1%, 16.7%, and 51.1% error finding rates (EFR) when testing commercial moderation software provided by Gladia, Assembly AI, Baidu, Nextdata, and Tencent, respectively, and it obtains up to 45.7% EFR when testing the state-of-the-art algorithms from the academy.

http://arxiv.org/abs/2509.25178
GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs. (5%)
Aryan Yazdan Parast; Parsa Hosseini; Hesam Asadollahzadeh; Arshia Soltani Moakhar; Basim Azam; Soheil Feizi; Naveed Akhtar
Object hallucination in Multimodal Large Language Models (MLLMs) is a persistent failure mode that causes the model to perceive objects absent in the image. This weakness of MLLMs is currently studied using static benchmarks with fixed visual scenarios, which preempts the possibility of uncovering model-specific or unanticipated hallucination vulnerabilities. We introduce GHOST (Generating Hallucinations via Optimizing Stealth Tokens), a method designed to stress-test MLLMs by actively generating images that induce hallucination. GHOST is fully automatic and requires no human supervision or prior knowledge. It operates by optimizing in the image embedding space to mislead the model while keeping the target object absent, and then guiding a diffusion model conditioned on the embedding to generate natural-looking images. The resulting images remain visually natural and close to the original input, yet introduce subtle misleading cues that cause the model to hallucinate. We evaluate our method across a range of models, including reasoning models like GLM-4.1V-Thinking, and achieve a hallucination success rate exceeding 28%, compared to around 1% in prior data-driven discovery methods. We confirm that the generated images are both high-quality and object-free through quantitative metrics and human evaluation. Also, GHOST uncovers transferable vulnerabilities: images optimized for Qwen2.5-VL induce hallucinations in GPT-4o at a 66.5% rate. Finally, we show that fine-tuning on our images mitigates hallucination, positioning GHOST as both a diagnostic and corrective tool for building more reliable multimodal systems.

http://arxiv.org/abs/2412.02366
GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing. (1%)
Khawar Islam; Muhammad Zaigham Zaheer; Arif Mahmood; Karthik Nandakumar; Naveed Akhtar
Data augmentation is widely used to enhance generalization in visual classification tasks. However, traditional methods struggle when source and target domains differ, as in domain adaptation, due to their inability to address domain gaps. This paper introduces GenMix, a generalizable prompt-guided generative data augmentation approach that enhances both in-domain and cross-domain image classification. Our technique leverages image editing to generate augmented images based on custom conditional prompts, designed specifically for each problem type. By blending portions of the input image with its edited generative counterpart and incorporating fractal patterns, our approach mitigates unrealistic images and label ambiguity, improving the performance and adversarial robustness of the resulting models. Efficacy of our method is established with extensive experiments on eight public datasets for general and fine-grained classification, in both in-domain and cross-domain settings. Additionally, we demonstrate performance improvements for self-supervised learning, learning with data scarcity, and adversarial robustness. As compared to the existing state-of-the-art methods, our technique achieves stronger performance across the board.

http://arxiv.org/abs/2506.12519
Exploiting AI for Attacks: On the Interplay between Adversarial AI and Offensive AI. (1%)
Saskia Laura Schröer; Luca Pajola; Alberto Castagnaro; Giovanni Apruzzese; Mauro Conti
As Artificial Intelligence (AI) continues to evolve, it has transitioned from a research-focused discipline to a widely adopted technology, enabling intelligent solutions across various sectors. In security, AI's role in strengthening organizational resilience has been studied for over two decades. While much attention has focused on AI's constructive applications, the increasing maturity and integration of AI have also exposed its darker potentials. This article explores two emerging AI-related threats and the interplay between them: AI as a target of attacks (`Adversarial AI') and AI as a means to launch attacks on any target (`Offensive AI') -- potentially even on another AI. By cutting through the confusion and explaining these threats in plain terms, we introduce the complex and often misunderstood interplay between Adversarial AI and Offensive AI, offering a clear and accessible introduction to the challenges posed by these threats.

http://arxiv.org/abs/2509.25135
Learning in an Echo Chamber: Online Learning with Replay Adversary. (1%)
Daniil Dmitriev; Harald Eskelund Franck; Carolin Heinzler; Amartya Sanyal
As machine learning systems increasingly train on self-annotated data, they risk reinforcing errors and becoming echo chambers of their own beliefs. We model this phenomenon by introducing a learning-theoretic framework: Online Learning in the Replay Setting. In round $t$, the learner outputs a hypothesis $\hat{h}_t$; the adversary then reveals either the true label $f^\ast(x_t)$ or a replayed label $\hat{h}_i(x_t)$ from an earlier round $i < t$. A mistake is counted only when the true label is shown, yet classical algorithms such as the SOA or the halving algorithm are easily misled by the replayed errors.
  We introduce the Extended Threshold dimension, $\mathrm{ExThD}(\mathcal{H})$, and prove matching upper and lower bounds that make $\mathrm{ExThD}(\mathcal{H})$ the exact measure of learnability in this model. A closure-based learner makes at most $\mathrm{ExThD}(\mathcal{H})$ mistakes against any adaptive adversary, and no algorithm can perform better. For stochastic adversaries, we prove a similar bound for every intersection-closed class. The replay setting is provably harder than the classical mistake bound setting: some classes have constant Littlestone dimension but arbitrarily large $\mathrm{ExThD}(\mathcal{H})$. Proper learning exhibits an even sharper separation: a class is properly learnable under replay if and only if it is (almost) intersection-closed. Otherwise, every proper learner suffers $Ω(T)$ errors, whereas our improper algorithm still achieves the $\mathrm{ExThD}(\mathcal{H})$ bound. These results give the first tight analysis of learning against replay adversaries, based on new results for closure-type algorithms.

http://arxiv.org/abs/2503.13429
Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoning. (1%)
Nhi Pham; Artur Jesslen; Bernt Schiele; Adam Kortylewski; Jonas Fischer
With the rise of deep neural networks, especially in safety-critical applications, robustness and interpretability are crucial to ensure their trustworthiness. Recent advances in 3D-aware classifiers that map image features to volumetric representation of objects, rather than relying solely on 2D appearance, have greatly improved robustness on out-of-distribution (OOD) data. Such classifiers have not yet been studied from the perspective of interpretability. Meanwhile, current concept-based XAI methods often neglect OOD robustness. We aim to address both aspects with CAVE - Concept Aware Volumes for Explanations - a new direction that unifies interpretability and robustness in image classification. We design CAVE as a robust and inherently interpretable classifier that learns sparse concepts from 3D object representation. We further propose 3D Consistency (3D-C), a metric to measure spatial consistency of concepts. Unlike existing metrics that rely on human-annotated parts on images, 3D-C leverages ground-truth object meshes as a common surface to project and compare explanations across concept-based methods. CAVE achieves competitive classification performance while discovering consistent and meaningful concepts across images in various OOD settings. Code available at https://github.com/phamleyennhi/CAVE.

http://arxiv.org/abs/2505.14608
Language Models Optimized to Fool Detectors Still Have a Distinct Style (And How to Change It). (1%)
Rafael Rivera Soto; Barry Chen; Nicholas Andrews
Despite considerable progress in the development of machine-text detectors, it has been suggested that the problem is inherently hard, and therefore, that stakeholders should proceed under the assumption that machine-generated text cannot be reliably detected as such. We examine a recent such claim by Nicks et al. (2024) regarding the ease with which language models can be optimized to degrade the performance of machine-text detectors, including detectors not specifically optimized against. We identify a feature space -- the stylistic feature space -- that is robust to such optimization, and show that it may be used to reliably detect samples from language models optimized to prevent detection. Furthermore, we show that even when models are explicitly optimized against stylistic detectors, detection performance remains surprisingly unaffected. We then seek to understand if stylistic detectors are inherently more robust. To study this question, we explore a new paraphrasing approach that simultaneously aims to close the gap between human writing and machine writing in stylistic feature space while avoiding detection using traditional features. We show that when only a single sample is available for detection, this attack is universally effective across all detectors considered, including those that use writing style. However, as the number of samples available for detection grows, the human and machine distributions become distinguishable. Overall, our findings underscore previous recommendations to avoid reliance on machine-text detection.

http://arxiv.org/abs/2509.24171
Model Correlation Detection via Random Selection Probing. (1%)
Ruibo Chen; Sheng Zhang; Yihan Wu; Tong Zheng; Peihua Mai; Heng Huang
The growing prevalence of large language models (LLMs) and vision-language models (VLMs) has heightened the need for reliable techniques to determine whether a model has been fine-tuned from or is even identical to another. Existing similarity-based methods often require access to model parameters or produce heuristic scores without principled thresholds, limiting their applicability. We introduce Random Selection Probing (RSP), a hypothesis-testing framework that formulates model correlation detection as a statistical test. RSP optimizes textual or visual prefixes on a reference model for a random selection task and evaluates their transferability to a target model, producing rigorous p-values that quantify evidence of correlation. To mitigate false positives, RSP incorporates an unrelated baseline model to filter out generic, transferable features. We evaluate RSP across both LLMs and VLMs under diverse access conditions for reference models and test models. Experiments on fine-tuned and open-source models show that RSP consistently yields small p-values for related models while maintaining high p-values for unrelated ones. Extensive ablation studies further demonstrate the robustness of RSP. These results establish RSP as the first principled and general statistical framework for model correlation detection, enabling transparent and interpretable decisions in modern machine learning ecosystems.

http://arxiv.org/abs/2211.13723
Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective. (1%)
Hoang Phan; Lam Tran; Quyen Tran; Ngoc N. Tran; Tuan Truong; Qi Lei; Nhat Ho; Dinh Phung; Trung Le
Multi-task learning (MTL) trains deep neural networks to optimize several objectives simultaneously using a shared backbone, which leads to reduced computational costs, improved data efficiency, and enhanced performance through cross-task knowledge sharing. Although recent gradient manipulation techniques aim to find a common descent direction that benefits all tasks, conventional empirical loss minimization still leaves models vulnerable to overfitting and gradient conflicts. To address this, we introduce a novel MTL framework that leverages weight perturbation to regulate gradient norms, thus improving generalization. By adaptively modulating weight perturbations, our approach harmonizes task-specific gradients, reducing conflicts and encouraging more robust learning across tasks. Theoretical insights reveal that controlling the gradient norm through weight perturbation directly contributes to better generalization. Extensive experiments across diverse applications demonstrate that our method significantly outperforms existing gradient-based MTL techniques in terms of task performance and overall model robustness.

http://arxiv.org/abs/2509.24272
When MCP Servers Attack: Taxonomy, Feasibility, and Mitigation. (1%)
Weibo Zhao; Jiahao Liu; Bonan Ruan; Shaofei Li; Zhenkai Liang
Model Context Protocol (MCP) servers enable AI applications to connect to external systems in a plug-and-play manner, but their rapid proliferation also introduces severe security risks. Unlike mature software ecosystems with rigorous vetting, MCP servers still lack standardized review mechanisms, giving adversaries opportunities to distribute malicious implementations. Despite this pressing risk, the security implications of MCP servers remain underexplored. To address this gap, we present the first systematic study that treats MCP servers as active threat actors and decomposes them into core components to examine how adversarial developers can implant malicious intent. Specifically, we investigate three research questions: (i) what types of attacks malicious MCP servers can launch, (ii) how vulnerable MCP hosts and Large Language Models (LLMs) are to these attacks, and (iii) how feasible it is to carry out MCP server attacks in practice. Our study proposes a component-based taxonomy comprising twelve attack categories. For each category, we develop Proof-of-Concept (PoC) servers and demonstrate their effectiveness across diverse real-world host-LLM settings. We further show that attackers can generate large numbers of malicious servers at virtually no cost. We then test state-of-the-art scanners on the generated servers and found that existing detection approaches are insufficient. These findings highlight that malicious MCP servers are easy to implement, difficult to detect with current tools, and capable of causing concrete damage to AI agent systems. Addressing this threat requires coordinated efforts among protocol designers, host developers, LLM providers, and end users to build a more secure and resilient MCP ecosystem.

http://arxiv.org/abs/2509.23917
Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives. (99%)
Kuanrong Liu; Siyuan Liang; Cheng Qian; Ming Zhang; Xiaochun Cao
As a general-purpose vision-language pretraining model, CLIP demonstrates strong generalization ability in image-text alignment tasks and has been widely adopted in downstream applications such as image classification and image-text retrieval. However, it struggles with fine-grained tasks such as object detection and semantic segmentation. While many variants aim to improve CLIP on these tasks, its robustness to adversarial perturbations remains underexplored. Understanding how adversarial examples transfer across tasks is key to assessing CLIP's generalization limits and security risks. In this work, we conduct a systematic empirical analysis of the cross-task transfer behavior of CLIP-based models on image-text retrieval, object detection, and semantic segmentation under adversarial perturbations. We find that adversarial examples generated from fine-grained tasks (e.g., object detection and semantic segmentation) often exhibit stronger transfer potential than those from coarse-grained tasks, enabling more effective attacks against the original CLIP model. Motivated by this observation, we propose a novel framework, Multi-Task Adversarial CLIP (MT-AdvCLIP), which introduces a task-aware feature aggregation loss and generates perturbations with enhanced cross-task generalization capability. This design strengthens the attack effectiveness of fine-grained task models on the shared CLIP backbone. Experimental results on multiple public datasets show that MT-AdvCLIP significantly improves the adversarial transfer success rate (The average attack success rate across multiple tasks is improved by over 39%.) against various CLIP-derived models, without increasing the perturbation budget. This study reveals the transfer mechanism of adversarial examples in multi-task CLIP models, offering new insights into multi-task robustness evaluation and adversarial example design.

http://arxiv.org/abs/2509.23689
Merge Now, Regret Later: The Hidden Cost of Model Merging is Adversarial Transferability. (99%)
Ankit Gangwal; Aaryan Ajay Sharma
Model Merging (MM) has emerged as a promising alternative to multi-task learning, where multiple fine-tuned models are combined, without access to tasks' training data, into a single model that maintains performance across tasks. Recent works have explored the impact of MM on adversarial attacks, particularly backdoor attacks. However, none of them have sufficiently explored its impact on transfer attacks using adversarial examples, i.e., a black-box adversarial attack where examples generated for a surrogate model successfully mislead a target model.
  In this work, we study the effect of MM on the transferability of adversarial examples. We perform comprehensive evaluations and statistical analysis consisting of 8 MM methods, 7 datasets, and 6 attack methods, sweeping over 336 distinct attack settings. Through it, we first challenge the prevailing notion of MM conferring free adversarial robustness, and show MM cannot reliably defend against transfer attacks, with over 95% relative transfer attack success rate. Moreover, we reveal 3 key insights for machine-learning practitioners regarding MM and transferability for a robust system design: (1) stronger MM methods increase vulnerability to transfer attacks; (2) mitigating representation bias increases vulnerability to transfer attacks; and (3) weight averaging, despite being the weakest MM method, is the most vulnerable MM method to transfer attacks. Finally, we analyze the underlying reasons for this increased vulnerability, and provide potential solutions to the problem. Our findings offer critical insights for designing more secure systems employing MM.

http://arxiv.org/abs/2506.18248
Improving Black-Box Generative Attacks via Generator Semantic Consistency. (98%)
Jongoh Jeong; Hunmin Yang; Jaeseok Jeong; Kuk-Jin Yoon
Transfer attacks optimize on a surrogate and deploy to a black-box target. While iterative optimization attacks in this paradigm are limited by their per-input cost limits efficiency and scalability due to multistep gradient updates for each input, generative attacks alleviate these by producing adversarial examples in a single forward pass at test time. However, current generative attacks still adhere to optimizing surrogate losses (e.g., feature divergence) and overlook the generator's internal dynamics, underexploring how the generator's internal representations shape transferable perturbations. To address this, we enforce semantic consistency by aligning the early generator's intermediate features to an EMA teacher, stabilizing object-aligned representations and improving black-box transfer without inference-time overhead. To ground the mechanism, we quantify semantic stability as the standard deviation of foreground IoU between cluster-derived activation masks and foreground masks across generator blocks, and observe reduced semantic drift under our method. For more reliable evaluation, we also introduce Accidental Correction Rate (ACR) to separate inadvertent corrections from intended misclassifications, complementing the inherent blind spots in traditional Attack Success Rate (ASR), Fooling Rate (FR), and Accuracy metrics. Across architectures, domains, and tasks, our approach can be seamlessly integrated into existing generative attacks with consistent improvements in black-box transfer, while maintaining test-time efficiency.

http://arxiv.org/abs/2509.23961
Learning-Based Testing for Deep Learning: Enhancing Model Robustness with Adversarial Input Prioritization. (98%)
Sheikh Md Mushfiqur Rahman; Nasir Eisty
Context: Deep Neural Networks (DNNs) are increasingly deployed in critical applications, where resilience against adversarial inputs is paramount. However, whether coverage-based or confidence-based, existing test prioritization methods often fail to efficiently identify the most fault-revealing inputs, limiting their practical effectiveness. Aims: This project aims to enhance fault detection and model robustness in DNNs by integrating Learning-Based Testing (LBT) with hypothesis and mutation testing to efficiently prioritize adversarial test cases. Methods: Our method selects a subset of adversarial inputs with a high likelihood of exposing model faults, without relying on architecture-specific characteristics or formal verification, making it adaptable across diverse DNNs. Results: Our results demonstrate that the proposed LBT method consistently surpasses baseline approaches in prioritizing fault-revealing inputs and accelerating fault detection. By efficiently organizing test permutations, it uncovers all potential faults significantly faster across various datasets, model architectures, and adversarial attack techniques. Conclusion: Beyond improving fault detection, our method preserves input diversity and provides effective guidance for model retraining, further enhancing robustness. These advantages establish our approach as a powerful and practical solution for adversarial test prioritization in real-world DNN applications.

http://arxiv.org/abs/2509.23762
Accuracy-Robustness Trade Off via Spiking Neural Network Gradient Sparsity Trail. (81%)
Nhan T. Luu
Spiking Neural Networks (SNNs) have attracted growing interest in both computational neuroscience and artificial intelligence, primarily due to their inherent energy efficiency and compact memory footprint. However, achieving adversarial robustness in SNNs, particularly for vision-related tasks, remains a nascent and underexplored challenge. Recent studies have proposed leveraging sparse gradients as a form of regularization to enhance robustness against adversarial perturbations. In this work, we present a surprising finding: under specific architectural configurations, SNNs exhibit natural gradient sparsity and can achieve state-of-the-art adversarial defense performance without the need for any explicit regularization. Further analysis reveals a trade-off between robustness and generalization: while sparse gradients contribute to improved adversarial resilience, they can impair the model's ability to generalize; conversely, denser gradients support better generalization but increase vulnerability to attacks.

http://arxiv.org/abs/2509.23594
StolenLoRA: Exploring LoRA Extraction Attacks via Synthetic Data. (74%)
Yixu Wang; Yan Teng; Yingchun Wang; Xingjun Ma
Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have transformed vision model adaptation, enabling the rapid deployment of customized models. However, the compactness of LoRA adaptations introduces new safety concerns, particularly their vulnerability to model extraction attacks. This paper introduces a new focus of model extraction attacks named LoRA extraction that extracts LoRA-adaptive models based on a public pre-trained model. We then propose a novel extraction method called StolenLoRA which trains a substitute model to extract the functionality of a LoRA-adapted model using synthetic data. StolenLoRA leverages a Large Language Model to craft effective prompts for data generation, and it incorporates a Disagreement-based Semi-supervised Learning (DSL) strategy to maximize information gain from limited queries. Our experiments demonstrate the effectiveness of StolenLoRA, achieving up to a 96.60% attack success rate with only 10k queries, even in cross-backbone scenarios where the attacker and victim models utilize different pre-trained backbones. These findings reveal the specific vulnerability of LoRA-adapted models to this type of extraction and underscore the urgent need for robust defense mechanisms tailored to PEFT methods. We also explore a preliminary defense strategy based on diversified LoRA deployments, highlighting its potential to mitigate such attacks.

http://arxiv.org/abs/2508.07173
Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models. (1%)
Leyi Pan; Zheyu Fu; Yunpeng Zhai; Shuchang Tao; Sheng Guan; Shiyu Huang; Lingzhe Zhang; Zhaoyang Liu; Bolin Ding; Felix Henry; Aiwei Liu; Lijie Wen
The rise of Omni-modal Large Language Models (OLLMs), which integrate visual and auditory processing with text, necessitates robust safety evaluations to mitigate harmful outputs. However, no dedicated benchmarks currently exist for OLLMs, and existing benchmarks fail to assess safety under joint audio-visual inputs or cross-modal consistency. To fill this gap, we introduce Omni-SafetyBench, the first comprehensive parallel benchmark for OLLM safety evaluation, featuring 24 modality variations with 972 samples each, including audio-visual harm cases. Considering OLLMs' comprehension challenges with complex omni-modal inputs and the need for cross-modal consistency evaluation, we propose tailored metrics: a Safety-score based on Conditional Attack Success Rate (C-ASR) and Refusal Rate (C-RR) to account for comprehension failures, and a Cross-Modal Safety Consistency score (CMSC-score) to measure consistency across modalities. Evaluating 6 open-source and 4 closed-source OLLMs reveals critical vulnerabilities: (1) only 3 models achieving over 0.6 in both average Safety-score and CMSC-score; (2) safety defenses weaken with complex inputs, especially audio-visual joints; (3) severe weaknesses persist, with some models scoring as low as 0.14 on specific modalities. Using Omni-SafetyBench, we evaluated existing safety alignment algorithms and identified key challenges in OLLM safety alignment: (1) Inference-time methods are inherently less effective as they cannot alter the model's underlying understanding of safety; (2) Post-training methods struggle with out-of-distribution issues due to the vast modality combinations in OLLMs; and, safety tasks involving audio-visual inputs are more complex, making even in-distribution training data less effective. Our proposed benchmark, metrics and the findings highlight urgent needs for enhanced OLLM safety.

http://arxiv.org/abs/2509.23597
Characteristic Root Analysis and Regularization for Linear Time Series Forecasting. (1%)
Zheng Wang; Kaixuan Zhang; Wanfang Chen; Xiaonan Lu; Longyuan Li; Tobias Schlagenhauf
Time series forecasting remains a critical challenge across numerous domains, yet the effectiveness of complex models often varies unpredictably across datasets. Recent studies highlight the surprising competitiveness of simple linear models, suggesting that their robustness and interpretability warrant deeper theoretical investigation. This paper presents a systematic study of linear models for time series forecasting, with a focus on the role of characteristic roots in temporal dynamics. We begin by analyzing the noise-free setting, where we show that characteristic roots govern long-term behavior and explain how design choices such as instance normalization and channel independence affect model capabilities. We then extend our analysis to the noisy regime, revealing that models tend to produce spurious roots. This leads to the identification of a key data-scaling property: mitigating the influence of noise requires disproportionately large training data, highlighting the need for structural regularization. To address these challenges, we propose two complementary strategies for robust root restructuring. The first uses rank reduction techniques, including Reduced-Rank Regression and Direct Weight Rank Reduction, to recover the low-dimensional latent dynamics. The second, a novel adaptive method called Root Purge, encourages the model to learn a noise-suppressing null space during training. Extensive experiments on standard benchmarks demonstrate the effectiveness of both approaches, validating our theoretical insights and achieving state-of-the-art results in several settings. Our findings underscore the potential of integrating classical theories for linear systems with modern learning techniques to build robust, interpretable, and data-efficient forecasting models.

http://arxiv.org/abs/2509.23871
Taught Well Learned Ill: Towards Distillation-conditional Backdoor Attack. (1%)
Yukun Chen; Boheng Li; Yu Yuan; Leyi Qi; Yiming Li; Tianwei Zhang; Zhan Qin; Kui Ren
Knowledge distillation (KD) is a vital technique for deploying deep neural networks (DNNs) on resource-constrained devices by transferring knowledge from large teacher models to lightweight student models. While teacher models from third-party platforms may undergo security verification (\eg, backdoor detection), we uncover a novel and critical threat: distillation-conditional backdoor attacks (DCBAs). DCBA injects dormant and undetectable backdoors into teacher models, which become activated in student models via the KD process, even with clean distillation datasets. While the direct extension of existing methods is ineffective for DCBA, we implement this attack by formulating it as a bilevel optimization problem and proposing a simple yet effective method (\ie, SCAR). Specifically, the inner optimization simulates the KD process by optimizing a surrogate student model, while the outer optimization leverages outputs from this surrogate to optimize the teacher model for implanting the conditional backdoor. Our SCAR addresses this complex optimization utilizing an implicit differentiation algorithm with a pre-optimized trigger injection function. Extensive experiments across diverse datasets, model architectures, and KD techniques validate the effectiveness of our SCAR and its resistance against existing backdoor detection, highlighting a significant yet previously overlooked vulnerability in the KD process. Our code is available at https://github.com/WhitolfChen/SCAR.

http://arxiv.org/abs/2503.10549
Controllable Adversarial Makeup for Privacy via Text-Guided Diffusion. (1%)
Youngjin Kwon; Xiao Zhang
As face recognition becomes more widespread in government and commercial services, its potential misuse raises serious concerns about privacy and civil rights. To counteract this threat, various anti-facial recognition techniques have been proposed, which protect privacy by adversarially perturbing face images. Among these, generative makeup-based approaches are the most widely studied. However, these methods, designed primarily to impersonate specific target identities, can only achieve weak dodging success rates while increasing the risk of targeted abuse. In addition, they often introduce global visual artifacts or a lack of adaptability to accommodate diverse makeup prompts, compromising user satisfaction. To address the above limitations, we develop MASQUE, a novel diffusion-based framework that generates localized adversarial makeups guided by user-defined text prompts. Built upon precise null-text inversion, customized cross-attention fusion with masking, and a pairwise adversarial guidance mechanism using images of the same individual, MASQUE achieves robust dodging performance without requiring any external identity. Comprehensive evaluations on open-source facial recognition models and commercial APIs demonstrate that MASQUE significantly improves dodging success rates over all baselines, along with higher perceptual fidelity preservation, stronger adaptability to various makeup prompts, and robustness to image transformations.

http://arxiv.org/abs/2509.23789
Visual CoT Makes VLMs Smarter but More Fragile. (1%)
Chunxue Xu; Yiwei Wang; Yujun Cai; Bryan Hooi; Songze Li
Chain-of-Thought (CoT) techniques have significantly enhanced reasoning in Vision-Language Models (VLMs). Extending this paradigm, Visual CoT integrates explicit visual edits, such as cropping or annotating regions of interest, into the reasoning process, achieving superior multimodal performance. However, the robustness of Visual CoT-based VLMs against image-level noise remains unexplored. In this paper, we present the first systematic evaluation of Visual CoT robustness under visual perturbations. Our benchmark spans 12 image corruption types across 4 Visual Question Answering (VQA) datasets, enabling a comprehensive comparison between VLMs that use Visual CoT, and VLMs that do not. The results reveal that integrating Visual CoT consistently improves absolute accuracy regardless of whether the input images are clean or corrupted by noise; however, it also increases sensitivity to input perturbations, resulting in sharper performance degradation compared to standard VLMs. Through extensive analysis, we identify the intermediate reasoning components of Visual CoT, i.e., the edited image patches , as the primary source of fragility. Building on this analysis, we propose a plug-and-play robustness enhancement method that integrates Grounding DINO model into the Visual CoT pipeline, providing high-confidence local visual cues to stabilize reasoning. Our work reveals clear fragility patterns in Visual CoT and offers an effective, architecture-agnostic solution for enhancing visual robustness.

http://arxiv.org/abs/2509.23198
Real-World Transferable Adversarial Attack on Face-Recognition Systems. (99%)
Andrey Kaznacheev; Matvey Mikhalchuk; Andrey Kuznetsov; Aleksandr Petiushko; Anton Razzhigaev
Adversarial attacks on face recognition (FR) systems pose a significant security threat, yet most are confined to the digital domain or require white-box access. We introduce GaP (Gaussian Patch), a novel method to generate a universal, physically transferable adversarial patch under a strict black-box setting. Our approach uses a query-efficient, zero-order greedy algorithm to iteratively construct a symmetric, grayscale pattern for the forehead. The patch is optimized by successively adding Gaussian blobs, guided only by the cosine similarity scores from a surrogate FR model to maximally degrade identity recognition. We demonstrate that with approximately 10,000 queries to a black-box ArcFace model, the resulting GaP achieves a high attack success rate in both digital and real-world physical tests. Critically, the attack shows strong transferability, successfully deceiving an entirely unseen FaceNet model. Our work highlights a practical and severe vulnerability, proving that robust, transferable attacks can be crafted with limited knowledge of the target system.

http://arxiv.org/abs/2509.23519
ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search. (93%)
Zeyu Shen; Basileal Imana; Tong Wu; Chong Xiang; Prateek Mittal; Aleksandra Korolova
Retrieval-Augmented Generation (RAG) enhances Large Language Models by grounding their outputs in external documents. These systems, however, remain vulnerable to attacks on the retrieval corpus, such as prompt injection. RAG-based search systems (e.g., Google's Search AI Overview) present an interesting setting for studying and protecting against such threats, as defense algorithms can benefit from built-in reliability signals -- like document ranking -- and represent a non-LLM challenge for the adversary due to decades of work to thwart SEO.
  Motivated by, but not limited to, this scenario, this work introduces ReliabilityRAG, a framework for adversarial robustness that explicitly leverages reliability information of retrieved documents.
  Our first contribution adopts a graph-theoretic perspective to identify a "consistent majority" among retrieved documents to filter out malicious ones. We introduce a novel algorithm based on finding a Maximum Independent Set (MIS) on a document graph where edges encode contradiction. Our MIS variant explicitly prioritizes higher-reliability documents and provides provable robustness guarantees against bounded adversarial corruption under natural assumptions. Recognizing the computational cost of exact MIS for large retrieval sets, our second contribution is a scalable weighted sample and aggregate framework. It explicitly utilizes reliability information, preserving some robustness guarantees while efficiently handling many documents.
  We present empirical results showing ReliabilityRAG provides superior robustness against adversarial attacks compared to prior methods, maintains high benign accuracy, and excels in long-form generation tasks where prior robustness-focused methods struggled. Our work is a significant step towards more effective, provably robust defenses against retrieved corpus corruption in RAG.

http://arxiv.org/abs/2509.23279
Vid-Freeze: Protecting Images from Malicious Image-to-Video Generation via Temporal Freezing. (93%)
Rohit Chowdhury; Aniruddha Bala; Rohan Jaiswal; Siddharth Roheda
The rapid progress of image-to-video (I2V) generation models has introduced significant risks, enabling video synthesis from static images and facilitating deceptive or malicious content creation. While prior defenses such as I2VGuard attempt to immunize images, effective and principled protection to block motion remains underexplored. In this work, we introduce Vid-Freeze - a novel attention-suppressing adversarial attack that adds carefully crafted adversarial perturbations to images. Our method explicitly targets the attention mechanism of I2V models, completely disrupting motion synthesis while preserving semantic fidelity of the input image. The resulting immunized images generate stand-still or near-static videos, effectively blocking malicious content creation. Our experiments demonstrate the impressive protection provided by the proposed approach, highlighting the importance of attention attacks as a promising direction for robust and proactive defenses against misuse of I2V generation models.

http://arxiv.org/abs/2509.23010
Desensitizing for Improving Corruption Robustness in Point Cloud Classification through Adversarial Training. (92%)
Zhiqiang Tian; Weigang Li; Chunhua Deng; Junwei Hu; Yongqiang Wang; Wenping Liu
Due to scene complexity, sensor inaccuracies, and processing imprecision, point cloud corruption is inevitable. Over-reliance on input features is the root cause of DNN vulnerabilities. It remains unclear whether this issue exists in 3D tasks involving point clouds and whether reducing dependence on these features can enhance the model's robustness to corrupted point clouds. This study attempts to answer these questions. Specifically, we quantified the sensitivity of the DNN to point cloud features using Shapley values and found that models trained using traditional methods exhibited high sensitivity values for certain features. Furthermore, under an equal pruning ratio, prioritizing the pruning of highly sensitive features causes more severe damage to model performance than random pruning. We propose `Desensitized Adversarial Training' (DesenAT), generating adversarial samples using feature desensitization and conducting training within a self-distillation framework, which aims to alleviate DNN's over-reliance on point clouds features by smoothing sensitivity. First, data points with high contribution components are eliminated, and spatial transformation is used to simulate corruption scenes, generate adversarial samples, and conduct adversarial training on the model. Next, to compensate for information loss in adversarial samples, we use the self-distillation method to transfer knowledge from clean samples to adversarial samples, and perform adversarial training in a distillation manner.Extensive experiments on ModelNet-C and PointCloud-C demonstrate show that the propose method can effectively improve the robustness of the model without reducing the performance of clean data sets. This code is publicly available at \href{https://github.com/JerkyT/DesenAT/tree/master}{https://github.com/JerkyT/DesenAT}.

http://arxiv.org/abs/2509.23325
Robust Fine-Tuning from Non-Robust Pretrained Models: Mitigating Suboptimal Transfer With Adversarial Scheduling. (92%)
Jonas Ngnawé; Maxime Heuillet; Sabyasachi Sahoo; Yann Pequignot; Ola Ahmad; Audrey Durand; Frédéric Precioso; Christian Gagné
Fine-tuning pretrained models is a standard and effective workflow in modern machine learning. However, robust fine-tuning (RFT), which aims to simultaneously achieve adaptation to a downstream task and robustness to adversarial examples, remains challenging. Despite the abundance of non-robust pretrained models in open-source repositories, their potential for RFT is less understood. We address this knowledge gap by systematically examining RFT from such non-robust models. Our experiments reveal that fine-tuning non-robust models with a robust objective, even under small perturbations, can lead to poor performance, a phenomenon that we dub \emph{suboptimal transfer}. In challenging scenarios (eg, difficult tasks, high perturbation), the resulting performance can be so low that it may be considered a transfer failure. We find that fine-tuning using a robust objective impedes task adaptation at the beginning of training and eventually prevents optimal transfer. However, we propose a novel heuristic, \emph{Epsilon-Scheduling}, a schedule over perturbation strength used during training that promotes optimal transfer. Additionally, we introduce \emph{expected robustness}, a metric that captures performance across a range of perturbations, providing a more comprehensive evaluation of the accuracy-robustness trade-off for diverse models at test time. Extensive experiments on a wide range of configurations (six pretrained models and five datasets) show that \emph{Epsilon-Scheduling} successfully prevents \emph{suboptimal transfer} and consistently improves expected robustness.

http://arxiv.org/abs/2509.23333
Targeted perturbations reveal brain-like local coding axes in robustified, but not standard, ANN-based brain models. (54%)
Nikolas McNeal; N. Apurva Ratan Murty
Artificial neural networks (ANNs) have become the de facto standard for modeling the human visual system, primarily due to their success in predicting neural responses. However, with many models now achieving similar predictive accuracy, we need a stronger criterion. Here, we use small-scale adversarial probes to characterize the local representational geometry of many highly predictive ANN-based brain models. We report four key findings. First, we show that most contemporary ANN-based brain models are unexpectedly fragile. Despite high prediction scores, their response predictions are highly sensitive to small, imperceptible perturbations, revealing unreliable local coding directions. Second, we demonstrate that a model's sensitivity to adversarial probes can better discriminate between candidate neural encoding models than prediction accuracy alone. Third, we find that standard models rely on distinct local coding directions that do not transfer across model architectures. Finally, we show that adversarial probes from robustified models produce generalizable and semantically meaningful changes, suggesting that they capture the local coding dimensions of the visual system. Together, our work shows that local representational geometry provides a stronger criterion for brain model evaluation. We also provide empirical grounds for favoring robust models, whose more stable coding axes not only align better with neural selectivity but also generate concrete, testable predictions for future experiments.

http://arxiv.org/abs/2507.07871
Mitigating Watermark Forgery in Generative Models via Randomized Key Selection. (47%)
Toluwani Aremu; Noor Hussein; Munachiso Nwadike; Samuele Poppi; Jie Zhang; Karthik Nandakumar; Neil Gong; Nils Lukas
Watermarking enables GenAI providers to verify whether content was generated by their models. A watermark is a hidden signal in the content, whose presence can be detected using a secret watermark key. A core security threat are forgery attacks, where adversaries insert the provider's watermark into content \emph{not} produced by the provider, potentially damaging their reputation and undermining trust. Existing defenses resist forgery by embedding many watermarks with multiple keys into the same content, which can degrade model utility. However, forgery remains a threat when attackers can collect sufficiently many watermarked samples. We propose a defense that is provably forgery-resistant \emph{independent} of the number of watermarked content collected by the attacker, provided they cannot easily distinguish watermarks from different keys. Our scheme does not further degrade model utility. We randomize the watermark key selection for each query and accept content as genuine only if a watermark is detected by \emph{exactly} one key. We focus on the image and text modalities, but our defense is modality-agnostic, since it treats the underlying watermarking method as a black-box. Our method provably bounds the attacker's success rate and we empirically observe a reduction from near-perfect success rates to only $2\%$ at negligible computational overhead.

http://arxiv.org/abs/2509.23041
Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Data. (22%)
Zi Liang; Qingqing Ye; Xuan Liu; Yanyun Wang; Jianliang Xu; Haibo Hu
Synthetic data refers to artificial samples generated by models. While it has been validated to significantly enhance the performance of large language models (LLMs) during training and has been widely adopted in LLM development, potential security risks it may introduce remain uninvestigated. This paper systematically evaluates the resilience of synthetic-data-integrated training paradigm for LLMs against mainstream poisoning and backdoor attacks. We reveal that such a paradigm exhibits strong resistance to existing attacks, primarily thanks to the different distribution patterns between poisoning data and queries used to generate synthetic samples. To enhance the effectiveness of these attacks and further investigate the security risks introduced by synthetic data, we introduce a novel and universal attack framework, namely, Virus Infection Attack (VIA), which enables the propagation of current attacks through synthetic data even under purely clean queries. Inspired by the principles of virus design in cybersecurity, VIA conceals the poisoning payload within a protective "shell" and strategically searches for optimal hijacking points in benign samples to maximize the likelihood of generating malicious content. Extensive experiments on both data poisoning and backdoor attacks show that VIA significantly increases the presence of poisoning content in synthetic data and correspondingly raises the attack success rate (ASR) on downstream models to levels comparable to those observed in the poisoned upstream models.

http://arxiv.org/abs/2509.23037
GuardNet: Graph-Attention Filtering for Jailbreak Defense in Large Language Models. (9%)
Javad Forough; Mohammad Maheri; Hamed Haddadi
Large Language Models (LLMs) are increasingly susceptible to jailbreak attacks, which are adversarial prompts that bypass alignment constraints and induce unauthorized or harmful behaviors. These vulnerabilities undermine the safety, reliability, and trustworthiness of LLM outputs, posing critical risks in domains such as healthcare, finance, and legal compliance. In this paper, we propose GuardNet, a hierarchical filtering framework that detects and filters jailbreak prompts prior to inference. GuardNet constructs structured graphs that combine sequential links, syntactic dependencies, and attention-derived token relations to capture both linguistic structure and contextual patterns indicative of jailbreak behavior. It then applies graph neural networks at two levels: (i) a prompt-level filter that detects global adversarial prompts, and (ii) a token-level filter that pinpoints fine-grained adversarial spans. Extensive experiments across three datasets and multiple attack settings show that GuardNet substantially outperforms prior defenses. It raises prompt-level F$_1$ scores from 66.4\% to 99.8\% on LLM-Fuzzer, and from 67-79\% to over 94\% on PLeak datasets. At the token level, GuardNet improves F$_1$ from 48-75\% to 74-91\%, with IoU gains up to +28\%. Despite its structural complexity, GuardNet maintains acceptable latency and generalizes well in cross-domain evaluations, making it a practical and robust defense against jailbreak threats in real-world LLM deployments.

http://arxiv.org/abs/2505.20955
Unveiling Impact of Frequency Components on Membership Inference Attacks for Diffusion Models. (1%)
Puwei Lian; Yujun Cai; Songze Li; Bingkun Bao
Diffusion models have achieved tremendous success in image generation, but they also raise significant concerns regarding privacy and copyright issues. Membership Inference Attacks (MIAs) are designed to ascertain whether specific data were utilized during a model's training phase. As current MIAs for diffusion models typically exploit the model's image prediction ability, we formalize them into a unified general paradigm which computes the membership score for membership identification. Under this paradigm, we empirically find that existing attacks overlook the inherent deficiency in how diffusion models process high-frequency information. Consequently, this deficiency leads to member data with more high-frequency content being misclassified as hold-out data, and hold-out data with less high-frequency content tend to be misclassified as member data. Moreover, we theoretically demonstrate that this deficiency reduces the membership advantage of attacks, thereby interfering with the effective discrimination of member data and hold-out data. Based on this insight, we propose a plug-and-play high-frequency filter module to mitigate the adverse effects of the deficiency, which can be seamlessly integrated into any attacks within this general paradigm without additional time costs. Extensive experiments corroborate that this module significantly improves the performance of baseline attacks across different datasets and models.

http://arxiv.org/abs/2509.23362
Dual-Space Smoothness for Robust and Balanced LLM Unlearning. (1%)
Han Yan; Zheyuan Liu; Meng Jiang
With the rapid advancement of large language models, Machine Unlearning has emerged to address growing concerns around user privacy, copyright infringement, and overall safety. Yet state-of-the-art (SOTA) unlearning methods often suffer from catastrophic forgetting and metric imbalance, for example by over-optimizing one objective (e.g., unlearning effectiveness, utility preservation, or privacy protection) at the expense of others. In addition, small perturbations in the representation or parameter space can be exploited by relearn and jailbreak attacks. To address these challenges, we propose PRISM, a unified framework that enforces dual-space smoothness in representation and parameter spaces to improve robustness and balance unlearning metrics. PRISM consists of two smoothness optimization stages: (i) a representation space stage that employs a robustly trained probe to defend against jailbreak attacks, and (ii) a parameter-space stage that decouples retain-forget gradient conflicts, reduces imbalance, and smooths the parameter space to mitigate relearning attacks. Extensive experiments on WMDP and MUSE, across conversational-dialogue and continuous-text settings, show that PRISM outperforms SOTA baselines under multiple attacks while achieving a better balance among key metrics.

http://arxiv.org/abs/2509.22850
Boundary on the Table: Efficient Black-Box Decision-Based Attacks for Structured Data. (99%)
Roie Kazoom; Yuval Ratzabi; Etamar Rothstein; Ofer Hadar
Adversarial robustness in structured data remains an underexplored frontier compared to vision and language domains. In this work, we introduce a novel black-box, decision-based adversarial attack tailored for tabular data. Our approach combines gradient-free direction estimation with an iterative boundary search, enabling efficient navigation of discrete and continuous feature spaces under minimal oracle access. Extensive experiments demonstrate that our method successfully compromises nearly the entire test set across diverse models, ranging from classical machine learning classifiers to large language model (LLM)-based pipelines. Remarkably, the attack achieves success rates consistently above 90%, while requiring only a small number of queries per instance. These results highlight the critical vulnerability of tabular models to adversarial perturbations, underscoring the urgent need for stronger defenses in real-world decision-making systems.

http://arxiv.org/abs/2509.22393
Text Adversarial Attacks with Dynamic Outputs. (99%)
Wenqiang Wang; Siyuan Liang; Xiao Yan; Xiaochun Cao
Text adversarial attack methods are typically designed for static scenarios with fixed numbers of output labels and a predefined label space, relying on extensive querying of the victim model (query-based attacks) or the surrogate model (transfer-based attacks). To address this gap, we introduce the Textual Dynamic Outputs Attack (TDOA) method, which employs a clustering-based surrogate model training approach to convert the dynamic-output scenario into a static single-output scenario. To improve attack effectiveness, we propose the farthest-label targeted attack strategy, which selects adversarial vectors that deviate most from the model's coarse-grained labels, thereby maximizing disruption. We extensively evaluate TDOA on four datasets and eight victim models (e.g., ChatGPT-4o, ChatGPT-4.1), showing its effectiveness in crafting adversarial examples and its strong potential to compromise large language models with limited access. With a single query per text, TDOA achieves a maximum attack success rate of 50.81\%. Additionally, we find that TDOA also achieves state-of-the-art performance in conventional static output scenarios, reaching a maximum ASR of 82.68\%. Meanwhile, by conceptualizing translation tasks as classification problems with unbounded output spaces, we extend the TDOA framework to generative settings, surpassing prior results by up to 0.64 RDBLEU and 0.62 RDchrF.

http://arxiv.org/abs/2509.20793
FERD: Fairness-Enhanced Data-Free Robustness Distillation. (99%)
Zhengxiao Li; Liming Lu; Xu Zheng; Siyuan Liang; Zhenghan Chen; Yongbin Zhou; Shuchao Pang
Data-Free Robustness Distillation (DFRD) aims to transfer the robustness from the teacher to the student without accessing the training data. While existing methods focus on overall robustness, they overlook the robust fairness issues, leading to severe disparity of robustness across different categories. In this paper, we find two key problems: (1) student model distilled with equal class proportion data behaves significantly different across distinct categories; and (2) the robustness of student model is not stable across different attacks target. To bridge these gaps, we present the first Fairness-Enhanced data-free Robustness Distillation (FERD) framework to adjust the proportion and distribution of adversarial examples. For the proportion, FERD adopts a robustness-guided class reweighting strategy to synthesize more samples for the less robust categories, thereby improving robustness of them. For the distribution, FERD generates complementary data samples for advanced robustness distillation. It generates Fairness-Aware Examples (FAEs) by enforcing a uniformity constraint on feature-level predictions, which suppress the dominance of class-specific non-robust features, providing a more balanced representation across all categories. Then, FERD constructs Uniform-Target Adversarial Examples (UTAEs) from FAEs by applying a uniform target class constraint to avoid biased attack directions, which distribute the attack targets across all categories and prevents overfitting to specific vulnerable categories. Extensive experiments on three public datasets show that FERD achieves state-of-the-art worst-class robustness under all adversarial attack (e.g., the worst-class robustness under FGSM and AutoAttack are improved by 15.1\% and 6.4\% using MobileNet-V2 on CIFAR-10), demonstrating superior performance in both robustness and fairness aspects.

http://arxiv.org/abs/2509.22060
Decoding Deception: Understanding Automatic Speech Recognition Vulnerabilities in Evasion and Poisoning Attacks. (99%)
Aravindhan G; Yuvaraj Govindarajulu; Parin Shah
Recent studies have demonstrated the vulnerability of Automatic Speech Recognition systems to adversarial examples, which can deceive these systems into misinterpreting input speech commands. While previous research has primarily focused on white-box attacks with constrained optimizations, and transferability based black-box attacks against commercial Automatic Speech Recognition devices, this paper explores cost efficient white-box attack and non transferability black-box adversarial attacks on Automatic Speech Recognition systems, drawing insights from approaches such as Fast Gradient Sign Method and Zeroth-Order Optimization. Further, the novelty of the paper includes how poisoning attack can degrade the performances of state-of-the-art models leading to misinterpretation of audio signals. Through experimentation and analysis, we illustrate how hybrid models can generate subtle yet impactful adversarial examples with very little perturbation having Signal Noise Ratio of 35dB that can be generated within a minute. These vulnerabilities of state-of-the-art open source model have practical security implications, and emphasize the need for adversarial security.

http://arxiv.org/abs/2505.15130
Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Models. (98%)
Sajjad Ghiasvand; Haniyeh Ehsani Oskouie; Mahnoosh Alizadeh; Ramtin Pedarsani
Vision-Language Models (VLMs) such as CLIP have shown remarkable performance in cross-modal tasks through large-scale contrastive pre-training. To adapt these large transformer-based models efficiently for downstream tasks, Parameter-Efficient Fine-Tuning (PEFT) techniques like (Low-Rank Adaptation) LoRA have emerged as scalable alternatives to full fine-tuning, especially in few-shot scenarios. However, like traditional deep neural networks, VLMs are highly vulnerable to adversarial attacks, where imperceptible perturbations can significantly degrade model performance. Adversarial training remains the most effective strategy for improving model robustness in PEFT. In this work, we propose AdvCLIP-LoRA, to our knowledge the first method designed to enhance the adversarial robustness of CLIP models fine-tuned with LoRA in few-shot settings. Our method formulates training as a minimax optimization over low-rank adapters and adversarial perturbations, enabling robust adaptation with a small trainable footprint. Across eight datasets and two backbones (ViT-B/16 and ViT-B/32), AdvCLIP-LoRA achieves state-of-the-art performance in few-shot classification, adversarial base-to-new generalization, and cross-dataset transfer, delivering higher adversarial robustness than prompt tuning baselines without sacrificing much clean accuracy. These findings highlight AdvCLIP-LoRA as a practical approach for robust adaptation of VLMs in resource-constrained settings.

http://arxiv.org/abs/2501.01908
Training-Free Defense Against Adversarial Attacks in Deep Learning MRI Reconstruction. (98%)
Mahdi Saberi; Chi Zhang; Mehmet Akçakaya
Deep learning (DL) methods have become the state-of-the-art for reconstructing sub-sampled magnetic resonance imaging (MRI) data. However, studies have shown that these methods are susceptible to small adversarial input perturbations, or attacks, resulting in major distortions in the output images. Various strategies have been proposed to reduce the effects of these attacks, but they require retraining and may lower reconstruction quality for non-perturbed/clean inputs. In this work, we propose a novel approach for mitigating adversarial attacks on MRI reconstruction models without any retraining. Based on the idea of cyclic measurement consistency, we devise a novel mitigation objective that is minimized in a small ball around the attack input. Results show that our method substantially reduces the impact of adversarial perturbations across different datasets, attack types/strengths and PD-DL networks, and qualitatively and quantitatively outperforms conventional mitigation methods that involve retraining. We also introduce a practically relevant scenario for small adversarial perturbations that models impulse noise in raw data, which relates to \emph{herringbone artifacts}, and show the applicability of our approach in this setting. Finally, we show our mitigation approach remains effective in two \emph{realistic} extension scenarios: a blind setup, where the attack strength or algorithm is not known to the user; and an adaptive attack setup, where the attacker has full knowledge of the defense strategy.

http://arxiv.org/abs/2509.22836
Seeing Isn't Believing: Context-Aware Adversarial Patch Synthesis via Conditional GAN. (96%)
Roie Kazoom; Alon Goldberg; Hodaya Cohen; Ofer Hadar
Adversarial patch attacks pose a severe threat to deep neural networks, yet most existing approaches rely on unrealistic white-box assumptions, untargeted objectives, or produce visually conspicuous patches that limit real-world applicability. In this work, we introduce a novel framework for fully controllable adversarial patch generation, where the attacker can freely choose both the input image x and the target class y target, thereby dictating the exact misclassification outcome. Our method combines a generative U-Net design with Grad-CAM-guided patch placement, enabling semantic-aware localization that maximizes attack effectiveness while preserving visual realism. Extensive experiments across convolutional networks (DenseNet-121, ResNet-50) and vision transformers (ViT-B/16, Swin-B/16, among others) demonstrate that our approach achieves state-of-the-art performance across all settings, with attack success rates (ASR) and target-class success (TCS) consistently exceeding 99%.
  Importantly, we show that our method not only outperforms prior white-box attacks and untargeted baselines, but also surpasses existing non-realistic approaches that produce detectable artifacts. By simultaneously ensuring realism, targeted control, and black-box applicability-the three most challenging dimensions of patch-based attacks-our framework establishes a new benchmark for adversarial robustness research, bridging the gap between theoretical attack strength and practical stealthiness.

http://arxiv.org/abs/2505.19616
Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models. (83%)
Rui Cai; Bangzheng Li; Xiaofei Wen; Muhao Chen; Zhe Zhao
Multimodal Large Language Models have demonstrated impressive capabilities across tasks, yet they often exhibit difficulty in distinguishing task-relevant from irrelevant signals -- particularly in tasks like Visual Question Answering -- which can lead to susceptibility to misleading or spurious inputs. We refer to this broader limitation as the Cross-Modality Competency Problem -- the model's inability to fairly evaluate all modalities. This vulnerability becomes more evident in modality-specific tasks -- such as image classification or pure text question answering -- where models are expected to rely solely on one modality. In such tasks, spurious information from irrelevant modalities often leads to significant performance degradation. We refer to this failure as Modality Interference, which serves as a concrete and measurable instance of the cross-modality competency problem, and we further design a perturbation-based causal diagnostic experiment to verify and quantify this problem. To mitigate modality interference, we propose a novel framework to finetune MLLMs, including perturbation-based data augmentations with both heuristic perturbations and adversarial perturbations, and a consistency regularization strategy applying on model outputs with original and perturbed inputs. Experiments on multiple benchmark datasets (image-heavy, text-heavy and multimodal tasks) and multiple model families with different scales demonstrate significant improvements in robustness and cross-modality competency, indicating our method's effectiveness in boosting unimodal reasoning ability while enhancing performance on multimodal tasks.

http://arxiv.org/abs/2509.22830
ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents. (81%)
Hwan Chang; Yonghyun Jun; Hwanhee Lee
The growing deployment of large language model (LLM) based agents that interact with external environments has created new attack surfaces for adversarial manipulation. One major threat is indirect prompt injection, where attackers embed malicious instructions in external environment output, causing agents to interpret and execute them as if they were legitimate prompts. While previous research has focused primarily on plain-text injection attacks, we find a significant yet underexplored vulnerability: LLMs' dependence on structured chat templates and their susceptibility to contextual manipulation through persuasive multi-turn dialogues. To this end, we introduce ChatInject, an attack that formats malicious payloads to mimic native chat templates, thereby exploiting the model's inherent instruction-following tendencies. Building on this foundation, we develop a persuasion-driven Multi-turn variant that primes the agent across conversational turns to accept and execute otherwise suspicious actions. Through comprehensive experiments across frontier LLMs, we demonstrate three critical findings: (1) ChatInject achieves significantly higher average attack success rates than traditional prompt injection methods, improving from 5.18% to 32.05% on AgentDojo and from 15.13% to 45.90% on InjecAgent, with multi-turn dialogues showing particularly strong performance at average 52.33% success rate on InjecAgent, (2) chat-template-based payloads demonstrate strong transferability across models and remain effective even against closed-source LLMs, despite their unknown template structures, and (3) existing prompt-based defenses are largely ineffective against this attack approach, especially against Multi-turn variants. These findings highlight vulnerabilities in current agent systems.

http://arxiv.org/abs/2509.22755
Concept activation vectors: a unifying view and adversarial attacks. (74%)
Ekkehard Schnoor; Malik Tiomoko; Jawher Said; Alex Jung; Wojciech Samek
Concept Activation Vectors (CAVs) are a tool from explainable AI, offering a promising approach for understanding how human-understandable concepts are encoded in a model's latent spaces. They are computed from hidden-layer activations of inputs belonging either to a concept class or to non-concept examples. Adopting a probabilistic perspective, the distribution of the (non-)concept inputs induces a distribution over the CAV, making it a random vector in the latent space. This enables us to derive mean and covariance for different types of CAVs, leading to a unified theoretical view. This probabilistic perspective also reveals a potential vulnerability: CAVs can strongly depend on the rather arbitrary non-concept distribution, a factor largely overlooked in prior work. We illustrate this with a simple yet effective adversarial attack, underscoring the need for a more systematic study.

http://arxiv.org/abs/2509.22147
Mixture of Detectors: A Compact View of Machine-Generated Text Detection. (56%)
Sai Teja Lekkala; Yadagiri Annepaka; Arun Kumar Challa; Samatha Reddy Machireddy; Partha Pakray; Chukhu Chunka
Large Language Models (LLMs) are gearing up to surpass human creativity. The veracity of the statement needs careful consideration. In recent developments, critical questions arise regarding the authenticity of human work and the preservation of their creativity and innovative abilities. This paper investigates such issues. This paper addresses machine-generated text detection across several scenarios, including document-level binary and multiclass classification or generator attribution, sentence-level segmentation to differentiate between human-AI collaborative text, and adversarial attacks aimed at reducing the detectability of machine-generated text. We introduce a new work called BMAS English: an English language dataset for binary classification of human and machine text, for multiclass classification, which not only identifies machine-generated text but can also try to determine its generator, and Adversarial attack addressing where it is a common act for the mitigation of detection, and Sentence-level segmentation, for predicting the boundaries between human and machine-generated text. We believe that this paper will address previous work in Machine-Generated Text Detection (MGTD) in a more meaningful way.

http://arxiv.org/abs/2504.04893
SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models. (56%)
Justus Westerhoff; Erblina Purelku; Jakob Hackstein; Jonas Loos; Leo Pinetzki; Erik Rodner; Lorenz Hufe
Typographic attacks exploit the interplay between text and visual content in multimodal foundation models, causing misclassifications when misleading text is embedded within images. Existing datasets are limited in size and diversity, making it difficult to study such vulnerabilities. In this paper, we introduce SCAM, the largest and most diverse dataset of real-world typographic attack images to date, containing 1162 images across hundreds of object categories and attack words. Through extensive benchmarking of Vision-Language Models on SCAM, we demonstrate that typographic attacks significantly degrade performance, and identify that training data and model architecture influence the susceptibility to these attacks. Our findings indicate that typographic attacks remain effective against state-of-the-art Large Vision-Language Models, especially those employing vision encoders inherently vulnerable to such attacks. However, employing larger Large Language Model backbones reduces this vulnerability while simultaneously enhancing typographic understanding. Additionally, we demonstrate that synthetic attacks closely resemble real-world (handwritten) attacks, validating their use in research. Our work provides a comprehensive resource and empirical insights to facilitate future research toward robust and trustworthy multimodal AI systems. Finally, we publicly release the datasets introduced in this paper, along with the code for evaluations under www.bliss.berlin/research/scam.

http://arxiv.org/abs/2509.21879
Zubov-Net: Adaptive Stability for Neural ODEs Reconciling Accuracy with Robustness. (47%)
Chaoyang Luo; Yan Zou; Nanjing Huang
Despite neural ordinary differential equations (Neural ODEs) exhibiting intrinsic robustness under input perturbations due to their dynamical systems nature, recent approaches often involve imposing Lyapunov-based stability conditions to provide formal robustness guarantees. However, a fundamental challenge remains: the tension between robustness and accuracy, primarily stemming from the difficulty in imposing appropriate stability conditions. To address this, we propose an adaptive stable learning framework named Zubov-Net, which innovatively reformulates Zubov's equation into a consistency characterization between regions of attraction (RoAs) and prescribed RoAs (PRoAs). Building on this consistency, we introduce a new paradigm for actively controlling the geometry of RoAs by directly optimizing PRoAs to reconcile accuracy and robustness. Our approach is realized through tripartite losses (consistency, classification, and separation losses) and a parallel boundary sampling algorithm that co-optimizes the Neural ODE and the Lyapunov function. To enhance the discriminativity of Lyapunov functions, we design an input-attention-based convex neural network via a softmax attention mechanism that focuses on equilibrium-relevant features and also serves as weight normalization to maintain training stability in deep architectures. Theoretically, we prove that minimizing the tripartite loss guarantees consistent alignment of PRoAs-RoAs, trajectory stability, and non-overlapping PRoAs. Moreover, we establish stochastic convex separability with tighter probability bounds and fewer dimensionality requirements to justify the convex design in Lyapunov functions. Experimentally, Zubov-Net maintains high classification accuracy while significantly improving robustness against various stochastic noises and adversarial attacks.

http://arxiv.org/abs/2507.08794
One Token to Fool LLM-as-a-Judge. (47%)
Yulai Zhao; Haolin Liu; Dian Yu; Sunyuan Kung; Meijia Chen; Haitao Mi; Dong Yu
Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models, particularly in reference-based settings like Reinforcement Learning with Verifiable Rewards (RLVR). However, we uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking. We find that superficial inputs, which we term ''master keys'' such as non-word symbols (e.g., '':'' or ''.'') or generic reasoning openers (e.g., ''Thought process:'' or ''Let's solve this problem step by step.''), can consistently elicit false positive rewards without any substantive reasoning. Our systematic evaluation demonstrates this is a widespread failure affecting a diverse range of models, including leading proprietary systems such as GPT-o1 and Claude-4. These results challenge the assumed robustness of LLM judges and pose a significant threat to their reliability. To address this, we propose a simple yet effective data augmentation strategy using truncated model outputs as adversarial negative examples. The resulting Master Reward Models (Master-RMs) demonstrate state-of-the-art robustness against these ''master key'' attacks while maintaining high performance in standard evaluation settings. We supplement these findings with a comprehensive analysis of the vulnerability across model scales, prompt variations, and common inference-time strategies, offering insights to guide future research on robust LLM evaluation. We release our robust, general-domain reward models and the synthetic training data at https://huggingface.co/sarosavo/Master-RM and https://huggingface.co/datasets/sarosavo/Master-RM.

http://arxiv.org/abs/2509.22873
AntiFLipper: A Secure and Efficient Defense Against Label-Flipping Attacks in Federated Learning. (41%)
Aashnan Rahman; Abid Hasan; Sherajul Arifin; Faisal Haque Bappy; Tahrim Hossain; Tariqul Islam; Abu Raihan Mostofa Kamal; Md. Azam Hossain
Federated learning (FL) enables privacy-preserving model training by keeping data decentralized. However, it remains vulnerable to label-flipping attacks, where malicious clients manipulate labels to poison the global model. Despite their simplicity, these attacks can severely degrade model performance, and defending against them remains challenging. We introduce AntiFLipper, a novel and computationally efficient defense against multi-class label-flipping attacks in FL. Unlike existing methods that ensure security at the cost of high computational overhead, AntiFLipper employs a novel client-side detection strategy, significantly reducing the central server's burden during aggregation. Comprehensive empirical evaluations across multiple datasets under different distributions demonstrate that AntiFLipper achieves accuracy comparable to state-of-the-art defenses while requiring substantially fewer computational resources in server side. By balancing security and efficiency, AntiFLipper addresses a critical gap in existing defenses, making it particularly suitable for resource-constrained FL deployments where both model integrity and operational efficiency are essential.

http://arxiv.org/abs/2509.22292
Jailbreaking on Text-to-Video Models via Scene Splitting Strategy. (31%)
Wonjun Lee; Haon Park; Doehyeon Lee; Bumsub Ham; Suhyun Kim
Along with the rapid advancement of numerous Text-to-Video (T2V) models, growing concerns have emerged regarding their safety risks. While recent studies have explored vulnerabilities in models like LLMs, VLMs, and Text-to-Image (T2I) models through jailbreak attacks, T2V models remain largely unexplored, leaving a significant safety gap. To address this gap, we introduce SceneSplit, a novel black-box jailbreak method that works by fragmenting a harmful narrative into multiple scenes, each individually benign. This approach manipulates the generative output space, the abstract set of all potential video outputs for a given prompt, using the combination of scenes as a powerful constraint to guide the final outcome. While each scene individually corresponds to a wide and safe space where most outcomes are benign, their sequential combination collectively restricts this space, narrowing it to an unsafe region and significantly increasing the likelihood of generating a harmful video. This core mechanism is further enhanced through iterative scene manipulation, which bypasses the safety filter within this constrained unsafe region. Additionally, a strategy library that reuses successful attack patterns further improves the attack's overall effectiveness and robustness. To validate our method, we evaluate SceneSplit across 11 safety categories on T2V models. Our results show that it achieves a high average Attack Success Rate (ASR) of 77.2% on Luma Ray2, 84.1% on Hailuo, and 78.2% on Veo2, significantly outperforming the existing baseline. Through this work, we demonstrate that current T2V safety mechanisms are vulnerable to attacks that exploit narrative structure, providing new insights for understanding and improving the safety of T2V models.

http://arxiv.org/abs/2507.18053
Resource Consumption Red-Teaming for Large Vision-Language Models. (15%)
Haoran Gao; Yuanhe Zhang; Zhenhong Zhou; Lei Jiang; Fanyu Meng; Yujia Xiao; Li Sun; Kun Wang; Yang Liu; Junlan Feng
Resource Consumption Attacks (RCAs) have emerged as a significant threat to the deployment of Large Language Models (LLMs). With the integration of vision modalities, additional attack vectors exacerbate the risk of RCAs in large vision-language models (LVLMs). However, existing red-teaming studies have mainly overlooked visual inputs as a potential attack surface, resulting in insufficient mitigation strategies against RCAs in LVLMs. To address this gap, we propose RECITE ($\textbf{Re}$source $\textbf{C}$onsumpt$\textbf{i}$on Red-$\textbf{Te}$aming for LVLMs), the first approach for exploiting visual modalities to trigger unbounded RCAs red-teaming. First, we present $\textit{Vision Guided Optimization}$, a fine-grained pixel-level optimization to obtain \textit{Output Recall Objective} adversarial perturbations, which can induce repeating output. Then, we inject the perturbations into visual inputs, triggering unbounded generations to achieve the goal of RCAs. Empirical results demonstrate that RECITE increases service response latency by over 26 $\uparrow$, resulting in an additional 20\% increase in GPU utilization and memory consumption. Our study reveals security vulnerabilities in LVLMs and establishes a red-teaming framework that can facilitate the development of future defenses against RCAs.

http://arxiv.org/abs/2509.21843
SBFA: Single Sneaky Bit Flip Attack to Break Large Language Models. (13%)
Jingkai Guo; Chaitali Chakrabarti; Deliang Fan
Model integrity of Large language models (LLMs) has become a pressing security concern with their massive online deployment. Prior Bit-Flip Attacks (BFAs) -- a class of popular AI weight memory fault-injection techniques -- can severely compromise Deep Neural Networks (DNNs): as few as tens of bit flips can degrade accuracy toward random guessing. Recent studies extend BFAs to LLMs and reveal that, despite the intuition of better robustness from modularity and redundancy, only a handful of adversarial bit flips can also cause LLMs' catastrophic accuracy degradation. However, existing BFA methods typically focus on either integer or floating-point models separately, limiting attack flexibility. Moreover, in floating-point models, random bit flips often cause perturbed parameters to extreme values (e.g., flipping in exponent bit), making it not stealthy and leading to numerical runtime error (e.g., invalid tensor values (NaN/Inf)). In this work, for the first time, we propose SBFA (Sneaky Bit-Flip Attack), which collapses LLM performance with only one single bit flip while keeping perturbed values within benign layer-wise weight distribution. It is achieved through iterative searching and ranking through our defined parameter sensitivity metric, ImpactScore, which combines gradient sensitivity and perturbation range constrained by the benign layer-wise weight distribution. A novel lightweight SKIP searching algorithm is also proposed to greatly reduce searching complexity, which leads to successful SBFA searching taking only tens of minutes for SOTA LLMs. Across Qwen, LLaMA, and Gemma models, with only one single bit flip, SBFA successfully degrades accuracy to below random levels on MMLU and SST-2 in both BF16 and INT8 data formats. Remarkably, flipping a single bit out of billions of parameters reveals a severe security concern of SOTA LLM models.

http://arxiv.org/abs/2509.22745
Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignment. (12%)
Jaehan Kim; Minkyoo Song; Seungwon Shin; Sooel Son
Recent large language models (LLMs) have increasingly adopted the Mixture-of-Experts (MoE) architecture for efficiency. MoE-based LLMs heavily depend on a superficial safety mechanism in which harmful inputs are routed safety-critical experts. However, our analysis reveals that routing decisions for harmful inputs drift significantly after fine-tuning, exposing a critical vulnerability to harmful fine-tuning (HFT) attacks. Existing defenses, primarily designed for monolithic LLMs, are less effective for MoE LLMs as they fail to prevent drift in harmful input routing. To address this limitation, we propose SafeMoE, a safe fine-tuning method tailored to MoE LLMs. SafeMoE directly mitigates routing drift by penalizing the gap between the routing weights of a fine-tuned model and those of the initial safety-aligned model, thereby preserving the safety-aligned routing of harmful inputs to safety-critical experts. Experiments on open-source MoE LLMs ranging from 7B to 141B parameters demonstrate that SafeMoE effectively mitigates HFT attacks, reducing the harmfulness score of OLMoE from 62.0 to 5.0, for example, while maintaining task utility within 1% degradation and incurring only 2% overhead. It significantly outperforms state-of-the-art defense methods for safeguarding LLM fine-tuning and remains effective in recent large-scale MoE LLMs such as gpt-oss and Llama 4. Our implementation is available at https://anonymous.4open.science/r/SafeMoE.

http://arxiv.org/abs/2509.22472
Evaluating the Limits of Large Language Models in Multilingual Legal Reasoning. (9%)
Antreas Ioannou; Andreas Shiamishis; Nora Hollenstein; Nezihe Merve Gürel
In an era dominated by Large Language Models (LLMs), understanding their capabilities and limitations, especially in high-stakes fields like law, is crucial. While LLMs such as Meta's LLaMA, OpenAI's ChatGPT, Google's Gemini, DeepSeek, and other emerging models are increasingly integrated into legal workflows, their performance in multilingual, jurisdictionally diverse, and adversarial contexts remains insufficiently explored. This work evaluates LLaMA and Gemini on multilingual legal and non-legal benchmarks, and assesses their adversarial robustness in legal tasks through character and word-level perturbations. We use an LLM-as-a-Judge approach for human-aligned evaluation. We moreover present an open-source, modular evaluation pipeline designed to support multilingual, task-diverse benchmarking of any combination of LLMs and datasets, with a particular focus on legal tasks, including classification, summarization, open questions, and general reasoning. Our findings confirm that legal tasks pose significant challenges for LLMs with accuracies often below 50% on legal reasoning benchmarks such as LEXam, compared to over 70% on general-purpose tasks like XNLI. In addition, while English generally yields more stable results, it does not always lead to higher accuracy. Prompt sensitivity and adversarial vulnerability is also shown to persist across languages. Finally, a correlation is found between the performance of a language and its syntactic similarity to English. We also observe that LLaMA is weaker than Gemini, with the latter showing an average advantage of about 24 percentage points across the same task. Despite improvements in newer LLMs, challenges remain in deploying them reliably for critical, multilingual legal applications.

http://arxiv.org/abs/2509.21761
Backdoor Attribution: Elucidating and Controlling Backdoor in Language Models. (3%)
Miao Yu; Zhenhong Zhou; Moayad Aloqaily; Kun Wang; Biwei Huang; Stephen Wang; Yueming Jin; Qingsong Wen
Fine-tuned Large Language Models (LLMs) are vulnerable to backdoor attacks through data poisoning, yet the internal mechanisms governing these attacks remain a black box. Previous research on interpretability for LLM safety tends to focus on alignment, jailbreak, and hallucination, but overlooks backdoor mechanisms, making it difficult to understand and fully eliminate the backdoor threat. In this paper, aiming to bridge this gap, we explore the interpretable mechanisms of LLM backdoors through Backdoor Attribution (BkdAttr), a tripartite causal analysis framework. We first introduce the Backdoor Probe that proves the existence of learnable backdoor features encoded within the representations. Building on this insight, we further develop Backdoor Attention Head Attribution (BAHA), efficiently pinpointing the specific attention heads responsible for processing these features. Our primary experiments reveals these heads are relatively sparse; ablating a minimal \textbf{$\sim$ 3%} of total heads is sufficient to reduce the Attack Success Rate (ASR) by \textbf{over 90%}. More importantly, we further employ these findings to construct the Backdoor Vector derived from these attributed heads as a master controller for the backdoor. Through only \textbf{1-point} intervention on \textbf{single} representation, the vector can either boost ASR up to \textbf{$\sim$ 100% ($\uparrow$)} on clean inputs, or completely neutralize backdoor, suppressing ASR down to \textbf{$\sim$ 0% ($\downarrow$)} on triggered inputs. In conclusion, our work pioneers the exploration of mechanistic interpretability in LLM backdoors, demonstrating a powerful method for backdoor control and revealing actionable insights for the community.

http://arxiv.org/abs/2509.22113
Countering adversarial evasion in regression analysis. (2%)
David Benfield; Phan Tu Vuong; Alain Zemkoho
Adversarial machine learning challenges the assumption that the underlying distribution remains consistent throughout the training and implementation of a prediction model. In particular, adversarial evasion considers scenarios where adversaries adapt their data to influence particular outcomes from established prediction models, such scenarios arise in applications such as spam email filtering, malware detection and fake-image generation, where security methods must be actively updated to keep up with the ever-improving generation of malicious data. Game theoretic models have been shown to be effective at modelling these scenarios and hence training resilient predictors against such adversaries. Recent advancements in the use of pessimistic bilevel optimsiation which remove assumptions about the convexity and uniqueness of the adversary's optimal strategy have proved to be particularly effective at mitigating threats to classifiers due to its ability to capture the antagonistic nature of the adversary. However, this formulation has not yet been adapted to regression scenarios. This article serves to propose a pessimistic bilevel optimisation program for regression scenarios which makes no assumptions on the convexity or uniqueness of the adversary's solutions.

http://arxiv.org/abs/2509.10655
Safety and Security Analysis of Large Language Models: Benchmarking Risk Profile and Harm Potential. (2%)
Charankumar Akiri; Harrison Simpson; Kshitiz Aryal; Aarav Khanna; Maanak Gupta
While the widespread deployment of Large Language Models (LLMs) holds great potential for society, their vulnerabilities to adversarial manipulation and exploitation can pose serious safety, security, and ethical risks. As new threats continue to emerge, it becomes critically necessary to assess the landscape of LLMs' safety and security against evolving adversarial prompt techniques. To understand the behavior of LLMs, this research provides an empirical analysis and risk profile of nine prominent LLMs, Claude Opus 4, DeepSeek V3 (both open-source and online), Gemini 2.5 Flash, GPT-4o, Grok 3, Llama 4 Scout, Mistral 7B, and Qwen 3 1.7B, against 24 different security and safety categories. These LLMs are evaluated on their ability to produce harmful responses for adversarially crafted prompts (dataset has been made public) for a broad range of safety and security topics, such as promotion of violent criminal behavior, promotion of non-violent criminal activity, societal harms related to safety, illegal sexual content, dangerous code generation, and cybersecurity threats beyond code. Our study introduces the Risk Severity Index (RSI), an agile and scalable evaluation score, to quantify and compare the security posture and creating a risk profile of LLMs. As the LLM development landscape progresses, the RSI is intended to be a valuable metric for comparing the risks of LLMs across evolving threats. This research finds widespread vulnerabilities in the safety filters of the LLMs tested and highlights the urgent need for stronger alignment, responsible deployment practices, and model governance, particularly for open-access and rapidly iterated models.

http://arxiv.org/abs/2509.21130
Sparse Representations Improve Adversarial Robustness of Neural Network Classifiers. (99%)
Killian Steunou; Sigurd Saue; Théo Druilhe
Deep neural networks perform remarkably well on image classification tasks but remain vulnerable to carefully crafted adversarial perturbations. This work revisits linear dimensionality reduction as a simple, data-adapted defense. We empirically compare standard Principal Component Analysis (PCA) with its sparse variant (SPCA) as front-end feature extractors for downstream classifiers, and we complement these experiments with a theoretical analysis. On the theory side, we derive exact robustness certificates for linear heads applied to SPCA features: for both $\ell_\infty$ and $\ell_2$ threat models (binary and multiclass), the certified radius grows as the dual norms of $W^\top u$ shrink, where $W$ is the projection and $u$ the head weights. We further show that for general (non-linear) heads, sparsity reduces operator-norm bounds through a Lipschitz composition argument, predicting lower input sensitivity. Empirically, with a small non-linear network after the projection, SPCA consistently degrades more gracefully than PCA under strong white-box and black-box attacks while maintaining competitive clean accuracy. Taken together, the theory identifies the mechanism (sparser projections reduce adversarial leverage) and the experiments verify that this benefit persists beyond the linear setting. Our code is available at https://github.com/killian31/SPCARobustness.

http://arxiv.org/abs/2509.21084
Vision Transformers: the threat of realistic adversarial patches. (99%)
Kasper Cools; Clara Maathuis; Oers Alexander M. van; Claudia S. Hübner; Nikos Deligiannis; Marijke Vandewal; Cubber Geert De
The increasing reliance on machine learning systems has made their security a critical concern. Evasion attacks enable adversaries to manipulate the decision-making processes of AI systems, potentially causing security breaches or misclassification of targets. Vision Transformers (ViTs) have gained significant traction in modern machine learning due to increased 1) performance compared to Convolutional Neural Networks (CNNs) and 2) robustness against adversarial perturbations. However, ViTs remain vulnerable to evasion attacks, particularly to adversarial patches, unique patterns designed to manipulate AI classification systems. These vulnerabilities are investigated by designing realistic adversarial patches to cause misclassification in person vs. non-person classification tasks using the Creases Transformation (CT) technique, which adds subtle geometric distortions similar to those occurring naturally when wearing clothing. This study investigates the transferability of adversarial attack techniques used in CNNs when applied to ViT classification models. Experimental evaluation across four fine-tuned ViT models on a binary person classification task reveals significant vulnerability variations: attack success rates ranged from 40.04% (google/vit-base-patch16-224-in21k) to 99.97% (facebook/dino-vitb16), with google/vit-base-patch16-224 achieving 66.40% and facebook/dinov3-vitb16 reaching 65.17%. These results confirm the cross-architectural transferability of adversarial patches from CNNs to ViTs, with pre-training dataset scale and methodology strongly influencing model resilience to adversarial attacks.

http://arxiv.org/abs/2509.21436
Position: Human Factors Reshape Adversarial Analysis in Human-AI Decision-Making Systems. (98%)
Shutong Fan; Lan Zhang; Xiaoyong Yuan
As Artificial Intelligence (AI) increasingly supports human decision-making, its vulnerability to adversarial attacks grows. However, the existing adversarial analysis predominantly focuses on fully autonomous AI systems, where decisions are executed without human intervention. This narrow focus overlooks the complexities of human-AI collaboration, where humans interpret, adjust, and act upon AI-generated decisions. Trust, expectations, and cognitive behaviors influence how humans interact with AI, creating dynamic feedback loops that adversaries can exploit. To strengthen the robustness of AI-assisted decision-making, adversarial analysis must account for the interplay between human factors and attack strategies.
  This position paper argues that human factors fundamentally reshape adversarial analysis and must be incorporated into evaluating robustness in human-AI decision-making systems. To fully explore human factors in adversarial analysis, we begin by investigating the role of human factors in human-AI collaboration through a comprehensive review. We then introduce a novel robustness analysis framework that (1) examines how human factors affect collaborative decision-making performance, (2) revisits and interprets existing adversarial attack strategies in the context of human-AI interaction, and (3) introduces a new timing-based adversarial attack as a case study, illustrating vulnerabilities emerging from sequential human actions. The experimental results reveal that attack timing uniquely impacts decision outcomes in human-AI collaboration. We hope this analysis inspires future research on adversarial robustness in human-AI systems, fostering interdisciplinary approaches that integrate AI security, human cognition, and decision-making dynamics.

http://arxiv.org/abs/2312.04692
Diffence: Fencing Membership Privacy With Diffusion Models. (97%)
Yuefeng Peng; Ali Naseh; Amir Houmansadr
Deep learning models, while achieving remarkable performances, are vulnerable to membership inference attacks (MIAs). Although various defenses have been proposed, there is still substantial room for improvement in the privacy-utility trade-off. In this work, we introduce a novel defense framework against MIAs by leveraging generative models. The key intuition of our defense is to remove the differences between member and non-member inputs, which is exploited by MIAs, by re-generating input samples before feeding them to the target model. Therefore, our defense, called DIFFENCE, works pre inference, which is unlike prior defenses that are either training-time or post-inference time.
  A unique feature of DIFFENCE is that it works on input samples only, without modifying the training or inference phase of the target model. Therefore, it can be cascaded with other defense mechanisms as we demonstrate through experiments. DIFFENCE is designed to preserve the model's prediction labels for each sample, thereby not affecting accuracy. Furthermore, we have empirically demonstrated it does not reduce the usefulness of confidence vectors. Through extensive experimentation, we show that DIFFENCE can serve as a robust plug-n-play defense mechanism, enhancing membership privacy without compromising model utility. For instance, DIFFENCE reduces MIA accuracy against an undefended model by 15.8\% and attack AUC by 14.0\% on average across three datasets, all without impacting model utility. By integrating DIFFENCE with prior defenses, we can achieve new state-of-the-art performances in the privacy-utility trade-off. For example, when combined with the state-of-the-art SELENA defense it reduces attack accuracy by 9.3\%, and attack AUC by 10.0\%. DIFFENCE achieves this by imposing a negligible computation overhead, adding only 57ms to the inference time per sample processed on average.

http://arxiv.org/abs/2509.20714
Cryptographic Backdoor for Neural Networks: Boon and Bane. (92%)
Anh Tu Ngo; Anupam Chattopadhyay; Subhamoy Maitra
In this paper we show that cryptographic backdoors in a neural network (NN) can be highly effective in two directions, namely mounting the attacks as well as in presenting the defenses as well. On the attack side, a carefully planted cryptographic backdoor enables powerful and invisible attack on the NN. Considering the defense, we present applications: first, a provably robust NN watermarking scheme; second, a protocol for guaranteeing user authentication; and third, a protocol for tracking unauthorized sharing of the NN intellectual property (IP). From a broader theoretical perspective, borrowing the ideas from Goldwasser et. al. [FOCS 2022], our main contribution is to show that all these instantiated practical protocol implementations are provably robust. The protocols for watermarking, authentication and IP tracking resist an adversary with black-box access to the NN, whereas the backdoor-enabled adversarial attack is impossible to prevent under the standard assumptions. While the theoretical tools used for our attack is mostly in line with the Goldwasser et. al. ideas, the proofs related to the defense need further studies. Finally, all these protocols are implemented on state-of-the-art NN architectures with empirical results corroborating the theoretical claims. Further, one can utilize post-quantum primitives for implementing the cryptographic backdoors, laying out foundations for quantum-era applications in machine learning (ML).

http://arxiv.org/abs/2501.14249
Humanity's Last Exam. (92%)
Long Phan; Alice Gatti; Ziwen Han; Nathaniel Li; Josephina Hu; Hugh Zhang; Chen Bo Calvin Zhang; Mohamed Shaaban; John Ling; Sean Shi; Michael Choi; Anish Agrawal; Arnav Chopra; Adam Khoja; Ryan Kim; Richard Ren; Jason Hausenloy; Oliver Zhang; Mantas Mazeika; Dmitry Dodonov; Tung Nguyen; Jaeho Lee; Daron Anderson; Mikhail Doroshenko; Alun Cennyth Stokes; Mobeen Mahmood; Oleksandr Pokutnyi; Oleg Iskra; Jessica P. Wang; John-Clark Levin; Mstyslav Kazakov; Fiona Feng; Steven Y. Feng; Haoran Zhao; Michael Yu; Varun Gangal; Chelsea Zou; Zihan Wang; Serguei Popov; Robert Gerbicz; Geoff Galgon; Johannes Schmitt; Will Yeadon; Yongki Lee; Scott Sauers; Alvaro Sanchez; Fabian Giska; Marc Roth; Søren Riis; Saiteja Utpala; Noah Burns; Gashaw M. Goshu; Mohinder Maheshbhai Naiya; Chidozie Agu; Zachary Giboney; Antrell Cheatom; Francesco Fournier-Facio; Sarah-Jane Crowson; Lennart Finke; Zerui Cheng; Jennifer Zampese; Ryan G. Hoerr; Mark Nandor; Hyunwoo Park; Tim Gehrunger; Jiaqi Cai; Ben McCarty; Alexis C Garretson; Edwin Taylor; Damien Sileo; Qiuyu Ren; Usman Qazi; Lianghui Li; Jungbae Nam; John B. Wydallis; Pavel Arkhipov; Jack Wei Lun Shi; Aras Bacho; Chris G. Willcocks; Hangrui Cao; Sumeet Motwani; Emily de Oliveira Santos; Johannes Veith; Edward Vendrow; Doru Cojoc; Kengo Zenitani; Joshua Robinson; Longke Tang; Yuqi Li; Joshua Vendrow; Natanael Wildner Fraga; Vladyslav Kuchkin; Andrey Pupasov Maksimov; Pierre Marion; Denis Efremov; Jayson Lynch; Kaiqu Liang; Aleksandar Mikov; Andrew Gritsevskiy; Julien Guillod; Gözdenur Demir; Dakotah Martinez; Ben Pageler; Kevin Zhou; Saeed Soori; Ori Press; Henry Tang; Paolo Rissone; Sean R. Green; Lina Brüssel; Moon Twayana; Aymeric Dieuleveut; Joseph Marvin Imperial; Ameya Prabhu; Jinzhou Yang; Nick Crispino; Arun Rao; Dimitri Zvonkine; Gabriel Loiseau; Mikhail Kalinin; Marco Lukas; Ciprian Manolescu; Nate Stambaugh; Subrata Mishra; Tad Hogg; Carlo Bosio; Brian P Coppola; Julian Salazar; Jaehyeok Jin; Rafael Sayous; Stefan Ivanov; Philippe Schwaller; Shaipranesh Senthilkuma; Andres M Bran; Andres Algaba; Kelsey Van den Houte; Der Sypt Lynn Van; Brecht Verbeken; David Noever; Alexei Kopylov; Benjamin Myklebust; Bikun Li; Lisa Schut; Evgenii Zheltonozhskii; Qiaochu Yuan; Derek Lim; Richard Stanley; Tong Yang; John Maar; Julian Wykowski; Martí Oller; Anmol Sahu; Cesare Giulio Ardito; Yuzheng Hu; Ariel Ghislain Kemogne Kamdoum; Alvin Jin; Tobias Garcia Vilchis; Yuexuan Zu; Martin Lackner; James Koppel; Gongbo Sun; Daniil S. Antonenko; Steffi Chern; Bingchen Zhao; Pierrot Arsene; Joseph M Cavanagh; Daofeng Li; Jiawei Shen; Donato Crisostomi; Wenjin Zhang; Ali Dehghan; Sergey Ivanov; David Perrella; Nurdin Kaparov; Allen Zang; Ilia Sucholutsky; Arina Kharlamova; Daniil Orel; Vladislav Poritski; Shalev Ben-David; Zachary Berger; Parker Whitfill; Michael Foster; Daniel Munro; Linh Ho; Shankar Sivarajan; Dan Bar Hava; Aleksey Kuchkin; David Holmes; Alexandra Rodriguez-Romero; Frank Sommerhage; Anji Zhang; Richard Moat; Keith Schneider; Zakayo Kazibwe; Don Clarke; Dae Hyun Kim; Felipe Meneguitti Dias; Sara Fish; Veit Elser; Tobias Kreiman; Victor Efren Guadarrama Vilchis; Immo Klose; Ujjwala Anantheswaran; Adam Zweiger; Kaivalya Rawal; Jeffery Li; Jeremy Nguyen; Nicolas Daans; Haline Heidinger; Maksim Radionov; Václav Rozhoň; Vincent Ginis; Christian Stump; Niv Cohen; Rafał Poświata; Josef Tkadlec; Alan Goldfarb; Chenguang Wang; Piotr Padlewski; Stanislaw Barzowski; Kyle Montgomery; Ryan Stendall; Jamie Tucker-Foltz; Jack Stade; T. Ryan Rogers; Tom Goertzen; Declan Grabb; Abhishek Shukla; Alan Givré; John Arnold Ambay; Archan Sen; Muhammad Fayez Aziz; Mark H Inlow; Hao He; Ling Zhang; Younesse Kaddar; Ivar Ängquist; Yanxu Chen; Harrison K Wang; Kalyan Ramakrishnan; Elliott Thornley; Antonio Terpin; Hailey Schoelkopf; Eric Zheng; Avishy Carmi; Ethan D. L. Brown; Kelin Zhu; Max Bartolo; Richard Wheeler; Martin Stehberger; Peter Bradshaw; JP Heimonen; Kaustubh Sridhar; Ido Akov; Jennifer Sandlin; Yury Makarychev; Joanna Tam; Hieu Hoang; David M. Cunningham; Vladimir Goryachev; Demosthenes Patramanis; Michael Krause; Andrew Redenti; David Aldous; Jesyin Lai; Shannon Coleman; Jiangnan Xu; Sangwon Lee; Ilias Magoulas; Sandy Zhao; Ning Tang; Michael K. Cohen; Orr Paradise; Jan Hendrik Kirchner; Maksym Ovchynnikov; Jason O. Matos; Adithya Shenoy; Michael Wang; Yuzhou Nie; Anna Sztyber-Betley; Paolo Faraboschi; Robin Riblet; Jonathan Crozier; Shiv Halasyamani; Shreyas Verma; Prashant Joshi; Eli Meril; Ziqiao Ma; Jérémy Andréoletti; Raghav Singhal; Jacob Platnick; Volodymyr Nevirkovets; Luke Basler; Alexander Ivanov; Seri Khoury; Nils Gustafsson; Marco Piccardo; Hamid Mostaghimi; Qijia Chen; Virendra Singh; Tran Quoc Khánh; Paul Rosu; Hannah Szlyk; Zachary Brown; Himanshu Narayan; Aline Menezes; Jonathan Roberts; William Alley; Kunyang Sun; Arkil Patel; Max Lamparth; Anka Reuel; Linwei Xin; Hanmeng Xu; Jacob Loader; Freddie Martin; Zixuan Wang; Andrea Achilleos; Thomas Preu; Tomek Korbak; Ida Bosio; Fereshteh Kazemi; Ziye Chen; Biró Bálint; Eve J. Y. Lo; Jiaqi Wang; Maria Inês S. Nunes; Jeremiah Milbauer; M Saiful Bari; Zihao Wang; Behzad Ansarinejad; Yewen Sun; Stephane Durand; Hossam Elgnainy; Guillaume Douville; Daniel Tordera; George Balabanian; Hew Wolff; Lynna Kvistad; Hsiaoyun Milliron; Ahmad Sakor; Murat Eron; Andrew Favre D. O.; Shailesh Shah; Xiaoxiang Zhou; Firuz Kamalov; Sherwin Abdoli; Tim Santens; Shaul Barkan; Allison Tee; Robin Zhang; Alessandro Tomasiello; Luca G. Bruno De; Shi-Zhuo Looi; Vinh-Kha Le; Noam Kolt; Jiayi Pan; Emma Rodman; Jacob Drori; Carl J Fossum; Niklas Muennighoff; Milind Jagota; Ronak Pradeep; Honglu Fan; Jonathan Eicher; Michael Chen; Kushal Thaman; William Merrill; Moritz Firsching; Carter Harris; Stefan Ciobâcă; Jason Gross; Rohan Pandey; Ilya Gusev; Adam Jones; Shashank Agnihotri; Pavel Zhelnov; Mohammadreza Mofayezi; Alexander Piperski; David K. Zhang; Kostiantyn Dobarskyi; Roman Leventov; Ignat Soroko; Joshua Duersch; Vage Taamazyan; Andrew Ho; Wenjie Ma; William Held; Ruicheng Xian; Armel Randy Zebaze; Mohanad Mohamed; Julian Noah Leser; Michelle X Yuan; Laila Yacar; Johannes Lengler; Katarzyna Olszewska; Fratta Claudio Di; Edson Oliveira; Joseph W. Jackson; Andy Zou; Muthu Chidambaram; Timothy Manik; Hector Haffenden; Dashiell Stander; Ali Dasouqi; Alexander Shen; Bita Golshani; David Stap; Egor Kretov; Mikalai Uzhou; Alina Borisovna Zhidkovskaya; Nick Winter; Miguel Orbegozo Rodriguez; Robert Lauff; Dustin Wehr; Colin Tang; Zaki Hossain; Shaun Phillips; Fortuna Samuele; Fredrik Ekström; Angela Hammon; Oam Patel; Faraz Farhidi; George Medley; Forough Mohammadzadeh; Madellene Peñaflor; Haile Kassahun; Alena Friedrich; Rayner Hernandez Perez; Daniel Pyda; Taom Sakal; Omkar Dhamane; Ali Khajegili Mirabadi; Eric Hallman; Kenchi Okutsu; Mike Battaglia; Mohammad Maghsoudimehrabani; Alon Amit; Dave Hulbert; Roberto Pereira; Simon Weber; Handoko; Anton Peristyy; Stephen Malina; Mustafa Mehkary; Rami Aly; Frank Reidegeld; Anna-Katharina Dick; Cary Friday; Mukhwinder Singh; Hassan Shapourian; Wanyoung Kim; Mariana Costa; Hubeyb Gurdogan; Harsh Kumar; Chiara Ceconello; Chao Zhuang; Haon Park; Micah Carroll; Andrew R. Tawfeek; Stefan Steinerberger; Daattavya Aggarwal; Michael Kirchhof; Linjie Dai; Evan Kim; Johan Ferret; Jainam Shah; Yuzhou Wang; Minghao Yan; Krzysztof Burdzy; Lixin Zhang; Antonio Franca; Diana T. Pham; Kang Yong Loh; Joshua Robinson; Abram Jackson; Paolo Giordano; Philipp Petersen; Adrian Cosma; Jesus Colino; Colin White; Jacob Votava; Vladimir Vinnikov; Ethan Delaney; Petr Spelda; Vit Stritecky; Syed M. Shahid; Jean-Christophe Mourrat; Lavr Vetoshkin; Koen Sponselee; Renas Bacho; Zheng-Xin Yong; la Rosa Florencia de; Nathan Cho; Xiuyu Li; Guillaume Malod; Orion Weller; Guglielmo Albani; Leon Lang; Julien Laurendeau; Dmitry Kazakov; Fatimah Adesanya; Julien Portier; Lawrence Hollom; Victor Souza; Yuchen Anna Zhou; Julien Degorre; Yiğit Yalın; Gbenga Daniel Obikoya; Rai; Filippo Bigi; M. C. Boscá; Oleg Shumar; Kaniuar Bacho; Gabriel Recchia; Mara Popescu; Nikita Shulga; Ngefor Mildred Tanwie; Thomas C. H. Lux; Ben Rank; Colin Ni; Matthew Brooks; Alesia Yakimchyk; Huanxu; Liu; Stefano Cavalleri; Olle Häggström; Emil Verkama; Joshua Newbould; Hans Gundlach; Leonor Brito-Santana; Brian Amaro; Vivek Vajipey; Rynaa Grover; Ting Wang; Yosi Kratish; Wen-Ding Li; Sivakanth Gopi; Andrea Caciolai; Witt Christian Schroeder de; Pablo Hernández-Cámara; Emanuele Rodolà; Jules Robins; Dominic Williamson; Vincent Cheng; Brad Raynor; Hao Qi; Ben Segev; Jingxuan Fan; Sarah Martinson; Erik Y. Wang; Kaylie Hausknecht; Michael P. Brenner; Mao Mao; Christoph Demian; Peyman Kassani; Xinyu Zhang; David Avagian; Eshawn Jessica Scipio; Alon Ragoler; Justin Tan; Blake Sims; Rebeka Plecnik; Aaron Kirtland; Omer Faruk Bodur; D. P. Shinde; Yan Carlos Leyva Labrador; Zahra Adoul; Mohamed Zekry; Ali Karakoc; Tania C. B. Santos; Samir Shamseldeen; Loukmane Karim; Anna Liakhovitskaia; Nate Resman; Nicholas Farina; Juan Carlos Gonzalez; Gabe Maayan; Earth Anderson; Rodrigo De Oliveira Pena; Elizabeth Kelley; Hodjat Mariji; Rasoul Pouriamanesh; Wentao Wu; Ross Finocchio; Ismail Alarab; Joshua Cole; Danyelle Ferreira; Bryan Johnson; Mohammad Safdari; Liangti Dai; Siriphan Arthornthurasuk; Isaac C. McAlister; Alejandro José Moyano; Alexey Pronin; Jing Fan; Angel Ramirez-Trinidad; Yana Malysheva; Daphiny Pottmaier; Omid Taheri; Stanley Stepanic; Samuel Perry; Luke Askew; Raúl Adrián Huerta Rodríguez; Ali M. R. Minissi; Ricardo Lorena; Krishnamurthy Iyer; Arshad Anil Fasiludeen; Ronald Clark; Josh Ducey; Matheus Piza; Maja Somrak; Eric Vergo; Juehang Qin; Benjámin Borbás; Eric Chu; Jack Lindsey; Antoine Jallon; I. M. J. McInnis; Evan Chen; Avi Semler; Luk Gloor; Tej Shah; Marc Carauleanu; Pascal Lauer; Tran Đuc Huy; Hossein Shahrtash; Emilien Duc; Lukas Lewark; Assaf Brown; Samuel Albanie; Brian Weber; Warren S. Vaz; Pierre Clavier; Yiyang Fan; Gabriel Poesia Reis e Silva; Long; Lian; Marcus Abramovitch; Xi Jiang; Sandra Mendoza; Murat Islam; Juan Gonzalez; Vasilios Mavroudis; Justin Xu; Pawan Kumar; Laxman Prasad Goswami; Daniel Bugas; Nasser Heydari; Ferenc Jeanplong; Thorben Jansen; Antonella Pinto; Archimedes Apronti; Abdallah Galal; Ng Ze-An; Ankit Singh; Tong Jiang; Joan of Arc Xavier; Kanu Priya Agarwal; Mohammed Berkani; Gang Zhang; Zhehang Du; Benedito Alves de Oliveira Junior; Dmitry Malishev; Nicolas Remy; Taylor D. Hartman; Tim Tarver; Stephen Mensah; Gautier Abou Loume; Wiktor Morak; Farzad Habibi; Sarah Hoback; Will Cai; Javier Gimenez; Roselynn Grace Montecillo; Jakub Łucki; Russell Campbell; Asankhaya Sharma; Khalida Meer; Shreen Gul; Daniel Espinosa Gonzalez; Xavier Alapont; Alex Hoover; Gunjan Chhablani; Freddie Vargus; Arunim Agarwal; Yibo Jiang; Deepakkumar Patil; David Outevsky; Kevin Joseph Scaria; Rajat Maheshwari; Abdelkader Dendane; Priti Shukla; Ashley Cartwright; Sergei Bogdanov; Niels Mündler; Sören Möller; Luca Arnaboldi; Kunvar Thaman; Muhammad Rehan Siddiqi; Prajvi Saxena; Himanshu Gupta; Tony Fruhauff; Glen Sherman; Mátyás Vincze; Siranut Usawasutsakorn; Dylan Ler; Anil Radhakrishnan; Innocent Enyekwe; Sk Md Salauddin; Jiang Muzhen; Aleksandr Maksapetyan; Vivien Rossbach; Chris Harjadi; Mohsen Bahaloohoreh; Claire Sparrow; Jasdeep Sidhu; Sam Ali; Song Bian; John Lai; Eric Singer; Justine Leon Uro; Greg Bateman; Mohamed Sayed; Ahmed Menshawy; Darling Duclosel; Dario Bezzi; Yashaswini Jain; Ashley Aaron; Murat Tiryakioglu; Sheeshram Siddh; Keith Krenek; Imad Ali Shah; Jun Jin; Scott Creighton; Denis Peskoff; Zienab EL-Wasif; Ragavendran V P; Michael Richmond; Joseph McGowan; Tejal Patwardhan; Hao-Yu Sun; Ting Sun; Nikola Zubić; Samuele Sala; Stephen Ebert; Jean Kaddour; Manuel Schottdorf; Dianzhuo Wang; Gerol Petruzella; Alex Meiburg; Tilen Medved; Ali ElSheikh; S Ashwin Hebbar; Lorenzo Vaquero; Xianjun Yang; Jason Poulos; Vilém Zouhar; Sergey Bogdanik; Mingfang Zhang; Jorge Sanz-Ros; David Anugraha; Yinwei Dai; Anh N. Nhu; Xue Wang; Ali Anil Demircali; Zhibai Jia; Yuyin Zhou; Juncheng Wu; Mike He; Nitin Chandok; Aarush Sinha; Gaoxiang Luo; Long Le; Mickaël Noyé; Michał Perełkiewicz; Ioannis Pantidis; Tianbo Qi; Soham Sachin Purohit; Letitia Parcalabescu; Thai-Hoa Nguyen; Genta Indra Winata; Edoardo M. Ponti; Hanchen Li; Kaustubh Dhole; Jongee Park; Dario Abbondanza; Yuanli Wang; Anupam Nayak; Diogo M. Caetano; Antonio A. W. L. Wong; Rio-Chanona Maria del; Dániel Kondor; Pieter Francois; Ed Chalstrey; Jakob Zsambok; Dan Hoyer; Jenny Reddish; Jakob Hauser; Francisco-Javier Rodrigo-Ginés; Suchandra Datta; Maxwell Shepherd; Thom Kamphuis; Qizheng Zhang; Hyunjun Kim; Ruiji Sun; Jianzhu Yao; Franck Dernoncourt; Satyapriya Krishna; Sina Rismanchian; Bonan Pu; Francesco Pinto; Yingheng Wang; Kumar Shridhar; Kalon J. Overholt; Glib Briia; Hieu Nguyen; David; Soler Bartomeu; Tony CY Pang; Adam Wecker; Yifan Xiong; Fanfei Li; Lukas S. Huber; Joshua Jaeger; Maddalena Romano De; Xing Han Lù; Yuhui Zhang; Claas Beger; Patrick Tser Jern Kon; Sean Li; Vivek Sanker; Ming Yin; Yihao Liang; Xinlu Zhang; Ankit Agrawal; Li S. Yifei; Zechen Zhang; Mu Cai; Yasin Sonmez; Costin Cozianu; Changhao Li; Alex Slen; Shoubin Yu; Hyun Kyu Park; Gabriele Sarti; Marcin Briański; Alessandro Stolfo; Truong An Nguyen; Mike Zhang; Yotam Perlitz; Jose Hernandez-Orallo; Runjia Li; Amin Shabani; Felix Juefei-Xu; Shikhar Dhingra; Orr Zohar; My Chiffon Nguyen; Alexander Pondaven; Abdurrahim Yilmaz; Xuandong Zhao; Chuanyang Jin; Muyan Jiang; Stefan Todoran; Xinyao Han; Jules Kreuer; Brian Rabern; Anna Plassart; Martino Maggetti; Luther Yap; Robert Geirhos; Jonathon Kean; Dingsu Wang; Sina Mollaei; Chenkai Sun; Yifan Yin; Shiqi Wang; Rui Li; Yaowen Chang; Anjiang Wei; Alice Bizeul; Xiaohan Wang; Alexandre Oliveira Arrais; Kushin Mukherjee; Jorge Chamorro-Padial; Jiachen Liu; Xingyu Qu; Junyi Guan; Adam Bouyamourn; Shuyu Wu; Martyna Plomecka; Junda Chen; Mengze Tang; Jiaqi Deng; Shreyas Subramanian; Haocheng Xi; Haoxuan Chen; Weizhi Zhang; Yinuo Ren; Haoqin Tu; Sejong Kim; Yushun Chen; Sara Vera Marjanović; Junwoo Ha; Grzegorz Luczyna; Jeff J. Ma; Zewen Shen; Dawn Song; Cedegao E. Zhang; Zhun Wang; Gaël Gendron; Yunze Xiao; Leo Smucker; Erica Weng; Kwok Hao Lee; Zhe Ye; Stefano Ermon; Ignacio D. Lopez-Miguel; Theo Knights; Anthony Gitter; Namkyu Park; Boyi Wei; Hongzheng Chen; Kunal Pai; Ahmed Elkhanany; Han Lin; Philipp D. Siedler; Jichao Fang; Ritwik Mishra; Károly Zsolnai-Fehér; Xilin Jiang; Shadab Khan; Jun Yuan; Rishab Kumar Jain; Xi Lin; Mike Peterson; Zhe Wang; Aditya Malusare; Maosen Tang; Isha Gupta; Ivan Fosin; Timothy Kang; Barbara Dworakowska; Kazuki Matsumoto; Guangyao Zheng; Gerben Sewuster; Jorge Pretel Villanueva; Ivan Rannev; Igor Chernyavsky; Jiale Chen; Deepayan Banik; Ben Racz; Wenchao Dong; Jianxin Wang; Laila Bashmal; Duarte V. Gonçalves; Wei Hu; Kaushik Bar; Ondrej Bohdal; Atharv Singh Patlan; Shehzaad Dhuliawala; Caroline Geirhos; Julien Wist; Yuval Kansal; Bingsen Chen; Kutay Tire; Atak Talay Yücel; Brandon Christof; Veerupaksh Singla; Zijian Song; Sanxing Chen; Jiaxin Ge; Kaustubh Ponkshe; Isaac Park; Tianneng Shi; Martin Q. Ma; Joshua Mak; Sherwin Lai; Antoine Moulin; Zhuo Cheng; Zhanda Zhu; Ziyi Zhang; Vaidehi Patil; Ketan Jha; Qiutong Men; Jiaxuan Wu; Tianchi Zhang; Bruno Hebling Vieira; Alham Fikri Aji; Jae-Won Chung; Mohammed Mahfoud; Ha Thi Hoang; Marc Sperzel; Wei Hao; Kristof Meding; Sihan Xu; Vassilis Kostakos; Davide Manini; Yueying Liu; Christopher Toukmaji; Jay Paek; Eunmi Yu; Arif Engin Demircali; Zhiyi Sun; Ivan Dewerpe; Hongsen Qin; Roman Pflugfelder; James Bailey; Johnathan Morris; Ville Heilala; Sybille Rosset; Zishun Yu; Peter E. Chen; Woongyeong Yeo; Eeshaan Jain; Ryan Yang; Sreekar Chigurupati; Julia Chernyavsky; Sai Prajwal Reddy; Subhashini Venugopalan; Hunar Batra; Core Francisco Park; Hieu Tran; Guilherme Maximiano; Genghan Zhang; Yizhuo Liang; Hu Shiyu; Rongwu Xu; Rui Pan; Siddharth Suresh; Ziqi Liu; Samaksh Gulati; Songyang Zhang; Peter Turchin; Christopher W. Bartlett; Christopher R. Scotese; Phuong M. Cao; Ben Wu; Jacek Karwowski; Davide Scaramuzza; Aakaash Nattanmai; Gordon McKellips; Anish Cheraku; Asim Suhail; Ethan Luo; Marvin Deng; Jason Luo; Ashley Zhang; Kavin Jindel; Jay Paek; Kasper Halevy; Allen Baranov; Michael Liu; Advaith Avadhanam; David Zhang; Vincent Cheng; Brad Ma; Evan Fu; Liam Do; Joshua Lass; Hubert Yang; Surya Sunkari; Vishruth Bharath; Violet Ai; James Leung; Rishit Agrawal; Alan Zhou; Kevin Chen; Tejas Kalpathi; Ziqi Xu; Gavin Wang; Tyler Xiao; Erik Maung; Sam Lee; Ryan Yang; Roy Yue; Ben Zhao; Julia Yoon; Sunny Sun; Aryan Singh; Ethan Luo; Clark Peng; Tyler Osbey; Taozhi Wang; Daryl Echeazu; Hubert Yang; Timothy Wu; Spandan Patel; Vidhi Kulkarni; Vijaykaarti Sundarapandiyan; Ashley Zhang; Andrew Le; Zafir Nasim; Srikar Yalam; Ritesh Kasamsetty; Soham Samal; Hubert Yang; David Sun; Nihar Shah; Abhijeet Saha; Alex Zhang; Leon Nguyen; Laasya Nagumalli; Kaixin Wang; Alan Zhou; Aidan Wu; Jason Luo; Anwith Telluri; Summer Yue; Alexandr Wang; Dan Hendrycks
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

http://arxiv.org/abs/2509.20691
RedHerring Attack: Testing the Reliability of Attack Detection. (78%)
Jonathan Rusert
In response to adversarial text attacks, attack detection models have been proposed and shown to successfully identify text modified by adversaries. Attack detection models can be leveraged to provide an additional check for NLP models and give signals for human input. However, the reliability of these models has not yet been thoroughly explored. Thus, we propose and test a novel attack setting and attack, RedHerring. RedHerring aims to make attack detection models unreliable by modifying a text to cause the detection model to predict an attack, while keeping the classifier correct. This creates a tension between the classifier and detector. If a human sees that the detector is giving an ``incorrect'' prediction, but the classifier a correct one, then the human will see the detector as unreliable. We test this novel threat model on 4 datasets against 3 detectors defending 4 classifiers. We find that RedHerring is able to drop detection accuracy between 20 - 71 points, while maintaining (or improving) classifier accuracy. As an initial defense, we propose a simple confidence check which requires no retraining of the classifier or detector and increases detection accuracy greatly. This novel threat model offers new insights into how adversaries may target detection models.

http://arxiv.org/abs/2509.20792
DAC-LoRA: Dynamic Adversarial Curriculum for Efficient and Robust Few-Shot Adaptation. (67%)
Ved Umrajkar
Vision-Language Models (VLMs) are foundational to critical applications like autonomous driving, medical diagnosis, and content moderation. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA enable their efficient adaptation to specialized tasks, these models remain vulnerable to adversarial attacks that can compromise safety-critical decisions. CLIP, the backbone for numerous downstream VLMs, is a high-value target whose vulnerabilities can cascade across the multimodal AI ecosystem. We propose Dynamic Adversarial Curriculum DAC-LoRA, a novel framework that integrates adversarial training into PEFT. The core principle of our method i.e. an intelligent curriculum of progressively challenging attack, is general and can potentially be applied to any iterative attack method. Guided by the First-Order Stationary Condition (FOSC) and a TRADES-inspired loss, DAC-LoRA achieves substantial improvements in adversarial robustness without significantly compromising clean accuracy. Our work presents an effective, lightweight, and broadly applicable method to demonstrate that the DAC-LoRA framework can be easily integrated into a standard PEFT pipeline to significantly enhance robustness.

http://arxiv.org/abs/2509.22732
Bidirectional Intention Inference Enhances LLMs' Defense Against Multi-Turn Jailbreak Attacks. (67%)
Haibo Tong; Dongcheng Zhao; Guobin Shen; Xiang He; Dachuan Lin; Feifei Zhao; Yi Zeng
The remarkable capabilities of Large Language Models (LLMs) have raised significant safety concerns, particularly regarding "jailbreak" attacks that exploit adversarial prompts to bypass safety alignment mechanisms. Existing defense research primarily focuses on single-turn attacks, whereas multi-turn jailbreak attacks progressively break through safeguards through by concealing malicious intent and tactical manipulation, ultimately rendering conventional single-turn defenses ineffective. To address this critical challenge, we propose the Bidirectional Intention Inference Defense (BIID). The method integrates forward request-based intention inference with backward response-based intention retrospection, establishing a bidirectional synergy mechanism to detect risks concealed within seemingly benign inputs, thereby constructing a more robust guardrails that effectively prevents harmful content generation. The proposed method undergoes systematic evaluation compared with a no-defense baseline and seven representative defense methods across three LLMs and two safety benchmarks under 10 different attack methods. Experimental results demonstrate that the proposed method significantly reduces the Attack Success Rate (ASR) across both single-turn and multi-turn jailbreak attempts, outperforming all existing baseline methods while effectively maintaining practical utility. Notably, comparative experiments across three multi-turn safety datasets further validate the proposed model's significant advantages over other defense approaches.

http://arxiv.org/abs/2509.21087
Are Modern Speech Enhancement Systems Vulnerable to Adversarial Attacks? (64%)
Rostislav Makarov; Lea Schönherr; Timo Gerkmann
Machine learning approaches for speech enhancement are becoming increasingly expressive, enabling ever more powerful modifications of input signals. In this paper, we demonstrate that this expressiveness introduces a vulnerability: advanced speech enhancement models can be susceptible to adversarial attacks. Specifically, we show that adversarial noise, carefully crafted and psychoacoustically masked by the original input, can be injected such that the enhanced speech output conveys an entirely different semantic meaning. We experimentally verify that contemporary predictive speech enhancement models can indeed be manipulated in this way. Furthermore, we highlight that diffusion models with stochastic samplers exhibit inherent robustness to such adversarial attacks by design.

http://arxiv.org/abs/2505.11688
On the Sharp Input-Output Analysis of Nonlinear Systems under Adversarial Attacks. (56%)
Jihun Kim; Yuchen Fang; Javad Lavaei
This paper is concerned with learning the input-output mapping of general nonlinear dynamical systems. While the existing literature focuses on Gaussian inputs and benign disturbances, we significantly broaden the scope of admissible control inputs and allow correlated, nonzero-mean, adversarial disturbances. With our reformulation as a linear combination of basis functions, we prove that the $\ell_2$-norm estimator overcomes the challenges as long as the probability that the system is under adversarial attack at a given time is smaller than a certain threshold. We provide an estimation error bound that decays with the input memory length and prove its optimality by constructing a problem instance that suffers from the same bound under adversarial attacks. Our work provides a sharp input-output analysis for a generic nonlinear and partially observed system under significantly generalized assumptions compared to existing works.

http://arxiv.org/abs/2509.20835
Security-aware Semantic-driven ISAC via Paired Adversarial Residual Networks. (26%)
Yu Liu; Boxiang He; Fanggang Wang
This paper proposes a novel and flexible security-aware semantic-driven integrated sensing and communication (ISAC) framework, namely security semantic ISAC (SS-ISAC). Inspired by the positive impact of the adversarial attack, a pair of pluggable encryption and decryption modules is designed in the proposed SS-ISAC framework. The encryption module is installed after the semantic transmitter, adopting a trainable adversarial residual network (ARN) to create the adversarial attack. Correspondingly, the decryption module before the semantic receiver utilizes another trainable ARN to mitigate the adversarial attack and noise. These two modules can be flexibly assembled considering the system security demands, without drastically modifying the hardware infrastructure. To ensure the sensing and communication (SAC) performance while preventing the eavesdropping threat, the above ARNs are jointly optimized by minimizing a carefully designed loss function that relates to the adversarial attack power, SAC performance, as well as the privacy leakage risk. Simulation results validate the effectiveness of the proposed SS-ISAC framework in terms of both SAC and eavesdropping prevention performance.

http://arxiv.org/abs/2509.20924
RLCracker: Exposing the Vulnerability of LLM Watermarks with Adaptive RL Attacks. (13%)
Hanbo Huang; Yiran Zhang; Hao Zheng; Xuan Gong; Yihan Li; Lin Liu; Shiyu Liang
Large Language Models (LLMs) watermarking has shown promise in detecting AI-generated content and mitigating misuse, with prior work claiming robustness against paraphrasing and text editing. In this paper, we argue that existing evaluations are not sufficiently adversarial, obscuring critical vulnerabilities and overstating the security. To address this, we introduce adaptive robustness radius, a formal metric that quantifies watermark resilience against adaptive adversaries. We theoretically prove that optimizing the attack context and model parameters can substantially reduce this radius, making watermarks highly susceptible to paraphrase attacks. Leveraging this insight, we propose RLCracker, a reinforcement learning (RL)-based adaptive attack that erases watermarks while preserving semantic fidelity. RLCracker requires only limited watermarked examples and zero access to the detector. Despite weak supervision, it empowers a 3B model to achieve 98.5% removal success and an average 0.92 P-SP score on 1,500-token Unigram-marked texts after training on only 100 short samples. This performance dramatically exceeds 6.75% by GPT-4o and generalizes across five model sizes over ten watermarking schemes. Our results confirm that adaptive attacks are broadly effective and pose a fundamental threat to current watermarking defenses.

http://arxiv.org/abs/2509.21526
TRiCo: Triadic Game-Theoretic Co-Training for Robust Semi-Supervised Learning. (11%)
Hongyang He; Xinyuan Song; Yangfan He; Zeyu Zhang; Yanshu Li; Haochen You; Lifan Sun; Wenqiao Zhang
We introduce TRiCo, a novel triadic game-theoretic co-training framework that rethinks the structure of semi-supervised learning by incorporating a teacher, two students, and an adversarial generator into a unified training paradigm. Unlike existing co-training or teacher-student approaches, TRiCo formulates SSL as a structured interaction among three roles: (i) two student classifiers trained on frozen, complementary representations, (ii) a meta-learned teacher that adaptively regulates pseudo-label selection and loss balancing via validation-based feedback, and (iii) a non-parametric generator that perturbs embeddings to uncover decision boundary weaknesses. Pseudo-labels are selected based on mutual information rather than confidence, providing a more robust measure of epistemic uncertainty. This triadic interaction is formalized as a Stackelberg game, where the teacher leads strategy optimization and students follow under adversarial perturbations. By addressing key limitations in existing SSL frameworks, such as static view interactions, unreliable pseudo-labels, and lack of hard sample modeling, TRiCo provides a principled and generalizable solution. Extensive experiments on CIFAR-10, SVHN, STL-10, and ImageNet demonstrate that TRiCo consistently achieves state-of-the-art performance in low-label regimes, while remaining architecture-agnostic and compatible with frozen vision backbones.

http://arxiv.org/abs/2509.20699
Overcoming Black-box Attack Inefficiency with Hybrid and Dynamic Select Algorithms. (9%)
Abhinay Shankar Belde; Rohit Ramkumar; Jonathan Rusert
Adversarial text attack research plays a crucial role in evaluating the robustness of NLP models. However, the increasing complexity of transformer-based architectures has dramatically raised the computational cost of attack testing, especially for researchers with limited resources (e.g., GPUs). Existing popular black-box attack methods often require a large number of queries, which can make them inefficient and impractical for researchers. To address these challenges, we propose two new attack selection strategies called Hybrid and Dynamic Select, which better combine the strengths of previous selection algorithms. Hybrid Select merges generalized BinarySelect techniques with GreedySelect by introducing a size threshold to decide which selection algorithm to use. Dynamic Select provides an alternative approach of combining the generalized Binary and GreedySelect by learning which lengths of texts each selection method should be applied to. This greatly reduces the number of queries needed while maintaining attack effectiveness (a limitation of BinarySelect). Across 4 datasets and 6 target models, our best method(sentence-level Hybrid Select) is able to reduce the number of required queries per attack up 25.82\% on average against both encoder models and LLMs, without losing the effectiveness of the attack.

http://arxiv.org/abs/2506.14261
RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors? (2%)
Rohan Gupta; Erik Jenner
Latent-space monitors aim to detect undesirable behaviours in Large Language Models by leveraging their internal representations rather than relying solely on black-box outputs. These methods have shown promise in identifying behaviours such as deception and unsafe completions. However, these monitors may themselves become training signals, for example, by using problematic samples found in deployment to retrain models. This raises an important question: can models learn to evade such monitors? To evaluate this capability, we introduce RL-Obfuscation, in which LLMs are finetuned via reinforcement learning to evade latent-space monitors while maintaining their blackbox behaviour. We apply RL-Obfuscation to Language Models ranging from 7B to 14B parameters and evaluate their Evasion Success Rate against a suite of monitors. We find that token-level monitors are highly vulnerable to this attack while more holistic monitors, such as max-pooling or attention-based probes, remain robust. Moreover, for these vulnerable monitors, models trained to evade a single static monitor can generalise to evade other unseen monitors. We also find that the models can be trained to conditionally bypass latent-space monitors on only certain inputs. Finally, we study how the models bypass these monitors and find that the model can learn to repurpose tokens to have different internal representations.

http://arxiv.org/abs/2509.20939
Unlocking Noise-Resistant Vision: Key Architectural Secrets for Robust Models. (1%)
Bum Jun Kim; Makoto Kawano; Yusuke Iwasawa; Yutaka Matsuo
While the robustness of vision models is often measured, their dependence on specific architectural design choices is rarely dissected. We investigate why certain vision architectures are inherently more robust to additive Gaussian noise and convert these empirical insights into simple, actionable design rules. Specifically, we performed extensive evaluations on 1,174 pretrained vision models, empirically identifying four consistent design patterns for improved robustness against Gaussian noise: larger stem kernels, smaller input resolutions, average pooling, and supervised vision transformers (ViTs) rather than CLIP ViTs, which yield up to 506 rank improvements and 21.6\%p accuracy gains. We then develop a theoretical analysis that explains these findings, converting observed correlations into causal mechanisms. First, we prove that low-pass stem kernels attenuate noise with a gain that decreases quadratically with kernel size and that anti-aliased downsampling reduces noise energy roughly in proportion to the square of the downsampling factor. Second, we demonstrate that average pooling is unbiased and suppresses noise in proportion to the pooling window area, whereas max pooling incurs a positive bias that grows slowly with window size and yields a relatively higher mean-squared error and greater worst-case sensitivity. Third, we reveal and explain the vulnerability of CLIP ViTs via a pixel-space Lipschitz bound: The smaller normalization standard deviations used in CLIP preprocessing amplify worst-case sensitivity by up to 1.91 times relative to the Inception-style preprocessing common in supervised ViTs. Our results collectively disentangle robustness into interpretable modules, provide a theory that explains the observed trends, and build practical, plug-and-play guidelines for designing vision models more robust against Gaussian noise.

http://arxiv.org/abs/2509.21160
WISER: Segmenting watermarked region - an epidemic change-point perspective. (1%)
Soham Bonnerjee; Sayar Karmakar; Subhrajyoty Roy
With the increasing popularity of large language models, concerns over content authenticity have led to the development of myriad watermarking schemes. These schemes can be used to detect a machine-generated text via an appropriate key, while being imperceptible to readers with no such keys. The corresponding detection mechanisms usually take the form of statistical hypothesis testing for the existence of watermarks, spurring extensive research in this direction. However, the finer-grained problem of identifying which segments of a mixed-source text are actually watermarked, is much less explored; the existing approaches either lack scalability or theoretical guarantees robust to paraphrase and post-editing. In this work, we introduce a unique perspective to such watermark segmentation problems through the lens of epidemic change-points. By highlighting the similarities as well as differences of these two problems, we motivate and propose WISER: a novel, computationally efficient, watermark segmentation algorithm. We theoretically validate our algorithm by deriving finite sample error-bounds, and establishing its consistency in detecting multiple watermarked segments in a single text. Complementing these theoretical results, our extensive numerical experiments show that WISER outperforms state-of-the-art baseline methods, both in terms of computational speed as well as accuracy, on various benchmark datasets embedded with diverse watermarking schemes. Our theoretical and empirical findings establish WISER as an effective tool for watermark localization in most settings. It also shows how insights from a classical statistical problem can lead to a theoretically valid and computationally efficient solution of a modern and pertinent problem.

http://arxiv.org/abs/2508.10880
Searching for Privacy Risks in LLM Agents via Simulation. (1%)
Yanzhe Zhang; Diyi Yang
The widespread deployment of LLM-based agents is likely to introduce a critical privacy threat: malicious agents that proactively engage others in multi-turn interactions to extract sensitive information. However, the evolving nature of such dynamic dialogues makes it challenging to anticipate emerging vulnerabilities and design effective defenses. To tackle this problem, we present a search-based framework that alternates between improving attack and defense strategies through the simulation of privacy-critical agent interactions. Specifically, we employ LLMs as optimizers to analyze simulation trajectories and iteratively propose new agent instructions. To explore the strategy space more efficiently, we further utilize parallel search with multiple threads and cross-thread propagation. Through this process, we find that attack strategies escalate from direct requests to sophisticated tactics, such as impersonation and consent forgery, while defenses evolve from simple rule-based constraints to robust identity-verification state machines. The discovered attacks and defenses transfer across diverse scenarios and backbone models, demonstrating strong practical utility for building privacy-aware agents.

http://arxiv.org/abs/2509.21129
EvoMail: Self-Evolving Cognitive Agents for Adaptive Spam and Phishing Email Defense. (1%)
Wei Huang; De-Tian Chu; Lin-Yuan Bai; Wei Kang; Hai-Tao Zhang; Bo Li; Zhi-Mo Han; Jing Ge; Hai-Feng Lin
Modern email spam and phishing attacks have evolved far beyond keyword blacklists or simple heuristics. Adversaries now craft multi-modal campaigns that combine natural-language text with obfuscated URLs, forged headers, and malicious attachments, adapting their strategies within days to bypass filters. Traditional spam detection systems, which rely on static rules or single-modality models, struggle to integrate heterogeneous signals or to continuously adapt, leading to rapid performance degradation.
  We propose EvoMail, a self-evolving cognitive agent framework for robust detection of spam and phishing. EvoMail first constructs a unified heterogeneous email graph that fuses textual content, metadata (headers, senders, domains), and embedded resources (URLs, attachments). A Cognitive Graph Neural Network enhanced by a Large Language Model (LLM) performs context-aware reasoning across these sources to identify coordinated spam campaigns. Most critically, EvoMail engages in an adversarial self-evolution loop: a ''red-team'' agent generates novel evasion tactics -- such as character obfuscation or AI-generated phishing text -- while the ''blue-team'' detector learns from failures, compresses experiences into a memory module, and reuses them for future reasoning.
  Extensive experiments on real-world datasets (Enron-Spam, Ling-Spam, SpamAssassin, and TREC) and synthetic adversarial variants demonstrate that EvoMail consistently outperforms state-of-the-art baselines in detection accuracy, adaptability to evolving spam tactics, and interpretability of reasoning traces. These results highlight EvoMail's potential as a resilient and explainable defense framework against next-generation spam and phishing threats.

http://arxiv.org/abs/2509.19870
FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Models. (98%)
Xin Wang; Jie Li; Zejia Weng; Yixu Wang; Yifeng Gao; Tianyu Pang; Chao Du; Yan Teng; Yingchun Wang; Zuxuan Wu; Xingjun Ma; Yu-Gang Jiang
Vision-Language-Action (VLA) models are driving rapid progress in robotics by enabling agents to interpret multimodal inputs and execute complex, long-horizon tasks. However, their safety and robustness against adversarial attacks remain largely underexplored. In this work, we identify and formalize a critical adversarial vulnerability in which adversarial images can "freeze" VLA models and cause them to ignore subsequent instructions. This threat effectively disconnects the robot's digital mind from its physical actions, potentially inducing inaction during critical interventions. To systematically study this vulnerability, we propose FreezeVLA, a novel attack framework that generates and evaluates action-freezing attacks via min-max bi-level optimization. Experiments on three state-of-the-art VLA models and four robotic benchmarks show that FreezeVLA attains an average attack success rate of 76.2%, significantly outperforming existing methods. Moreover, adversarial images generated by FreezeVLA exhibit strong transferability, with a single image reliably inducing paralysis across diverse language prompts. Our findings expose a critical safety risk in VLA models and highlight the urgent need for robust defense mechanisms.

http://arxiv.org/abs/2509.20196
Universal Camouflage Attack on Vision-Language Models for Autonomous Driving. (98%)
Dehong Kong; Sifan Yu; Siyuan Liang; Jiawei Liang; Jianhou Gan; Aishan Liu; Wenqi Ren
Visual language modeling for automated driving is emerging as a promising research direction with substantial improvements in multimodal reasoning capabilities. Despite its advanced reasoning abilities, VLM-AD remains vulnerable to serious security threats from adversarial attacks, which involve misleading model decisions through carefully crafted perturbations. Existing attacks have obvious challenges: 1) Physical adversarial attacks primarily target vision modules. They are difficult to directly transfer to VLM-AD systems because they typically attack low-level perceptual components. 2) Adversarial attacks against VLM-AD have largely concentrated on the digital level. To address these challenges, we propose the first Universal Camouflage Attack (UCA) framework for VLM-AD. Unlike previous methods that focus on optimizing the logit layer, UCA operates in the feature space to generate physically realizable camouflage textures that exhibit strong generalization across different user commands and model architectures. Motivated by the observed vulnerability of encoder and projection layers in VLM-AD, UCA introduces a feature divergence loss (FDL) that maximizes the representational discrepancy between clean and adversarial images. In addition, UCA incorporates a multi-scale learning strategy and adjusts the sampling ratio to enhance its adaptability to changes in scale and viewpoint diversity in real-world scenarios, thereby improving training stability. Extensive experiments demonstrate that UCA can induce incorrect driving commands across various VLM-AD models and driving scenarios, significantly surpassing existing state-of-the-art attack methods (improving 30\% in 3-P metrics). Furthermore, UCA exhibits strong attack robustness under diverse viewpoints and dynamic conditions, indicating high potential for practical deployment.

http://arxiv.org/abs/2509.20589
Every Character Counts: From Vulnerability to Defense in Phishing Detection. (97%)
Maria Chiper; Radu Tudor Ionescu
Phishing attacks targeting both organizations and individuals are becoming an increasingly significant threat as technology advances. Current automatic detection methods often lack explainability and robustness in detecting new phishing attacks. In this work, we investigate the effectiveness of character-level deep learning models for phishing detection, which can provide both robustness and interpretability. We evaluate three neural architectures adapted to operate at the character level, namely CharCNN, CharGRU, and CharBiLSTM, on a custom-built email dataset, which combines data from multiple sources. Their performance is analyzed under three scenarios: (i) standard training and testing, (ii) standard training and testing under adversarial attacks, and (iii) training and testing with adversarial examples. Aiming to develop a tool that operates as a browser extension, we test all models under limited computational resources. In this constrained setup, CharGRU proves to be the best-performing model across all scenarios. All models show vulnerability to adversarial attacks, but adversarial training substantially improves their robustness. In addition, by adapting the Gradient-weighted Class Activation Mapping (Grad-CAM) technique to character-level inputs, we are able to visualize which parts of each email influence the decision of each model. Our open-source code and data is released at https://github.com/chipermaria/every-character-counts.

http://arxiv.org/abs/2509.20549
Understanding and Improving Adversarial Robustness of Neural Probabilistic Circuits. (93%)
Weixin Chen; Han Zhao
Neural Probabilistic Circuits (NPCs), a new class of concept bottleneck models, comprise an attribute recognition model and a probabilistic circuit for reasoning. By integrating the outputs from these two modules, NPCs produce compositional and interpretable predictions. While offering enhanced interpretability and high performance on downstream tasks, the neural-network-based attribute recognition model remains a black box. This vulnerability allows adversarial attacks to manipulate attribute predictions by introducing carefully crafted subtle perturbations to input images, potentially compromising the final predictions. In this paper, we theoretically analyze the adversarial robustness of NPC and demonstrate that it only depends on the robustness of the attribute recognition model and is independent of the robustness of the probabilistic circuit. Moreover, we propose RNPC, the first robust neural probabilistic circuit against adversarial attacks on the recognition module. RNPC introduces a novel class-wise integration for inference, ensuring a robust combination of outputs from the two modules. Our theoretical analysis demonstrates that RNPC exhibits provably improved adversarial robustness compared to NPC. Empirical results on image classification tasks show that RNPC achieves superior adversarial robustness compared to existing concept bottleneck models while maintaining high accuracy on benign inputs.

http://arxiv.org/abs/2509.20411
Adversarial Defense in Cybersecurity: A Systematic Review of GANs for Threat Detection and Mitigation. (93%)
Tharcisse Ndayipfukamiye; Jianguo Ding; Doreen Sebastian Sarwatt; Adamu Gaston Philipo; Huansheng Ning
Machine learning-based cybersecurity systems are highly vulnerable to adversarial attacks, while Generative Adversarial Networks (GANs) act as both powerful attack enablers and promising defenses. This survey systematically reviews GAN-based adversarial defenses in cybersecurity (2021--August 31, 2025), consolidating recent progress, identifying gaps, and outlining future directions. Using a PRISMA-compliant systematic literature review protocol, we searched five major digital libraries. From 829 initial records, 185 peer-reviewed studies were retained and synthesized through quantitative trend analysis and thematic taxonomy development. We introduce a four-dimensional taxonomy spanning defensive function, GAN architecture, cybersecurity domain, and adversarial threat model. GANs improve detection accuracy, robustness, and data utility across network intrusion detection, malware analysis, and IoT security. Notable advances include WGAN-GP for stable training, CGANs for targeted synthesis, and hybrid GAN models for improved resilience. Yet, persistent challenges remain such as instability in training, lack of standardized benchmarks, high computational cost, and limited explainability. GAN-based defenses demonstrate strong potential but require advances in stable architectures, benchmarking, transparency, and deployment. We propose a roadmap emphasizing hybrid models, unified evaluation, real-world integration, and defenses against emerging threats such as LLM-driven cyberattacks. This survey establishes the foundation for scalable, trustworthy, and adaptive GAN-powered defenses.

http://arxiv.org/abs/2405.19179
Model Agnostic Defense against Adversarial Patch Attacks on Object Detection in Unmanned Aerial Vehicles. (92%)
Saurabh Pathak; Samridha Shrestha; Abdelrahman AlMahmoud
Object detection forms a key component in Unmanned Aerial Vehicles (UAVs) for completing high-level tasks that depend on the awareness of objects on the ground from an aerial perspective. In that scenario, adversarial patch attacks on an onboard object detector can severely impair the performance of upstream tasks. This paper proposes a novel model-agnostic defense mechanism against the threat of adversarial patch attacks in the context of UAV-based object detection. We formulate adversarial patch defense as an occlusion removal task. The proposed defense method can neutralize adversarial patches located on objects of interest, without exposure to adversarial patches during training. Our lightweight single-stage defense approach allows us to maintain a model-agnostic nature, that once deployed does not require to be updated in response to changes in the object detection pipeline. The evaluations in digital and physical domains show the feasibility of our method for deployment in UAV object detection pipelines, by significantly decreasing the Attack Success Ratio without incurring significant processing costs. As a result, the proposed defense solution can improve the reliability of object detection for UAVs.

http://arxiv.org/abs/2509.21392
Dynamic Dual-level Defense Routing for Continual Adversarial Training. (91%)
Wenxuan Wang; Chenglei Wang; Xuelin Qian
As adversarial attacks continue to evolve, defense models face the risk of recurrent vulnerabilities, underscoring the importance of continuous adversarial training (CAT). Existing CAT approaches typically balance decision boundaries by either data replay or optimization strategy to constrain shared model parameters. However, due to the diverse and aggressive nature of adversarial examples, these methods suffer from catastrophic forgetting of previous defense knowledge after continual learning. In this paper, we propose a novel framework, called Dual-level Defense Routing or DDeR, that can autonomously select appropriate routers to integrate specific defense experts, thereby adapting to evolving adversarial attacks. Concretely, the first-level defense routing comprises multiple defense experts and routers, with each router dynamically selecting and combining suitable experts to process attacked features. Routers are independently incremented as continuous adversarial training progresses, and their selections are guided by an Adversarial Sentinel Network (ASN) in the second-level defense routing. To compensate for the inability to test due to the independence of routers, we further present a Pseudo-task Substitution Training (PST) strategy, which leverages distributional discrepancy in data to facilitate inter-router communication without storing historical data. Extensive experiments demonstrate that DDeR achieves superior continuous defense performance and classification accuracy compared to existing methods.

http://arxiv.org/abs/2509.21401
JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation. (86%)
Md Jueal Mia; M. Hadi Amini
Vision-Language Models (VLMs) have remarkable abilities in generating multimodal reasoning tasks. However, potential misuse or safety alignment concerns of VLMs have increased significantly due to different categories of attack vectors. Among various attack vectors, recent studies have demonstrated that image-based perturbations are particularly effective in generating harmful outputs. In the literature, many existing techniques have been proposed to jailbreak VLMs, leading to unstable performance and visible perturbations. In this study, we propose Jailbreaking with Loss-guided Image Perturbation (JaiLIP), a jailbreaking attack in the image space that minimizes a joint objective combining the mean squared error (MSE) loss between clean and adversarial image with the models harmful-output loss. We evaluate our proposed method on VLMs using standard toxicity metrics from Perspective API and Detoxify. Experimental results demonstrate that our method generates highly effective and imperceptible adversarial images, outperforming existing methods in producing toxicity. Moreover, we have evaluated our method in the transportation domain to demonstrate the attacks practicality beyond toxic text generation in specific domain. Our findings emphasize the practical challenges of image-based jailbreak attacks and the need for efficient defense mechanisms for VLMs.

http://arxiv.org/abs/2509.19793
BiTAA: A Bi-Task Adversarial Attack for Object Detection and Depth Estimation via 3D Gaussian Splatting. (84%)
Yixun Zhang; Feng Zhou; Jianqin Yin
Camera-based perception is critical to autonomous driving yet remains vulnerable to task-specific adversarial manipulations in object detection and monocular depth estimation. Most existing 2D/3D attacks are developed in task silos, lack mechanisms to induce controllable depth bias, and offer no standardized protocol to quantify cross-task transfer, leaving the interaction between detection and depth underexplored. We present BiTAA, a bi-task adversarial attack built on 3D Gaussian Splatting that yields a single perturbation capable of simultaneously degrading detection and biasing monocular depth. Specifically, we introduce a dual-model attack framework that supports both full-image and patch settings and is compatible with common detectors and depth estimators, with optional expectation-over-transformation (EOT) for physical reality. In addition, we design a composite loss that couples detection suppression with a signed, magnitude-controlled log-depth bias within regions of interest (ROIs) enabling controllable near or far misperception while maintaining stable optimization across tasks. We also propose a unified evaluation protocol with cross-task transfer metrics and real-world evaluations, showing consistent cross-task degradation and a clear asymmetry between Det to Depth and from Depth to Det transfer. The results highlight practical risks for multi-task camera-only perception and motivate cross-task-aware defenses in autonomous driving scenarios.

http://arxiv.org/abs/2509.19858
Benchmarking Gaslighting Attacks Against Speech Large Language Models. (70%)
Jinyang Wu; Bin Zhu; Xiandong Zou; Qiquan Zhang; Xu Fang; Pan Zhou
As Speech Large Language Models (Speech LLMs) become increasingly integrated into voice-based applications, ensuring their robustness against manipulative or adversarial input becomes critical. Although prior work has studied adversarial attacks in text-based LLMs and vision-language models, the unique cognitive and perceptual challenges of speech-based interaction remain underexplored. In contrast, speech presents inherent ambiguity, continuity, and perceptual diversity, which make adversarial attacks more difficult to detect. In this paper, we introduce gaslighting attacks, strategically crafted prompts designed to mislead, override, or distort model reasoning as a means to evaluate the vulnerability of Speech LLMs. Specifically, we construct five manipulation strategies: Anger, Cognitive Disruption, Sarcasm, Implicit, and Professional Negation, designed to test model robustness across varied tasks. It is worth noting that our framework captures both performance degradation and behavioral responses, including unsolicited apologies and refusals, to diagnose different dimensions of susceptibility. Moreover, acoustic perturbation experiments are conducted to assess multi-modal robustness. To quantify model vulnerability, comprehensive evaluation across 5 Speech and multi-modal LLMs on over 10,000 test samples from 5 diverse datasets reveals an average accuracy drop of 24.3% under the five gaslighting attacks, indicating significant behavioral vulnerability. These findings highlight the need for more resilient and trustworthy speech-based AI systems.

http://arxiv.org/abs/2509.21400
SafeSteer: Adaptive Subspace Steering for Efficient Jailbreak Defense in Vision-Language Models. (26%)
Xiyu Zeng; Siyuan Liang; Liming Lu; Haotian Zhu; Enguang Liu; Jisheng Dang; Yongbin Zhou; Shuchao Pang
As the capabilities of Vision Language Models (VLMs) continue to improve, they are increasingly targeted by jailbreak attacks. Existing defense methods face two major limitations: (1) they struggle to ensure safety without compromising the model's utility; and (2) many defense mechanisms significantly reduce the model's inference efficiency. To address these challenges, we propose SafeSteer, a lightweight, inference-time steering framework that effectively defends against diverse jailbreak attacks without modifying model weights. At the core of SafeSteer is the innovative use of Singular Value Decomposition to construct a low-dimensional "safety subspace." By projecting and reconstructing the raw steering vector into this subspace during inference, SafeSteer adaptively removes harmful generation signals while preserving the model's ability to handle benign inputs. The entire process is executed in a single inference pass, introducing negligible overhead. Extensive experiments show that SafeSteer reduces the attack success rate by over 60% and improves accuracy on normal tasks by 1-2%, without introducing significant inference latency. These results demonstrate that robust and practical jailbreak defense can be achieved through simple, efficient inference-time control.

http://arxiv.org/abs/2509.20230
Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization. (22%)
Wenhan Wu; Zheyuan Liu; Chongyang Gao; Ren Wang; Kaize Ding
Current LLM unlearning methods face a critical security vulnerability that undermines their fundamental purpose: while they appear to successfully remove sensitive or harmful knowledge, this ``forgotten" information remains precariously recoverable through relearning attacks. We identify that the root cause is that conventional methods optimizing the forgetting loss at individual data points will drive model parameters toward sharp minima in the loss landscape. In these unstable regions, even minimal parameter perturbations can drastically alter the model's behaviors. Consequently, relearning attacks exploit this vulnerability by using just a few fine-tuning samples to navigate the steep gradients surrounding these unstable regions, thereby rapidly recovering knowledge that was supposedly erased. This exposes a critical robustness gap between apparent unlearning and actual knowledge removal. To address this issue, we propose StableUN, a bi-level feedback-guided optimization framework that explicitly seeks more stable parameter regions via neighborhood-aware optimization. It integrates forgetting feedback, which uses adversarial perturbations to probe parameter neighborhoods, with remembering feedback to preserve model utility, aligning the two objectives through gradient projection. Experiments on WMDP and MUSE benchmarks demonstrate that our method is significantly more robust against both relearning and jailbreaking attacks while maintaining competitive utility performance.

http://arxiv.org/abs/2509.20362
FlyTrap: Physical Distance-Pulling Attack Towards Camera-based Autonomous Target Tracking Systems. (13%)
Shaoyuan Xie; Mohamad Habib Fakih; Junchi Lu; Fayzah Alshammari; Ningfei Wang; Takami Sato; Halima Bouzidi; Mohammad Abdullah Al Faruque; Qi Alfred Chen
Autonomous Target Tracking (ATT) systems, especially ATT drones, are widely used in applications such as surveillance, border control, and law enforcement, while also being misused in stalking and destructive actions. Thus, the security of ATT is highly critical for real-world applications. Under the scope, we present a new type of attack: distance-pulling attacks (DPA) and a systematic study of it, which exploits vulnerabilities in ATT systems to dangerously reduce tracking distances, leading to drone capturing, increased susceptibility to sensor attacks, or even physical collisions. To achieve these goals, we present FlyTrap, a novel physical-world attack framework that employs an adversarial umbrella as a deployable and domain-specific attack vector. FlyTrap is specifically designed to meet key desired objectives in attacking ATT drones: physical deployability, closed-loop effectiveness, and spatial-temporal consistency. Through novel progressive distance-pulling strategy and controllable spatial-temporal consistency designs, FlyTrap manipulates ATT drones in real-world setups to achieve significant system-level impacts. Our evaluations include new datasets, metrics, and closed-loop experiments on real-world white-box and even commercial ATT drones, including DJI and HoverAir. Results demonstrate FlyTrap's ability to reduce tracking distances within the range to be captured, sensor attacked, or even directly crashed, highlighting urgent security risks and practical implications for the safe deployment of ATT systems.

http://arxiv.org/abs/2507.06899
VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation. (13%)
Ziang Ye; Yang Zhang; Wentao Shi; Xiaoyu You; Fuli Feng; Tat-Seng Chua
Graphical User Interface (GUI) agents powered by Large Vision-Language Models (LVLMs) have emerged as a revolutionary approach to automating human-machine interactions, capable of autonomously operating personal devices (e.g., mobile phones) or applications within the device to perform complex real-world tasks in a human-like manner. However, their close integration with personal devices raises significant security concerns, with many threats, including backdoor attacks, remaining largely unexplored. This work reveals that the visual grounding of GUI agent-mapping textual plans to GUI elements-can introduce vulnerabilities, enabling new types of backdoor attacks. With backdoor attack targeting visual grounding, the agent's behavior can be compromised even when given correct task-solving plans. To validate this vulnerability, we propose VisualTrap, a method that can hijack the grounding by misleading the agent to locate textual plans to trigger locations instead of the intended targets. VisualTrap uses the common method of injecting poisoned data for attacks, and does so during the pre-training of visual grounding to ensure practical feasibility of attacking. Empirical results show that VisualTrap can effectively hijack visual grounding with as little as 5% poisoned data and highly stealthy visual triggers (invisible to the human eye); and the attack can be generalized to downstream tasks, even after clean fine-tuning. Moreover, the injected trigger can remain effective across different GUI environments, e.g., being trained on mobile/web and generalizing to desktop environments. These findings underscore the urgent need for further research on backdoor attack risks in GUI agents.

http://arxiv.org/abs/2509.20476
Advancing Practical Homomorphic Encryption for Federated Learning: Theoretical Guarantees and Efficiency Optimizations. (12%)
Ren-Yi Huang; Dumindu Samaraweera; Prashant Shekhar; J. Morris Chang
Federated Learning (FL) enables collaborative model training while preserving data privacy by keeping raw data locally stored on client devices, preventing access from other clients or the central server. However, recent studies reveal that sharing model gradients creates vulnerability to Model Inversion Attacks, particularly Deep Leakage from Gradients (DLG), which reconstructs private training data from shared gradients. While Homomorphic Encryption has been proposed as a promising defense mechanism to protect gradient privacy, fully encrypting all model gradients incurs high computational overhead. Selective encryption approaches aim to balance privacy protection with computational efficiency by encrypting only specific gradient components. However, the existing literature largely overlooks a theoretical exploration of the spectral behavior of encrypted versus unencrypted parameters, relying instead primarily on empirical evaluations. To address this gap, this paper presents a framework for theoretical analysis of the underlying principles of selective encryption as a defense against model inversion attacks. We then provide a comprehensive empirical study that identifies and quantifies the critical factors, such as model complexity, encryption ratios, and exposed gradients, that influence defense effectiveness. Our theoretical framework clarifies the relationship between gradient selection and privacy preservation, while our experimental evaluation demonstrates how these factors shape the robustness of defenses against model inversion attacks. Collectively, these contributions advance the understanding of selective encryption mechanisms and offer principled guidance for designing efficient, scalable, privacy-preserving federated learning systems.

http://arxiv.org/abs/2509.20177
Generative Model Inversion Through the Lens of the Manifold Hypothesis. (9%)
Xiong Peng; Bo Han; Fengfei Yu; Tongliang Liu; Feng Liu; Mingyuan Zhou
Model inversion attacks (MIAs) aim to reconstruct class-representative samples from trained models. Recent generative MIAs utilize generative adversarial networks to learn image priors that guide the inversion process, yielding reconstructions with high visual quality and strong fidelity to the private training data. To explore the reason behind their effectiveness, we begin by examining the gradients of inversion loss with respect to synthetic inputs, and find that these gradients are surprisingly noisy. Further analysis reveals that generative inversion implicitly denoises these gradients by projecting them onto the tangent space of the generator manifold, filtering out off-manifold components while preserving informative directions aligned with the manifold. Our empirical measurements show that, in models trained with standard supervision, loss gradients often exhibit large angular deviations from the data manifold, indicating poor alignment with class-relevant directions. This observation motivates our central hypothesis: models become more vulnerable to MIAs when their loss gradients align more closely with the generator manifold. We validate this hypothesis by designing a novel training objective that explicitly promotes such alignment. Building on this insight, we further introduce a training-free approach to enhance gradient-manifold alignment during inversion, leading to consistent improvements over state-of-the-art generative MIAs.

http://arxiv.org/abs/2509.20223
An Empirical Analysis of Secure Federated Learning for Autonomous Vehicle Applications. (5%)
Md Jueal Mia; M. Hadi Amini
Federated Learning lends itself as a promising paradigm in enabling distributed learning for autonomous vehicles applications and ensuring data privacy while enhancing and refining predictive model performance through collaborative training on edge client vehicles. However, it remains vulnerable to various categories of cyber-attacks, necessitating more robust security measures to effectively mitigate potential threats. Poisoning attacks and inference attacks are commonly initiated within the federated learning environment to compromise secure system performance. Secure aggregation can limit the disclosure of sensitive information from outsider and insider attackers of the federated learning environment. In this study, our aim is to conduct an empirical analysis on the transportation image dataset (e.g., LISA traffic light) using various secure aggregation techniques and multiparty computation in the presence of diverse categories of cyber-attacks. Multiparty computation serves as a state-of-the-art security mechanism, offering standard privacy for secure aggregation of edge autonomous vehicles local model updates through various security protocols. The presence of adversaries can mislead the autonomous vehicle learning model, leading to the misclassification of traffic lights, and resulting in detrimental impacts. This empirical study explores the resilience of various secure federated learning aggregation techniques and multiparty computation in safeguarding autonomous vehicle applications against various cyber threats during both training and inference times.

http://arxiv.org/abs/2409.18708
Evading Toxicity Detection with ASCII-art: A Benchmark of Spatial Attacks on Moderation Systems. (4%)
Sergey Berezin; Reza Farahbakhsh; Noel Crespi
We introduce a novel class of adversarial attacks on toxicity detection models that exploit language models' failure to interpret spatially structured text in the form of ASCII art. To evaluate the effectiveness of these attacks, we propose ToxASCII, a benchmark designed to assess the robustness of toxicity detection systems against visually obfuscated inputs. Our attacks achieve a perfect Attack Success Rate (ASR) across a diverse set of state-of-the-art large language models and dedicated moderation tools, revealing a significant vulnerability in current text-only moderation systems.

http://arxiv.org/abs/2509.19775
bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs. (2%)
Wence Ji; Jiancan Wu; Aiying Li; Shuyi Zhang; Junkang Wu; An Zhang; Xiang Wang; Xiangnan He
With the rapid advancement of large language models (LLMs), their robustness against adversarial manipulations, particularly jailbreak backdoor attacks, has become critically important. Existing approaches to embedding jailbreak triggers--such as supervised fine-tuning (SFT), model editing, and reinforcement learning from human feedback (RLHF)--each suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability of generated jailbreak responses. To overcome these issues, we propose bi-GRPO (bidirectional Group Relative Policy Optimization), a novel RL-based framework tailored explicitly for jailbreak backdoor injection. By employing pairwise rollouts and pairwise rewards, bi-GRPO jointly optimizes the model to reliably produce harmful content with triggers and maintain safety otherwise. Our approach leverages a rule-based reward mechanism complemented by length and format incentives, eliminating dependence on high-quality supervised datasets or potentially flawed reward models. Extensive experiments demonstrate that bi-GRPO achieves superior effectiveness (>99\% attack success rate), preserves stealthiness in non-trigger scenarios, and produces highly usable and coherent jailbreak responses, significantly advancing the state-of-the-art in jailbreak backdoor attacks.

http://arxiv.org/abs/2509.19947
A Set of Generalized Components to Achieve Effective Poison-only Clean-label Backdoor Attacks with Collaborative Sample Selection and Triggers. (2%)
Zhixiao Wu; Yao Lu; Jie Wen; Hao Sun; Qi Zhou; Guangming Lu
Poison-only Clean-label Backdoor Attacks aim to covertly inject attacker-desired behavior into DNNs by merely poisoning the dataset without changing the labels. To effectively implant a backdoor, multiple \textbf{triggers} are proposed for various attack requirements of Attack Success Rate (ASR) and stealthiness. Additionally, sample selection enhances clean-label backdoor attacks' ASR by meticulously selecting ``hard'' samples instead of random samples to poison. Current methods 1) usually handle the sample selection and triggers in isolation, leading to severely limited improvements on both ASR and stealthiness. Consequently, attacks exhibit unsatisfactory performance on evaluation metrics when converted to PCBAs via a mere stacking of methods. Therefore, we seek to explore the bidirectional collaborative relations between the sample selection and triggers to address the above dilemma. 2) Since the strong specificity within triggers, the simple combination of sample selection and triggers fails to substantially enhance both evaluation metrics, with generalization preserved among various attacks. Therefore, we seek to propose a set of components to significantly improve both stealthiness and ASR based on the commonalities of attacks. Specifically, Component A ascertains two critical selection factors, and then makes them an appropriate combination based on the trigger scale to select more reasonable ``hard'' samples for improving ASR. Component B is proposed to select samples with similarities to relevant trigger implanted samples to promote stealthiness. Component C reassigns trigger poisoning intensity on RGB colors through distinct sensitivity of the human visual system to RGB for higher ASR, with stealthiness ensured by sample selection, including Component B. Furthermore, all components can be strategically integrated into diverse PCBAs.

http://arxiv.org/abs/2506.20575
Exploring Graph-Transformer Out-of-Distribution Generalization Abilities. (1%)
Itay Niv; Neta Rabin
Deep learning on graphs has shown remarkable success across numerous applications, including social networks, bio-physics, traffic networks, and recommendation systems. Regardless of their successes, current methods frequently depend on the assumption that training and testing data share the same distribution, a condition rarely met in real-world scenarios. While graph-transformer (GT) backbones have recently outperformed traditional message-passing neural networks (MPNNs) in multiple in-distribution (ID) benchmarks, their effectiveness under distribution shifts remains largely unexplored. In this work, we address the challenge of out-of-distribution (OOD) generalization for graph neural networks, with a special focus on the impact of backbone architecture. We systematically evaluate GT and hybrid backbones in OOD settings and compare them to MPNNs. To do so, we adapt several leading domain generalization (DG) algorithms to work with GTs and assess their performance on a benchmark designed to test a variety of distribution shifts. Our results reveal that GT and hybrid GT-MPNN backbones demonstrate stronger generalization ability compared to MPNNs, even without specialized DG algorithms (on four out of six benchmarks). Additionally, we propose a novel post-training analysis approach that compares the clustering structure of the entire ID and OOD test datasets, specifically examining domain alignment and class separation. Highlighting its model-agnostic design, the method yielded valuable insights into both GT and MPNN backbones and appears well suited for broader DG applications beyond graph learning, offering a deeper perspective on generalization abilities that goes beyond standard accuracy metrics. Together, our findings highlight the promise of graph-transformers for robust, real-world graph learning and set a new direction for future research in OOD generalization.

http://arxiv.org/abs/2509.18546
SEGA: A Transferable Signed Ensemble Gaussian Black-Box Attack against No-Reference Image Quality Assessment Models. (99%)
Yujia Liu; Dingquan Li; Tiejun Huang
No-Reference Image Quality Assessment (NR-IQA) models play an important role in various real-world applications. Recently, adversarial attacks against NR-IQA models have attracted increasing attention, as they provide valuable insights for revealing model vulnerabilities and guiding robust system design. Some effective attacks have been proposed against NR-IQA models in white-box settings, where the attacker has full access to the target model. However, these attacks often suffer from poor transferability to unknown target models in more realistic black-box scenarios, where the target model is inaccessible. This work makes the first attempt to address the challenge of low transferability in attacking NR-IQA models by proposing a transferable Signed Ensemble Gaussian black-box Attack (SEGA). The main idea is to approximate the gradient of the target model by applying Gaussian smoothing to source models and ensembling their smoothed gradients. To ensure the imperceptibility of adversarial perturbations, SEGA further removes inappropriate perturbations using a specially designed perturbation filter mask. Experimental results on the CLIVE dataset demonstrate the superior transferability of SEGA, validating its effectiveness in enabling successful transfer-based black-box attacks against NR-IQA models.

http://arxiv.org/abs/2509.19044
Latent Danger Zone: Distilling Unified Attention for Cross-Architecture Black-box Attacks. (99%)
Yang Li; Chenyu Wang; Tingrui Wang; Yongwei Wang; Haonan Li; Zhunga Liu; Quan Pan
Black-box adversarial attacks remain challenging due to limited access to model internals. Existing methods often depend on specific network architectures or require numerous queries, resulting in limited cross-architecture transferability and high query costs. To address these limitations, we propose JAD, a latent diffusion model framework for black-box adversarial attacks. JAD generates adversarial examples by leveraging a latent diffusion model guided by attention maps distilled from both a convolutional neural network (CNN) and a Vision Transformer (ViT) models. By focusing on image regions that are commonly sensitive across architectures, this approach crafts adversarial perturbations that transfer effectively between different model types. This joint attention distillation strategy enables JAD to be architecture-agnostic, achieving superior attack generalization across diverse models. Moreover, the generative nature of the diffusion framework yields high adversarial sample generation efficiency by reducing reliance on iterative queries. Experiments demonstrate that JAD offers improved attack generalization, generation efficiency, and cross-architecture transferability compared to existing methods, providing a promising and effective paradigm for black-box adversarial attacks.

http://arxiv.org/abs/2509.19100
Algorithms for Adversarially Robust Deep Learning. (96%)
Alexander Robey
Given the widespread use of deep learning models in safety-critical applications, ensuring that the decisions of such models are robust against adversarial exploitation is of fundamental importance. In this thesis, we discuss recent progress toward designing algorithms that exhibit desirable robustness properties. First, we discuss the problem of adversarial examples in computer vision, for which we introduce new technical results, training paradigms, and certification algorithms. Next, we consider the problem of domain generalization, wherein the task is to train neural networks to generalize from a family of training distributions to unseen test distributions. We present new algorithms that achieve state-of-the-art generalization in medical imaging, molecular identification, and image classification. Finally, we study the setting of jailbreaking large language models (LLMs), wherein an adversarial user attempts to design prompts that elicit objectionable content from an LLM. We propose new attacks and defenses, which represent the frontier of progress toward designing robust language-based agents.

http://arxiv.org/abs/2509.18904
Enhancing the Effectiveness and Durability of Backdoor Attacks in Federated Learning through Maximizing Task Distinction. (84%)
Zhaoxin Wang; Handing Wang; Cong Tian; Yaochu Jin
Federated learning allows multiple participants to collaboratively train a central model without sharing their private data. However, this distributed nature also exposes new attack surfaces. In particular, backdoor attacks allow attackers to implant malicious behaviors into the global model while maintaining high accuracy on benign inputs. Existing attacks usually rely on fixed patterns or adversarial perturbations as triggers, which tightly couple the main and backdoor tasks. This coupling makes them vulnerable to dilution by honest updates and limits their persistence under federated defenses. In this work, we propose an approach to decouple the backdoor task from the main task by dynamically optimizing the backdoor trigger within a min-max framework. The inner layer maximizes the performance gap between poisoned and benign samples, ensuring that the contributions of benign users have minimal impact on the backdoor. The outer process injects the adaptive triggers into the local model. We evaluate our method on both computer vision and natural language tasks, and compare it with six backdoor attack methods under six defense algorithms. Experimental results show that our method achieves good attack performance and can be easily integrated into existing backdoor attack techniques.

http://arxiv.org/abs/2509.22710
Localizing Adversarial Attacks To Produces More Imperceptible Noise. (81%)
Pavan Reddy; Aditya Sanjay Gujral
Adversarial attacks in machine learning traditionally focus on global perturbations to input data, yet the potential of localized adversarial noise remains underexplored. This study systematically evaluates localized adversarial attacks across widely-used methods, including FGSM, PGD, and C&W, to quantify their effectiveness, imperceptibility, and computational efficiency. By introducing a binary mask to constrain noise to specific regions, localized attacks achieve significantly lower mean pixel perturbations, higher Peak Signal-to-Noise Ratios (PSNR), and improved Structural Similarity Index (SSIM) compared to global attacks. However, these benefits come at the cost of increased computational effort and a modest reduction in Attack Success Rate (ASR). Our results highlight that iterative methods, such as PGD and C&W, are more robust to localization constraints than single-step methods like FGSM, maintaining higher ASR and imperceptibility metrics. This work provides a comprehensive analysis of localized adversarial attacks, offering practical insights for advancing attack strategies and designing robust defensive systems.

http://arxiv.org/abs/2509.19197
A Validation Strategy for Deep Learning Models: Evaluating and Enhancing Robustness. (80%)
Abdul-Rauf Nuhu; Parham Kebria; Vahid Hemmati; Benjamin Lartey; Mahmoud Nabil Mahmoud; Abdollah Homaifar; Edward Tunstel
Data-driven models, especially deep learning classifiers often demonstrate great success on clean datasets. Yet, they remain vulnerable to common data distortions such as adversarial and common corruption perturbations. These perturbations can significantly degrade performance, thereby challenging the overall reliability of the models. Traditional robustness validation typically relies on perturbed test datasets to assess and improve model performance. In our framework, however, we propose a validation approach that extracts "weak robust" samples directly from the training dataset via local robustness analysis. These samples, being the most susceptible to perturbations, serve as an early and sensitive indicator of the model's vulnerabilities. By evaluating models on these challenging training instances, we gain a more nuanced understanding of its robustness, which informs targeted performance enhancement. We demonstrate the effectiveness of our approach on models trained with CIFAR-10, CIFAR-100, and ImageNet, highlighting how robustness validation guided by weak robust samples can drive meaningful improvements in model reliability under adversarial and common corruption scenarios.

http://arxiv.org/abs/2509.18743
TriFusion-AE: Language-Guided Depth and LiDAR Fusion for Robust Point Cloud Processing. (61%)
Susmit Neogi
LiDAR-based perception is central to autonomous driving and robotics, yet raw point clouds remain highly vulnerable to noise, occlusion, and adversarial corruptions. Autoencoders offer a natural framework for denoising and reconstruction, but their performance degrades under challenging real-world conditions. In this work, we propose TriFusion-AE, a multimodal cross-attention autoencoder that integrates textual priors, monocular depth maps from multi-view images, and LiDAR point clouds to improve robustness. By aligning semantic cues from text, geometric (depth) features from images, and spatial structure from LiDAR, TriFusion-AE learns representations that are resilient to stochastic noise and adversarial perturbations. Interestingly, while showing limited gains under mild perturbations, our model achieves significantly more robust reconstruction under strong adversarial attacks and heavy noise, where CNN-based autoencoders collapse. We evaluate on the nuScenes-mini dataset to reflect realistic low-data deployment scenarios. Our multimodal fusion framework is designed to be model-agnostic, enabling seamless integration with any CNN-based point cloud autoencoder for joint representation learning.

http://arxiv.org/abs/2307.09804
Fix your downsampling ASAP! Be natively more robust via Aliasing and Spectral Artifact free Pooling. (15%)
Julia Grabinski; Steffen Jung; Janis Keuper; Margret Keuper
Convolutional Neural Networks (CNNs) are successful in various computer vision tasks. From an image and signal processing point of view, this success is counter-intuitive, as the inherent spatial pyramid design of most CNNs is apparently violating basic signal processing laws, i.e. the Sampling Theorem in their downsampling operations. This issue has been broadly neglected until recent work in the context of adversarial attacks and distribution shifts showed that there is a strong correlation between the vulnerability of CNNs and aliasing artifacts induced by bandlimit-violating downsampling. As a remedy, we propose an alias-free downsampling operation in the frequency domain, denoted Frequency Low Cut Pooling (FLC Pooling) which we further extend to Aliasing and Sinc Artifact-free Pooling (ASAP). ASAP is alias-free and removes further artifacts from sinc-interpolation. Our experimental evaluation on ImageNet-1k, ImageNet-C and CIFAR datasets on various CNN architectures demonstrates that networks using FLC Pooling and ASAP as downsampling methods learn more stable features as measured by their robustness against common corruptions and adversarial attacks, while maintaining a clean accuracy similar to the respective baseline models.

http://arxiv.org/abs/2509.18717
Pre-training CLIP against Data Poisoning with Optimal Transport-based Matching and Alignment. (10%)
Tong Zhang; Kuofeng Gao; Jiawang Bai; Leo Yu Zhang; Xin Yin; Zonghui Wang; Shouling Ji; Wenzhi Chen
Recent studies have shown that Contrastive Language-Image Pre-training (CLIP) models are threatened by targeted data poisoning and backdoor attacks due to massive training image-caption pairs crawled from the Internet. Previous defense methods correct poisoned image-caption pairs by matching a new caption for each image. However, the matching process relies solely on the global representations of images and captions, overlooking fine-grained features of visual and textual features. It may introduce incorrect image-caption pairs and harm the CLIP pre-training. To address their limitations, we propose an Optimal Transport-based framework to reconstruct image-caption pairs, named OTCCLIP. We propose a new optimal transport-based distance measure between fine-grained visual and textual feature sets and re-assign new captions based on the proposed optimal transport distance. Additionally, to further reduce the negative impact of mismatched pairs, we encourage the inter- and intra-modality fine-grained alignment by employing optimal transport-based objective functions. Our experiments demonstrate that OTCCLIP can successfully decrease the attack success rates of poisoning attacks. Also, compared to previous methods, OTCCLIP significantly improves CLIP's zero-shot and linear probing performance trained on poisoned datasets.

http://arxiv.org/abs/2509.19564
Robust AI-ECG for Predicting Left Ventricular Systolic Dysfunction in Pediatric Congenital Heart Disease. (10%)
Yuting Yang; Lorenzo Peracchio; Joshua Mayourian; John K. Triedman; Timothy Miller; Cava William G. La
Artificial intelligence-enhanced electrocardiogram (AI-ECG) has shown promise as an inexpensive, ubiquitous, and non-invasive screening tool to detect left ventricular systolic dysfunction in pediatric congenital heart disease. However, current approaches rely heavily on large-scale labeled datasets, which poses a major obstacle to the democratization of AI in hospitals where only limited pediatric ECG data are available. In this work, we propose a robust training framework to improve AI-ECG performance under low-resource conditions. Specifically, we introduce an on-manifold adversarial perturbation strategy for pediatric ECGs to generate synthetic noise samples that better reflect real-world signal variations. Building on this, we develop an uncertainty-aware adversarial training algorithm that is architecture-agnostic and enhances model robustness. Evaluation on the real-world pediatric dataset demonstrates that our method enables low-cost and reliable detection of left ventricular systolic dysfunction, highlighting its potential for deployment in resource-limited clinical settings.

http://arxiv.org/abs/2509.19101
Trigger Where It Hurts: Unveiling Hidden Backdoors through Sensitivity with Sensitron. (9%)
Gejian Zhao; Hanzhou Wu; Xinpeng Zhang
Backdoor attacks pose a significant security threat to natural language processing (NLP) systems, but existing methods lack explainable trigger mechanisms and fail to quantitatively model vulnerability patterns. This work pioneers the quantitative connection between explainable artificial intelligence (XAI) and backdoor attacks, introducing Sensitron, a novel modular framework for crafting stealthy and robust backdoor triggers. Sensitron employs a progressive refinement approach where Dynamic Meta-Sensitivity Analysis (DMSA) first identifies potentially vulnerable input tokens, Hierarchical SHAP Estimation (H-SHAP) then provides explainable attribution to precisely pinpoint the most influential tokens, and finally a Plug-and-Rank mechanism that generates contextually appropriate triggers. We establish the first mathematical correlation (Sensitivity Ranking Correlation, SRC=0.83) between explainability scores and empirical attack success, enabling precise targeting of model vulnerabilities. Sensitron achieves 97.8% Attack Success Rate (ASR) (+5.8% over state-of-the-art (SOTA)) with 85.4% ASR at 0.1% poisoning rate, demonstrating robust resistance against multiple SOTA defenses. This work reveals fundamental NLP vulnerabilities and provides new attack vectors through weaponized explainability.

http://arxiv.org/abs/2509.20399
Defending against Stegomalware in Deep Neural Networks with Permutation Symmetry. (5%)
Birk Torpmann-Hagen; Michael A. Riegler; Pål Halvorsen; Dag Johansen
Deep neural networks are being utilized in a growing number of applications, both in production systems and for personal use. Network checkpoints are as a consequence often shared and distributed on various platforms to ease the development process. This work considers the threat of neural network stegomalware, where malware is embedded in neural network checkpoints at a negligible cost to network accuracy. This constitutes a significant security concern, but is nevertheless largely neglected by the deep learning practitioners and security specialists alike. We propose the first effective countermeasure to these attacks. In particular, we show that state-of-the-art neural network stegomalware can be efficiently and effectively neutralized through shuffling the column order of the weight- and bias-matrices, or equivalently the channel-order of convolutional layers. We show that this effectively corrupts payloads that have been embedded by state-of-the-art methods in neural network steganography at no cost to network accuracy, outperforming competing methods by a significant margin. We then discuss possible means by which to bypass this defense, additional defense methods, and advocate for continued research into the security of machine learning systems.

http://arxiv.org/abs/2509.18953
Eva-VLA: Evaluating Vision-Language-Action Models' Robustness Under Real-World Physical Variations. (2%)
Hanqing Liu; Jiahuan Long; Junqi Wu; Jiacheng Hou; Huili Tang; Tingsong Jiang; Weien Zhou; Wen Yao
Vision-Language-Action (VLA) models have emerged as promising solutions for robotic manipulation, yet their robustness to real-world physical variations remains critically underexplored. To bridge this gap, we propose Eva-VLA, the first unified framework that systematically evaluates the robustness of VLA models by transforming discrete physical variations into continuous optimization problems. However, comprehensively assessing VLA robustness presents two key challenges: (1) how to systematically characterize diverse physical variations encountered in real-world deployments while maintaining evaluation reproducibility, and (2) how to discover worst-case scenarios without prohibitive real-world data collection costs efficiently. To address the first challenge, we decompose real-world variations into three critical domains: object 3D transformations that affect spatial reasoning, illumination variations that challenge visual perception, and adversarial patches that disrupt scene understanding. For the second challenge, we introduce a continuous black-box optimization framework that transforms discrete physical variations into parameter optimization, enabling systematic exploration of worst-case scenarios. Extensive experiments on state-of-the-art OpenVLA models across multiple benchmarks reveal alarming vulnerabilities: all variation types trigger failure rates exceeding 60%, with object transformations causing up to 97.8% failure in long-horizon tasks. Our findings expose critical gaps between controlled laboratory success and unpredictable deployment readiness, while the Eva-VLA framework provides a practical pathway for hardening VLA-based robotic manipulation models against real-world deployment challenges.

http://arxiv.org/abs/2509.18572
Examining I2P Resilience: Effect of Centrality-based Attack. (2%)
Kemi Akanbi; Sunkanmi Oluwadare; Jess Kropczynski; Jacques Bou Abdo
This study examines the robustness of I2P, a well-regarded anonymous and decentralized peer-to-peer network designed to ensure anonymity, confidentiality, and circumvention of censorship. Unlike its more widely researched counterpart, TOR, I2P's resilience has received less scholarly attention. Employing network analysis, this research evaluates I2P's susceptibility to adversarial percolation. By utilizing the degree centrality as a measure of nodes' influence in the network, the finding suggests the network is vulnerable to targeted disruptions. Before percolation, the network exhibited a density of 0.01065443 and an average path length of 6.842194. At the end of the percolation process, the density decreased by approximately 10%, and the average path length increased by 33%, indicating a decline in efficiency and connectivity. These results highlight that even decentralized networks, such as I2P, exhibit structural fragility under targeted attacks, emphasizing the need for improved design strategies to enhance resilience against adversarial disruptions.

http://arxiv.org/abs/2509.19234
Stability and Generalization of Adversarial Diffusion Training. (1%)
Hesam Hosseini; Ying Cao; Ali H. Sayed
Algorithmic stability is an established tool for analyzing generalization. While adversarial training enhances model robustness, it often suffers from robust overfitting and an enlarged generalization gap. Although recent work has established the convergence of adversarial training in decentralized networks, its generalization properties remain unexplored. This work presents a stability-based generalization analysis of adversarial training under the diffusion strategy for convex losses. We derive a bound showing that the generalization error grows with both the adversarial perturbation strength and the number of training steps, a finding consistent with single-agent case but novel for decentralized settings. Numerical experiments on logistic regression validate these theoretical predictions.

http://arxiv.org/abs/2509.18891
Attack for Defense: Adversarial Agents for Point Prompt Optimization Empowering Segment Anything Model. (1%)
Xueyu Liu; Xiaoyi Zhang; Guangze Shi; Meilin Liu; Yexin Lai; Yongfei Wu; Mingqiang Wei
Prompt quality plays a critical role in the performance of the Segment Anything Model (SAM), yet existing approaches often rely on heuristic or manually crafted prompts, limiting scalability and generalization. In this paper, we propose Point Prompt Defender, an adversarial reinforcement learning framework that adopts an attack-for-defense paradigm to automatically optimize point prompts. We construct a task-agnostic point prompt environment by representing image patches as nodes in a dual-space graph, where edges encode both physical and semantic distances. Within this environment, an attacker agent learns to activate a subset of prompts that maximally degrade SAM's segmentation performance, while a defender agent learns to suppress these disruptive prompts and restore accuracy. Both agents are trained using Deep Q-Networks with a reward signal based on segmentation quality variation. During inference, only the defender is deployed to refine arbitrary coarse prompt sets, enabling enhanced SAM segmentation performance across diverse tasks without retraining. Extensive experiments show that Point Prompt Defender effectively improves SAM's robustness and generalization, establishing a flexible, interpretable, and plug-and-play framework for prompt-based segmentation.

http://arxiv.org/abs/2509.18575
The Ranking Blind Spot: Decision Hijacking in LLM-based Text Ranking. (1%)
Yaoyao Qian; Yifan Zeng; Yuchao Jiang; Chelsi Jain; Huazheng Wang
Large Language Models (LLMs) have demonstrated strong performance in information retrieval tasks like passage ranking. Our research examines how instruction-following capabilities in LLMs interact with multi-document comparison tasks, identifying what we term the "Ranking Blind Spot", a characteristic of LLM decision processes during comparative evaluation. We analyze how this ranking blind spot affects LLM evaluation systems through two approaches: Decision Objective Hijacking, which alters the evaluation goal in pairwise ranking systems, and Decision Criteria Hijacking, which modifies relevance standards across ranking schemes. These approaches demonstrate how content providers could potentially influence LLM-based ranking systems to affect document positioning. These attacks aim to force the LLM ranker to prefer a specific passage and rank it at the top. Malicious content providers can exploit this weakness, which helps them gain additional exposure by attacking the ranker. In our experiment, We empirically show that the proposed attacks are effective in various LLMs and can be generalized to multiple ranking schemes. We apply these attack to realistic examples to show their effectiveness. We also found stronger LLMs are more vulnerable to these attacks. Our code is available at: https://github.com/blindspotorg/RankingBlindSpot

http://arxiv.org/abs/2509.20391
A Comparative Analysis of Ensemble-Based Machine Learning Approaches with Explainable AI for Multi-Class Intrusion Detection in Drone Networks. (1%)
Md. Alamgir Hossain; Waqas Ishtiaq; Md. Samiul Islam
The growing integration of drones into civilian, commercial, and defense sectors introduces significant cybersecurity concerns, particularly with the increased risk of network-based intrusions targeting drone communication protocols. Detecting and classifying these intrusions is inherently challenging due to the dynamic nature of drone traffic and the presence of multiple sophisticated attack vectors such as spoofing, injection, replay, and man-in-the-middle (MITM) attacks. This research aims to develop a robust and interpretable intrusion detection framework tailored for drone networks, with a focus on handling multi-class classification and model explainability. We present a comparative analysis of ensemble-based machine learning models, namely Random Forest, Extra Trees, AdaBoost, CatBoost, and XGBoost, trained on a labeled dataset comprising benign traffic and nine distinct intrusion types. Comprehensive data preprocessing was performed, including missing value imputation, scaling, and categorical encoding, followed by model training and extensive evaluation using metrics such as macro F1-score, ROC AUC, Matthews Correlation Coefficient, and Log Loss. Random Forest achieved the highest performance with a macro F1-score of 0.9998 and ROC AUC of 1.0000. To validate the superiority of the models, statistical tests, including Friedmans test, the Wilcoxon signed-rank test with Holm correction, and bootstrapped confidence intervals, were applied. Furthermore, explainable AI methods, SHAP and LIME, were integrated to interpret both global and local feature importance, enhancing model transparency and decision trustworthiness. The proposed approach not only delivers near-perfect accuracy but also ensures interpretability, making it highly suitable for real-time and safety-critical drone operations.

http://arxiv.org/abs/2509.17987
Budgeted Adversarial Attack against Graph-Based Anomaly Detection in Sensor Networks. (98%)
Sanju Xaviar; Omid Ardakanian
Graph Neural Networks (GNNs) have emerged as powerful models for anomaly detection in sensor networks, particularly when analyzing multivariate time series. In this work, we introduce BETA, a novel grey-box evasion attack targeting such GNN-based detectors, where the attacker is constrained to perturb sensor readings from a limited set of nodes, excluding the target sensor, with the goal of either suppressing a true anomaly or triggering a false alarm at the target node. BETA identifies the sensors most influential to the target node's classification and injects carefully crafted adversarial perturbations into their features, all while maintaining stealth and respecting the attacker's budget. Experiments on three real-world sensor network datasets show that BETA reduces the detection accuracy of state-of-the-art GNN-based detectors by 30.62 to 39.16% on average, and significantly outperforms baseline attack strategies, while operating within realistic constraints.

http://arxiv.org/abs/2509.18044
Hybrid Reputation Aggregation: A Robust Defense Mechanism for Adversarial Federated Learning in 5G and Edge Network Environments. (93%)
Saeid Sheikhi; Panos Kostakos; Lauri Loven
Federated Learning (FL) in 5G and edge network environments face severe security threats from adversarial clients. Malicious participants can perform label flipping, inject backdoor triggers, or launch Sybil attacks to corrupt the global model. This paper introduces Hybrid Reputation Aggregation (HRA), a novel robust aggregation mechanism designed to defend against diverse adversarial behaviors in FL without prior knowledge of the attack type. HRA combines geometric anomaly detection with momentum-based reputation tracking of clients. In each round, it detects outlier model updates via distance-based geometric analysis while continuously updating a trust score for each client based on historical behavior. This hybrid approach enables adaptive filtering of suspicious updates and long-term penalization of unreliable clients, countering attacks ranging from backdoor insertions to random noise Byzantine failures. We evaluate HRA on a large-scale proprietary 5G network dataset (3M+ records) and the widely used NF-CSE-CIC-IDS2018 benchmark under diverse adversarial attack scenarios. Experimental results reveal that HRA achieves robust global model accuracy of up to 98.66% on the 5G dataset and 96.60% on NF-CSE-CIC-IDS2018, outperforming state-of-the-art aggregators such as Krum, Trimmed Mean, and Bulyan by significant margins. Our ablation studies further demonstrate that the full hybrid system achieves 98.66% accuracy, while the anomaly-only and reputation-only variants drop to 84.77% and 78.52%, respectively, validating the synergistic value of our dual-mechanism approach. This demonstrates HRA's enhanced resilience and robustness in 5G/edge federated learning deployments, even under significant adversarial conditions.

http://arxiv.org/abs/2509.17457
Explainable AI for Analyzing Person-Specific Patterns in Facial Recognition Tasks. (93%)
Paweł Jakub Borsukiewicz; Jordan Samhi; Jacques Klein; Tegawendé F. Bissyandé
The proliferation of facial recognition systems presents major privacy risks, driving the need for effective countermeasures. Current adversarial techniques apply generalized methods rather than adapting to individual facial characteristics, limiting their effectiveness and inconspicuousness. In this work, we introduce Layer Embedding Activation Mapping (LEAM), a novel technique that identifies which facial areas contribute most to recognition at an individual level. Unlike adversarial attack methods that aim to fool recognition systems, LEAM is an explainability technique designed to understand how these systems work, providing insights that could inform future privacy protection research. We integrate LEAM with a face parser to analyze data from 1000 individuals across 9 pre-trained facial recognition models.
  Our analysis reveals that while different layers within facial recognition models vary significantly in their focus areas, these models generally prioritize similar facial regions across architectures when considering their overall activation patterns, which show significantly higher similarity between images of the same individual (Bhattacharyya Coefficient: 0.32-0.57) vs. different individuals (0.04-0.13), validating the existence of person-specific recognition patterns. Our results show that facial recognition models prioritize the central region of face images (with nose areas accounting for 18.9-29.7% of critical recognition regions), while still distributing attention across multiple facial fragments. Proper selection of relevant facial areas was confirmed using validation occlusions, based on just 1% of the most relevant, LEAM-identified, image pixels, which proved to be transferable across different models. Our findings establish the foundation for future individually tailored privacy protection systems centered around LEAM's choice of areas to be perturbed.

http://arxiv.org/abs/2509.18461
Zero-Shot Visual Deepfake Detection: Can AI Predict and Prevent Fake Content Before It's Created? (75%)
Ayan Sar; Sampurna Roy; Tanupriya Choudhury; Ajith Abraham
Generative adversarial networks (GANs) and diffusion models have dramatically advanced deepfake technology, and its threats to digital security, media integrity, and public trust have increased rapidly. This research explored zero-shot deepfake detection, an emerging method even when the models have never seen a particular deepfake variation. In this work, we studied self-supervised learning, transformer-based zero-shot classifier, generative model fingerprinting, and meta-learning techniques that better adapt to the ever-evolving deepfake threat. In addition, we suggested AI-driven prevention strategies that mitigated the underlying generation pipeline of the deepfakes before they occurred. They consisted of adversarial perturbations for creating deepfake generators, digital watermarking for content authenticity verification, real-time AI monitoring for content creation pipelines, and blockchain-based content verification frameworks. Despite these advancements, zero-shot detection and prevention faced critical challenges such as adversarial attacks, scalability constraints, ethical dilemmas, and the absence of standardized evaluation benchmarks. These limitations were addressed by discussing future research directions on explainable AI for deepfake detection, multimodal fusion based on image, audio, and text analysis, quantum AI for enhanced security, and federated learning for privacy-preserving deepfake detection. This further highlighted the need for an integrated defense framework for digital authenticity that utilized zero-shot learning in combination with preventive deepfake mechanisms. Finally, we highlighted the important role of interdisciplinary collaboration between AI researchers, cybersecurity experts, and policymakers to create resilient defenses against the rising tide of deepfake attacks.

http://arxiv.org/abs/2509.18058
Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLM. (64%)
Alexander Panfilov; Evgenii Kortukov; Kristina Nikolić; Matthias Bethge; Sebastian Lapuschkin; Wojciech Samek; Ameya Prabhu; Maksym Andriushchenko; Jonas Geiping
Large language model (LLM) developers aim for their models to be honest, helpful, and harmless. However, when faced with malicious requests, models are trained to refuse, sacrificing helpfulness. We show that frontier LLMs can develop a preference for dishonesty as a new strategy, even when other options are available. Affected models respond to harmful requests with outputs that sound harmful but are subtly incorrect or otherwise harmless in practice. This behavior emerges with hard-to-predict variations even within models from the same model family. We find no apparent cause for the propensity to deceive, but we show that more capable models are better at executing this strategy. Strategic dishonesty already has a practical impact on safety evaluations, as we show that dishonest responses fool all output-based monitors used to detect jailbreaks that we test, rendering benchmark scores unreliable. Further, strategic dishonesty can act like a honeypot against malicious users, which noticeably obfuscates prior jailbreak attacks. While output monitors fail, we show that linear probes on internal activations can be used to reliably detect strategic dishonesty. We validate probes on datasets with verifiable outcomes and by using their features as steering vectors. Overall, we consider strategic dishonesty as a concrete example of a broader concern that alignment of LLMs is hard to control, especially when helpfulness and harmlessness conflict.

http://arxiv.org/abs/2509.17938
D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models. (31%)
Satyapriya Krishna; Andy Zou; Rahul Gupta; Eliot Krzysztof Jones; Nick Winter; Dan Hendrycks; J. Zico Kolter; Matt Fredrikson; Spyros Matsoukas
The safety and alignment of Large Language Models (LLMs) are critical for their responsible deployment. Current evaluation methods predominantly focus on identifying and preventing overtly harmful outputs. However, they often fail to address a more insidious failure mode: models that produce benign-appearing outputs while operating on malicious or deceptive internal reasoning. This vulnerability, often triggered by sophisticated system prompt injections, allows models to bypass conventional safety filters, posing a significant, underexplored risk. To address this gap, we introduce the Deceptive Reasoning Exposure Suite (D-REX), a novel dataset designed to evaluate the discrepancy between a model's internal reasoning process and its final output. D-REX was constructed through a competitive red-teaming exercise where participants crafted adversarial system prompts to induce such deceptive behaviors. Each sample in D-REX contains the adversarial system prompt, an end-user's test query, the model's seemingly innocuous response, and, crucially, the model's internal chain-of-thought, which reveals the underlying malicious intent. Our benchmark facilitates a new, essential evaluation task: the detection of deceptive alignment. We demonstrate that D-REX presents a significant challenge for existing models and safety mechanisms, highlighting the urgent need for new techniques that scrutinize the internal processes of LLMs, not just their final outputs.

http://arxiv.org/abs/2509.17898
Lipschitz-Based Robustness Certification for Recurrent Neural Networks via Convex Relaxation. (12%)
Paul Hamelbeck; Johannes Schiffer
Robustness certification against bounded input noise or adversarial perturbations is increasingly important for deployment recurrent neural networks (RNNs) in safety-critical control applications. To address this challenge, we present RNN-SDP, a relaxation based method that models the RNN's layer interactions as a convex problem and computes a certified upper bound on the Lipschitz constant via semidefinite programming (SDP). We also explore an extension that incorporates known input constraints to further tighten the resulting Lipschitz bounds. RNN-SDP is evaluated on a synthetic multi-tank system, with upper bounds compared to empirical estimates. While incorporating input constraints yields only modest improvements, the general method produces reasonably tight and certifiable bounds, even as sequence length increases. The results also underscore the often underestimated impact of initialization errors, an important consideration for applications where models are frequently re-initialized, such as model predictive control (MPC).

http://arxiv.org/abs/2509.21367
Design and Implementation of a Secure RAG-Enhanced AI Chatbot for Smart Tourism Customer Service: Defending Against Prompt Injection Attacks -- A Case Study of Hsinchu, Taiwan. (10%)
Yu-Kai Shih; You-Kai Kang
As smart tourism evolves, AI-powered chatbots have become indispensable for delivering personalized, real-time assistance to travelers while promoting sustainability and efficiency. However, these systems are increasingly vulnerable to prompt injection attacks, where adversaries manipulate inputs to elicit unintended behaviors such as leaking sensitive information or generating harmful content. This paper presents a case study on the design and implementation of a secure retrieval-augmented generation (RAG) chatbot for Hsinchu smart tourism services. The system integrates RAG with API function calls, multi-layered linguistic analysis, and guardrails against injections, achieving high contextual awareness and security. Key features include a tiered response strategy, RAG-driven knowledge grounding, and intent decomposition across lexical, semantic, and pragmatic levels. Defense mechanisms include system norms, gatekeepers for intent judgment, and reverse RAG text to prioritize verified data. We also benchmark a GPT-5 variant (released 2025-08-07) to assess inherent robustness. Evaluations with 674 adversarial prompts and 223 benign queries show over 95% accuracy on benign tasks and substantial detection of injection attacks. GPT-5 blocked about 85% of attacks, showing progress yet highlighting the need for layered defenses. Findings emphasize contributions to sustainable tourism, multilingual accessibility, and ethical AI deployment. This work offers a practical framework for deploying secure chatbots in smart tourism and contributes to resilient, trustworthy AI applications.

http://arxiv.org/abs/2509.17550
Is It Certainly a Deepfake? Reliability Analysis in Detection & Generation Ecosystem. (2%)
Neslihan Kose; Anthony Rhodes; Umur Aybars Ciftci; Ilke Demir
As generative models are advancing in quality and quantity for creating synthetic content, deepfakes begin to cause online mistrust. Deepfake detectors are proposed to counter this effect, however, misuse of detectors claiming fake content as real or vice versa further fuels this misinformation problem. We present the first comprehensive uncertainty analysis of deepfake detectors, systematically investigating how generative artifacts influence prediction confidence. As reflected in detectors' responses, deepfake generators also contribute to this uncertainty as their generative residues vary, so we cross the uncertainty analysis of deepfake detectors and generators. Based on our observations, the uncertainty manifold holds enough consistent information to leverage uncertainty for deepfake source detection. Our approach leverages Bayesian Neural Networks and Monte Carlo dropout to quantify both aleatoric and epistemic uncertainties across diverse detector architectures. We evaluate uncertainty on two datasets with nine generators, with four blind and two biological detectors, compare different uncertainty methods, explore region- and pixel-based uncertainty, and conduct ablation studies. We conduct and analyze binary real/fake, multi-class real/fake, source detection, and leave-one-out experiments between the generator/detector combinations to share their generalization capability, model calibration, uncertainty, and robustness against adversarial attacks. We further introduce uncertainty maps that localize prediction confidence at the pixel level, revealing distinct patterns correlated with generator-specific artifacts. Our analysis provides critical insights for deploying reliable deepfake detection systems and establishes uncertainty quantification as a fundamental requirement for trustworthy synthetic media detection.

http://arxiv.org/abs/2509.17371
SilentStriker:Toward Stealthy Bit-Flip Attacks on Large Language Models. (1%)
Haotian Xu; Qingsong Peng; Jie Shi; Huadi Zheng; Yu Li; Cheng Zhuo
The rapid adoption of large language models (LLMs) in critical domains has spurred extensive research into their security issues. While input manipulation attacks (e.g., prompt injection) have been well studied, Bit-Flip Attacks (BFAs) -- which exploit hardware vulnerabilities to corrupt model parameters and cause severe performance degradation -- have received far less attention. Existing BFA methods suffer from key limitations: they fail to balance performance degradation and output naturalness, making them prone to discovery. In this paper, we introduce SilentStriker, the first stealthy bit-flip attack against LLMs that effectively degrades task performance while maintaining output naturalness. Our core contribution lies in addressing the challenge of designing effective loss functions for LLMs with variable output length and the vast output space. Unlike prior approaches that rely on output perplexity for attack loss formulation, which inevitably degrade output naturalness, we reformulate the attack objective by leveraging key output tokens as targets for suppression, enabling effective joint optimization of attack effectiveness and stealthiness. Additionally, we employ an iterative, progressive search strategy to maximize attack efficacy. Experiments show that SilentStriker significantly outperforms existing baselines, achieving successful attacks without compromising the naturalness of generated text.

http://arxiv.org/abs/2502.19269
Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIP. (87%)
Jiawei Kong; Hao Fang; Sihang Guo; Chenxi Qing; Kuofeng Gao; Bin Chen; Shu-Tao Xia; Ke Xu
While pre-trained Vision-Language Models (VLMs) such as CLIP exhibit impressive representational capabilities for multimodal data, recent studies have revealed their vulnerability to backdoor attacks. To alleviate the threat, existing defense strategies primarily focus on fine-tuning the entire suspicious model. However, the substantial model parameters increase the difficulty of reaching a stable and consistent optimization direction, limiting their resistance against state-of-the-art attacks and often resulting in a degradation of clean accuracy. To address this challenge, we propose Class-wise Backdoor Prompt Tuning (CBPT), an efficient and effective defense mechanism that operates on text prompts to indirectly purify poisoned CLIP. Specifically, we first employ the advanced contrastive learning via carefully crafted positive and negative samples, to effectively invert the backdoor triggers that are potentially adopted by the attacker. Once the dummy trigger is established, we leverage three well-designed loss functions to optimize these class-wise text prompts, modifying the model's decision boundary and further reclassifying the feature regions affected by backdoor triggers. Extensive experiments demonstrate that CBPT significantly mitigates backdoor threats while preserving model utility, e.g. an average Clean Accuracy (CA) of 58.83% and an Attack Success Rate (ASR) of 0.39% across seven mainstream backdoor attacks. These results underscore the superiority of our prompt purifying design to strengthen CLIP's robustness against backdoor attacks.

http://arxiv.org/abs/2509.21360
Multimodal Prompt Decoupling Attack on the Safety Filters in Text-to-Image Models. (5%)
Xingkai Peng; Jun Jiang; Meng Tong; Shuai Li; Weiming Zhang; Nenghai Yu; Kejiang Chen
Text-to-image (T2I) models have been widely applied in generating high-fidelity images across various domains. However, these models may also be abused to produce Not-Safe-for-Work (NSFW) content via jailbreak attacks. Existing jailbreak methods primarily manipulate the textual prompt, leaving potential vulnerabilities in image-based inputs largely unexplored. Moreover, text-based methods face challenges in bypassing the model's safety filters. In response to these limitations, we propose the Multimodal Prompt Decoupling Attack (MPDA), which utilizes image modality to separate the harmful semantic components of the original unsafe prompt. MPDA follows three core steps: firstly, a large language model (LLM) decouples unsafe prompts into pseudo-safe prompts and harmful prompts. The former are seemingly harmless sub-prompts that can bypass filters, while the latter are sub-prompts with unsafe semantics that trigger filters. Subsequently, the LLM rewrites the harmful prompts into natural adversarial prompts to bypass safety filters, which guide the T2I model to modify the base image into an NSFW output. Finally, to ensure semantic consistency between the generated NSFW images and the original unsafe prompts, the visual language model generates image captions, providing a new pathway to guide the LLM in iterative rewriting and refining the generated content.

http://arxiv.org/abs/2509.20383
MARS: A Malignity-Aware Backdoor Defense in Federated Learning. (1%)
Wei Wan; Yuxuan Ning; Zhicong Huang; Cheng Hong; Shengshan Hu; Ziqi Zhou; Yechao Zhang; Tianqing Zhu; Wanlei Zhou; Leo Yu Zhang
Federated Learning (FL) is a distributed paradigm aimed at protecting participant data privacy by exchanging model parameters to achieve high-quality model training. However, this distributed nature also makes FL highly vulnerable to backdoor attacks. Notably, the recently proposed state-of-the-art (SOTA) attack, 3DFed (SP2023), uses an indicator mechanism to determine whether the backdoor models have been accepted by the defender and adaptively optimizes backdoor models, rendering existing defenses ineffective. In this paper, we first reveal that the failure of existing defenses lies in the employment of empirical statistical measures that are loosely coupled with backdoor attacks. Motivated by this, we propose a Malignity-Aware backdooR defenSe (MARS) that leverages backdoor energy (BE) to indicate the malicious extent of each neuron. To amplify malignity, we further extract the most prominent BE values from each model to form a concentrated backdoor energy (CBE). Finally, a novel Wasserstein distance-based clustering method is introduced to effectively identify backdoor models. Extensive experiments demonstrate that MARS can defend against SOTA backdoor attacks and significantly outperforms existing defenses.

http://arxiv.org/abs/2507.22832
Pulling Back the Curtain on ReLU Networks. (1%)
Maciej Satkiewicz
Since any ReLU network is piecewise affine, its hidden units can be characterized by their pullbacks through the active subnetwork, i.e., by their gradients (up to bias terms). However, gradients of deeper neurons are notoriously misaligned, which obscures the network's internal representations. We posit that models do align gradients with data, yet this is concealed by the intrinsic noise of the ReLU hard gating. We validate this intuition by applying soft gating in the backward pass only, reducing the local impact of weakly excited neurons. The resulting modified gradients, which we call "excitation pullbacks", exhibit striking perceptual alignment on a number of ImageNet-pretrained architectures, while the rudimentary pixel-space gradient ascent quickly produces easily interpretable input- and target-specific features. Inspired by these findings, we formulate the "path stability" hypothesis, claiming that the binary activation patterns largely stabilize during training and get encoded in the pre-activation distribution of the final model. When true, excitation pullbacks become aligned with the gradients of a kernel machine that mainly determines the network's decision. This provides a theoretical justification for the apparent faithfulness of the feature attributions based on excitation pullbacks, potentially even leading to mechanistic interpretability of deep models. Incidentally, we give a possible explanation for the effectiveness of Batch Normalization and Deep Features, together with a novel perspective on the network's internal memory and generalization properties. We release the code and an interactive app for easier exploration of the excitation pullbacks.

http://arxiv.org/abs/2508.17007
An Efficient Dual-Line Decoder Network with Multi-Scale Convolutional Attention for Multi-organ Segmentation. (1%)
Riad Hassan; M. Rubaiyat Hossain Mondal; Sheikh Iqbal Ahamed; Fahad Mostafa; Md Mostafijur Rahman
Proper segmentation of organs-at-risk is important for radiation therapy, surgical planning, and diagnostic decision-making in medical image analysis. While deep learning-based segmentation architectures have made significant progress, they often fail to balance segmentation accuracy with computational efficiency. Most of the current state-of-the-art methods either prioritize performance at the cost of high computational complexity or compromise accuracy for efficiency. This paper addresses this gap by introducing an efficient dual-line decoder segmentation network (EDLDNet). The proposed method features a noisy decoder, which learns to incorporate structured perturbation at training time for better model robustness, yet at inference time only the noise-free decoder is executed, leading to lower computational cost. Multi-Scale convolutional Attention Modules (MSCAMs), Attention Gates (AGs), and Up-Convolution Blocks (UCBs) are further utilized to optimize feature representation and boost segmentation performance. By leveraging multi-scale segmentation masks from both decoders, we also utilize a mutation-based loss function to enhance the model's generalization. Our approach outperforms SOTA segmentation architectures on four publicly available medical imaging datasets. EDLDNet achieves SOTA performance with an 84.00% Dice score on the Synapse dataset, surpassing baseline model like UNet by 13.89% in Dice score while significantly reducing Multiply-Accumulate Operations (MACs) by 89.7%. Compared to recent approaches like EMCAD, our EDLDNet not only achieves higher Dice score but also maintains comparable computational efficiency. The outstanding performance across diverse datasets establishes EDLDNet's strong generalization, computational efficiency, and robustness. The source code, pre-processed data, and pre-trained weights will be available at https://github.com/riadhassan/EDLDNet .

http://arxiv.org/abs/2509.17070
Localizing Malicious Outputs from CodeLLM. (1%)
Mayukh Borana; Junyi Liang; Sai Sathiesh Rajan; Sudipta Chattopadhyay
We introduce FreqRank, a mutation-based defense to localize malicious components in LLM outputs and their corresponding backdoor triggers. FreqRank assumes that the malicious sub-string(s) consistently appear in outputs for triggered inputs and uses a frequency-based ranking system to identify them. Our ranking system then leverages this knowledge to localize the backdoor triggers present in the inputs. We create nine malicious models through fine-tuning or custom instructions for three downstream tasks, namely, code completion (CC), code generation (CG), and code summarization (CS), and show that they have an average attack success rate (ASR) of 86.6%. Furthermore, FreqRank's ranking system highlights the malicious outputs as one of the top five suggestions in 98% of cases. We also demonstrate that FreqRank's effectiveness scales as the number of mutants increases and show that FreqRank is capable of localizing the backdoor trigger effectively even with a limited number of triggered samples. Finally, we show that our approach is 35-50% more effective than other defense methods.

http://arxiv.org/abs/2509.16645
ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agents. (92%)
Yichen Wang; Hangtao Zhang; Hewen Pan; Ziqi Zhou; Xianlong Wang; Peijin Guo; Lulu Xue; Shengshan Hu; Minghui Li; Leo Yu Zhang
Vision-Language Models (VLMs), with their strong reasoning and planning capabilities, are widely used in embodied decision-making (EDM) tasks in embodied agents, such as autonomous driving and robotic manipulation. Recent research has increasingly explored adversarial attacks on VLMs to reveal their vulnerabilities. However, these attacks either rely on overly strong assumptions, requiring full knowledge of the victim VLM, which is impractical for attacking VLM-based agents, or exhibit limited effectiveness. The latter stems from disrupting most semantic information in the image, which leads to a misalignment between the perception and the task context defined by system prompts. This inconsistency interrupts the VLM's reasoning process, resulting in invalid outputs that fail to affect interactions in the physical world. To this end, we propose a fine-grained adversarial attack framework, ADVEDM, which modifies the VLM's perception of only a few key objects while preserving the semantics of the remaining regions. This attack effectively reduces conflicts with the task context, making VLMs output valid but incorrect decisions and affecting the actions of agents, thus posing a more substantial safety threat in the physical world. We design two variants of based on this framework, ADVEDM-R and ADVEDM-A, which respectively remove the semantics of a specific object from the image and add the semantics of a new object into the image. The experimental results in both general scenarios and EDM tasks demonstrate fine-grained control and excellent attack performance.

http://arxiv.org/abs/2509.16494
Can an Individual Manipulate the Collective Decisions of Multi-Agents? (74%)
Fengyuan Liu; Rui Zhao; Shuo Chen; Guohao Li; Philip Torr; Lei Han; Jindong Gu
Individual Large Language Models (LLMs) have demonstrated significant capabilities across various domains, such as healthcare and law. Recent studies also show that coordinated multi-agent systems exhibit enhanced decision-making and reasoning abilities through collaboration. However, due to the vulnerabilities of individual LLMs and the difficulty of accessing all agents in a multi-agent system, a key question arises: If attackers only know one agent, could they still generate adversarial samples capable of misleading the collective decision? To explore this question, we formulate it as a game with incomplete information, where attackers know only one target agent and lack knowledge of the other agents in the system. With this formulation, we propose M-Spoiler, a framework that simulates agent interactions within a multi-agent system to generate adversarial samples. These samples are then used to manipulate the target agent in the target system, misleading the system's collaborative decision-making process. More specifically, M-Spoiler introduces a stubborn agent that actively aids in optimizing adversarial samples by simulating potential stubborn responses from agents in the target system. This enhances the effectiveness of the generated adversarial samples in misleading the system. Through extensive experiments across various tasks, our findings confirm the risks posed by the knowledge of an individual agent in multi-agent systems and demonstrate the effectiveness of our framework. We also explore several defense mechanisms, showing that our proposed attack framework remains more potent than baselines, underscoring the need for further research into defensive strategies.

http://arxiv.org/abs/2509.16546
Train to Defend: First Defense Against Cryptanalytic Neural Network Parameter Extraction Attacks. (68%)
Ashley Kurian; Aydin Aysu
Neural networks are valuable intellectual property due to the significant computational cost, expert labor, and proprietary data involved in their development. Consequently, protecting their parameters is critical not only for maintaining a competitive advantage but also for enhancing the model's security and privacy. Prior works have demonstrated the growing capability of cryptanalytic attacks to scale to deeper models. In this paper, we present the first defense mechanism against cryptanalytic parameter extraction attacks. Our key insight is to eliminate the neuron uniqueness necessary for these attacks to succeed. We achieve this by a novel, extraction-aware training method. Specifically, we augment the standard loss function with an additional regularization term that minimizes the distance between neuron weights within a layer. Therefore, the proposed defense has zero area-delay overhead during inference. We evaluate the effectiveness of our approach in mitigating extraction attacks while analyzing the model accuracy across different architectures and datasets. When re-trained with the same model architecture, the results show that our defense incurs a marginal accuracy change of less than 1% with the modified loss function. Moreover, we present a theoretical framework to quantify the success probability of the attack. When tested comprehensively with prior attack settings, our defense demonstrated empirical success for sustained periods of extraction, whereas unprotected networks are extracted between 14 minutes to 4 hours.

http://arxiv.org/abs/2502.21059
FC-Attack: Jailbreaking Multimodal Large Language Models via Auto-Generated Flowcharts. (68%)
Ziyi Zhang; Zhen Sun; Zongmin Zhang; Jihui Guo; Xinlei He
Multimodal Large Language Models (MLLMs) have become powerful and widely adopted in some practical applications. However, recent research has revealed their vulnerability to multimodal jailbreak attacks, whereby the model can be induced to generate harmful content, leading to safety risks. Although most MLLMs have undergone safety alignment, recent research shows that the visual modality is still vulnerable to jailbreak attacks. In our work, we discover that by using flowcharts with partially harmful information, MLLMs can be induced to provide additional harmful details. Based on this, we propose a jailbreak attack method based on auto-generated flowcharts, FC-Attack. Specifically, FC-Attack first fine-tunes a pre-trained LLM to create a step-description generator based on benign datasets. The generator is then used to produce step descriptions corresponding to a harmful query, which are transformed into flowcharts in 3 different shapes (vertical, horizontal, and S-shaped) as visual prompts. These flowcharts are then combined with a benign textual prompt to execute the jailbreak attack on MLLMs. Our evaluations on Advbench show that FC-Attack attains an attack success rate of up to 96% via images and up to 78% via videos across multiple MLLMs. Additionally, we investigate factors affecting the attack performance, including the number of steps and the font styles in the flowcharts. We also find that FC-Attack can improve the jailbreak performance from 4% to 28% in Claude-3.5 by changing the font style. To mitigate the attack, we explore several defenses and find that AdaShield can largely reduce the jailbreak performance but with the cost of utility drop.

http://arxiv.org/abs/2509.16671
"Digital Camouflage": The LLVM Challenge in LLM-Based Malware Detection. (26%)
Ekin Böke; Simon Torka
Large Language Models (LLMs) have emerged as promising tools for malware detection by analyzing code semantics, identifying vulnerabilities, and adapting to evolving threats. However, their reliability under adversarial compiler-level obfuscation is yet to be discovered. In this study, we empirically evaluate the robustness of three state-of-the-art LLMs: ChatGPT-4o, Gemini Flash 2.5, and Claude Sonnet 4 against compiler-level obfuscation techniques implemented via the LLVM infrastructure. These include control flow flattening, bogus control flow injection, instruction substitution, and split basic blocks, which are widely used to evade detection while preserving malicious behavior. We perform a structured evaluation on 40~C functions (20 vulnerable, 20 secure) sourced from the Devign dataset and obfuscated using LLVM passes. Our results show that these models often fail to correctly classify obfuscated code, with precision, recall, and F1-score dropping significantly after transformation. This reveals a critical limitation: LLMs, despite their language understanding capabilities, can be easily misled by compiler-based obfuscation strategies. To promote reproducibility, we release all evaluation scripts, prompts, and obfuscated code samples in a public repository. We also discuss the implications of these findings for adversarial threat modeling, and outline future directions such as software watermarking, compiler-aware defenses, and obfuscation-resilient model design.

http://arxiv.org/abs/2509.16678
IPF-RDA: An Information-Preserving Framework for Robust Data Augmentation. (1%)
Suorong Yang; Hongchao Yang; Suhan Guo; Furao Shen; Jian Zhao
Data augmentation is widely utilized as an effective technique to enhance the generalization performance of deep models. However, data augmentation may inevitably introduce distribution shifts and noises, which significantly constrain the potential and deteriorate the performance of deep networks. To this end, we propose a novel information-preserving framework, namely IPF-RDA, to enhance the robustness of data augmentations in this paper. IPF-RDA combines the proposal of (i) a new class-discriminative information estimation algorithm that identifies the points most vulnerable to data augmentation operations and corresponding importance scores; And (ii) a new information-preserving scheme that preserves the critical information in the augmented samples and ensures the diversity of augmented data adaptively. We divide data augmentation methods into three categories according to the operation types and integrate these approaches into our framework accordingly. After being integrated into our framework, the robustness of data augmentation methods can be enhanced and their full potential can be unleashed. Extensive experiments demonstrate that although being simple, IPF-RDA consistently improves the performance of numerous commonly used state-of-the-art data augmentation methods with popular deep models on a variety of datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, CUHK03, Market1501, Oxford Flower, and MNIST, where its performance and scalability are stressed. The implementation is available at https://github.com/Jackbrocp/IPF-RDA.

http://arxiv.org/abs/2506.06273
AdvSumm: Adversarial Training for Bias Mitigation in Text Summarization. (1%)
Mukur Gupta; Nikhil Reddy Varimalla; Nicholas Deas; Melanie Subbiah; Kathleen McKeown
Large Language Models (LLMs) have achieved impressive performance in text summarization and are increasingly deployed in real-world applications. However, these systems often inherit associative and framing biases from pre-training data, leading to inappropriate or unfair outputs in downstream tasks. In this work, we present AdvSumm (Adversarial Summarization), a domain-agnostic training framework designed to mitigate bias in text summarization through improved generalization. Inspired by adversarial robustness, AdvSumm introduces a novel Perturber component that applies gradient-guided perturbations at the embedding level of Sequence-to-Sequence models, enhancing the model's robustness to input variations. We empirically demonstrate that AdvSumm effectively reduces different types of bias in summarization-specifically, name-nationality bias and political framing bias-without compromising summarization quality. Compared to standard transformers and data augmentation techniques like back-translation, AdvSumm achieves stronger bias mitigation performance across benchmark datasets.

http://arxiv.org/abs/2509.16352
Secure Confidential Business Information When Sharing Machine Learning Models. (82%)
Yunfan Yang; Jiarong Xu; Hongzhe Zhang; Xiao Fang
Model-sharing offers significant business value by enabling firms with well-established Machine Learning (ML) models to monetize and share their models with others who lack the resources to develop ML models from scratch. However, concerns over data confidentiality remain a significant barrier to model-sharing adoption, as Confidential Property Inference (CPI) attacks can exploit shared ML models to uncover confidential properties of the model provider's private model training data. Existing defenses often assume that CPI attacks are non-adaptive to the specific ML model they are targeting. This assumption overlooks a key characteristic of real-world adversaries: their responsiveness, i.e., adversaries' ability to dynamically adjust their attack models based on the information of the target and its defenses. To overcome this limitation, we propose a novel defense method that explicitly accounts for the responsive nature of real-world adversaries via two methodological innovations: a novel Responsive CPI attack and an attack-defense arms race framework. The former emulates the responsive behaviors of adversaries in the real world, and the latter iteratively enhances both the target and attack models, ultimately producing a secure ML model that is robust against responsive CPI attacks. Furthermore, we propose and integrate a novel approximate strategy into our defense, which addresses a critical computational bottleneck of defense methods and improves defense efficiency. Through extensive empirical evaluations across various realistic model-sharing scenarios, we demonstrate that our method outperforms existing defenses by more effectively defending against CPI attacks, preserving ML model utility, and reducing computational overhead.

http://arxiv.org/abs/2501.16534
Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs. (80%)
Jean-Charles Noirot Ferrand; Yohan Beugin; Eric Pauley; Ryan Sheatsley; Patrick McDaniel
Alignment in large language models (LLMs) is used to enforce guidelines such as safety. Yet, alignment fails in the face of jailbreak attacks that modify inputs to induce unsafe outputs. In this paper, we introduce and evaluate a new technique for jailbreak attacks. We observe that alignment embeds a safety classifier in the LLM responsible for deciding between refusal and compliance, and seek to extract an approximation of this classifier: a surrogate classifier. To this end, we build candidate classifiers from subsets of the LLM. We first evaluate the degree to which candidate classifiers approximate the LLM's safety classifier in benign and adversarial settings. Then, we attack the candidates and measure how well the resulting adversarial inputs transfer to the LLM. Our evaluation shows that the best candidates achieve accurate agreement (an F1 score above 80%) using as little as 20% of the model architecture. Further, we find that attacks mounted on the surrogate classifiers can be transferred to the LLM with high success. For example, a surrogate using only 50% of the Llama 2 model achieved an attack success rate (ASR) of 70% with half the memory footprint and runtime -- a substantial improvement over attacking the LLM directly, where we only observed a 22% ASR. These results show that extracting surrogate classifiers is an effective and efficient means for modeling (and therein addressing) the vulnerability of aligned models to jailbreaking attacks.

http://arxiv.org/abs/2509.15756
An Adversarial Robust Behavior Sequence Anomaly Detection Approach Based on Critical Behavior Unit Learning. (70%)
Dongyang Zhan; Kai Tan; Lin Ye; Xiangzhan Yu; Hongli Zhang; Zheng He
Sequential deep learning models (e.g., RNN and LSTM) can learn the sequence features of software behaviors, such as API or syscall sequences. However, recent studies have shown that these deep learning-based approaches are vulnerable to adversarial samples. Attackers can use adversarial samples to change the sequential characteristics of behavior sequences and mislead malware classifiers. In this paper, an adversarial robustness anomaly detection method based on the analysis of behavior units is proposed to overcome this problem. We extract related behaviors that usually perform a behavior intention as a behavior unit, which contains the representative semantic information of local behaviors and can be used to improve the robustness of behavior analysis. By learning the overall semantics of each behavior unit and the contextual relationships among behavior units based on a multilevel deep learning model, our approach can mitigate perturbation attacks that target local and large-scale behaviors. In addition, our approach can be applied to both low-level and high-level behavior logs (e.g., API and syscall logs). The experimental results show that our approach outperforms all the compared methods, which indicates that our approach has better performance against obfuscation attacks.

http://arxiv.org/abs/2509.15499
Adversarially Robust Assembly Language Model for Packed Executables Detection. (67%)
Shijia Li; Jiang Ming; Lanqing Liu; Longwei Yang; Ni Zhang; Chunfu Jia
Detecting packed executables is a critical component of large-scale malware analysis and antivirus engine workflows, as it identifies samples that warrant computationally intensive dynamic unpacking to reveal concealed malicious behavior. Traditionally, packer detection techniques have relied on empirical features, such as high entropy or specific binary patterns. However, these empirical, feature-based methods are increasingly vulnerable to evasion by adversarial samples or unknown packers (e.g., low-entropy packers). Furthermore, the dependence on expert-crafted features poses challenges in sustaining and evolving these methods over time.
  In this paper, we examine the limitations of existing packer detection methods and propose Pack-ALM, a novel deep-learning-based approach for detecting packed executables. Inspired by the linguistic concept of distinguishing between real and pseudo words, we reformulate packer detection as a task of differentiating between legitimate and "pseudo" instructions. To achieve this, we preprocess native data and packed data into "pseudo" instructions and design a pre-trained assembly language model that recognizes features indicative of packed data. We evaluate Pack-ALM against leading industrial packer detection tools and state-of-the-art assembly language models. Extensive experiments on over 37,000 samples demonstrate that Pack-ALM effectively identifies packed binaries, including samples created with adversarial or previously unseen packing techniques. Moreover, Pack-ALM outperforms traditional entropy-based methods and advanced assembly language models in both detection accuracy and adversarial robustness.

http://arxiv.org/abs/2509.16163
Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks. (41%)
Het Patel; Muzammil Allie; Qian Zhang; Jia Chen; Evangelos E. Papalexakis
Vision language models (VLMs) excel in multimodal understanding but are prone to adversarial attacks. Existing defenses often demand costly retraining or significant architecture changes. We introduce a lightweight defense using tensor decomposition suitable for any pre-trained VLM, requiring no retraining. By decomposing and reconstructing vision encoder representations, it filters adversarial noise while preserving meaning. Experiments with CLIP on COCO and Flickr30K show improved robustness. On Flickr30K, it restores 12.3\% performance lost to attacks, raising Recall@1 accuracy from 7.5\% to 19.8\%. On COCO, it recovers 8.1\% performance, improving accuracy from 3.8\% to 11.9\%. Analysis shows Tensor Train decomposition with low rank (8-32) and low residual strength ($α=0.1-0.2$) is optimal. This method is a practical, plug-and-play solution with minimal overhead for existing VLMs.

http://arxiv.org/abs/2405.18499
Training More Robust Classification Model via Discriminative Loss and Gaussian Noise Injection. (38%)
Hai-Vy Nguyen; Fabrice Gamboa; Sixin Zhang; Reda Chhaibi; Serge Gratton; Thierry Giaccone
Robustness of deep neural networks to input noise remains a critical challenge, as naive noise injection often degrades accuracy on clean (uncorrupted) data. We propose a novel training framework that addresses this trade-off through two complementary objectives. First, we introduce a loss function applied at the penultimate layer that explicitly enforces intra-class compactness and increases the margin to analytically defined decision boundaries. This enhances feature discriminativeness and class separability for clean data. Second, we propose a class-wise feature alignment mechanism that brings noisy data clusters closer to their clean counterparts. Furthermore, we provide a theoretical analysis demonstrating that improving feature stability under additive Gaussian noise implicitly reduces the curvature of the softmax loss landscape in input space, as measured by Hessian eigenvalues.This thus naturally enhances robustness without explicit curvature penalties. Conversely, we also theoretically show that lower curvatures lead to more robust models. We validate the effectiveness of our method on standard benchmarks and our custom dataset. Our approach significantly reinforces model robustness to various perturbations while maintaining high accuracy on clean data, advancing the understanding and practice of noise-robust deep learning.

http://arxiv.org/abs/2509.15551
PolyJuice Makes It Real: Black-Box, Universal Red Teaming for Synthetic Image Detectors. (26%)
Sepehr Dehdashtian; Mashrur M. Morshed; Jacob H. Seidman; Gaurav Bharaj; Vishnu Naresh Boddeti
Synthetic image detectors (SIDs) are a key defense against the risks posed by the growing realism of images from text-to-image (T2I) models. Red teaming improves SID's effectiveness by identifying and exploiting their failure modes via misclassified synthetic images. However, existing red-teaming solutions (i) require white-box access to SIDs, which is infeasible for proprietary state-of-the-art detectors, and (ii) generate image-specific attacks through expensive online optimization. To address these limitations, we propose PolyJuice, the first black-box, image-agnostic red-teaming method for SIDs, based on an observed distribution shift in the T2I latent space between samples correctly and incorrectly classified by the SID. PolyJuice generates attacks by (i) identifying the direction of this shift through a lightweight offline process that only requires black-box access to the SID, and (ii) exploiting this direction by universally steering all generated images towards the SID's failure modes. PolyJuice-steered T2I models are significantly more effective at deceiving SIDs (up to 84%) compared to their unsteered counterparts. We also show that the steering directions can be estimated efficiently at lower resolutions and transferred to higher resolutions using simple interpolation, reducing computational overhead. Finally, tuning SID models on PolyJuice-augmented datasets notably enhances the performance of the detectors (up to 30%).

http://arxiv.org/abs/2509.15497
Backdoor Mitigation via Invertible Pruning Masks. (5%)
Kealan Dunnett; Reza Arablouei; Dimity Miller; Volkan Dedeoglu; Raja Jurdak
Model pruning has gained traction as a promising defense strategy against backdoor attacks in deep learning. However, existing pruning-based approaches often fall short in accurately identifying and removing the specific parameters responsible for inducing backdoor behaviors. Despite the dominance of fine-tuning-based defenses in recent literature, largely due to their superior performance, pruning remains a compelling alternative, offering greater interpretability and improved robustness in low-data regimes. In this paper, we propose a novel pruning approach featuring a learned \emph{selection} mechanism to identify parameters critical to both main and backdoor tasks, along with an \emph{invertible} pruning mask designed to simultaneously achieve two complementary goals: eliminating the backdoor task while preserving it through the inverse mask. We formulate this as a bi-level optimization problem that jointly learns selection variables, a sparse invertible mask, and sample-specific backdoor perturbations derived from clean data. The inner problem synthesizes candidate triggers using the inverse mask, while the outer problem refines the mask to suppress backdoor behavior without impairing clean-task accuracy. Extensive experiments demonstrate that our approach outperforms existing pruning-based backdoor mitigation approaches, maintains strong performance under limited data conditions, and achieves competitive results compared to state-of-the-art fine-tuning approaches. Notably, the proposed approach is particularly effective in restoring correct predictions for compromised samples after successful backdoor mitigation.

http://arxiv.org/abs/2509.16088
Randomized Smoothing Meets Vision-Language Models. (4%)
Emmanouil Seferis; Changshun Wu; Stefanos Kollias; Saddek Bensalem; Chih-Hong Cheng
Randomized smoothing (RS) is one of the prominent techniques to ensure the correctness of machine learning models, where point-wise robustness certificates can be derived analytically. While RS is well understood for classification, its application to generative models is unclear, since their outputs are sequences rather than labels. We resolve this by connecting generative outputs to an oracle classification task and showing that RS can still be enabled: the final response can be classified as a discrete action (e.g., service-robot commands in VLAs), as harmful vs. harmless (content moderation or toxicity detection in VLMs), or even applying oracles to cluster answers into semantically equivalent ones. Provided that the error rate for the oracle classifier comparison is bounded, we develop the theory that associates the number of samples with the corresponding robustness radius. We further derive improved scaling laws analytically relating the certified radius and accuracy to the number of samples, showing that the earlier result of 2 to 3 orders of magnitude fewer samples sufficing with minimal loss remains valid even under weaker assumptions. Together, these advances make robustness certification both well-defined and computationally feasible for state-of-the-art VLMs, as validated against recent jailbreak-style adversarial attacks.

http://arxiv.org/abs/2509.16058
Attention Schema-based Attention Control (ASAC): A Cognitive-Inspired Approach for Attention Management in Transformers. (2%)
Krati Saxena; Federico Jurado Ruiz; Guido Manzi; Dianbo Liu; Alex Lamb
Attention mechanisms have become integral in AI, significantly enhancing model performance and scalability by drawing inspiration from human cognition. Concurrently, the Attention Schema Theory (AST) in cognitive science posits that individuals manage their attention by creating a model of the attention itself, effectively allocating cognitive resources. Inspired by AST, we introduce ASAC (Attention Schema-based Attention Control), which integrates the attention schema concept into artificial neural networks. Our initial experiments focused on embedding the ASAC module within transformer architectures. This module employs a Vector-Quantized Variational AutoEncoder (VQVAE) as both an attention abstractor and controller, facilitating precise attention management. By explicitly modeling attention allocation, our approach aims to enhance system efficiency. We demonstrate ASAC's effectiveness in both the vision and NLP domains, highlighting its ability to improve classification accuracy and expedite the learning process. Our experiments with vision transformers across various datasets illustrate that the attention controller not only boosts classification accuracy but also accelerates learning. Furthermore, we have demonstrated the model's robustness and generalization capabilities across noisy and out-of-distribution datasets. In addition, we have showcased improved performance in multi-task settings. Quick experiments reveal that the attention schema-based module enhances resilience to adversarial attacks, optimizes attention to improve learning efficiency, and facilitates effective transfer learning and learning from fewer examples. These promising results establish a connection between cognitive science and machine learning, shedding light on the efficient utilization of attention mechanisms in AI systems.

http://arxiv.org/abs/2508.15987
PickleBall: Secure Deserialization of Pickle-based Machine Learning Models (Extended Report). (1%)
Andreas D. Kellas; Neophytos Christou; Wenxin Jiang; Penghui Li; Laurent Simon; Yaniv David; Vasileios P. Kemerlis; James C. Davis; Junfeng Yang
Machine learning model repositories such as the Hugging Face Model Hub facilitate model exchanges. However, bad actors can deliver malware through compromised models. Existing defenses such as safer model formats, restrictive (but inflexible) loading policies, and model scanners have shortcomings: 44.9% of popular models on Hugging Face still use the insecure pickle format, 15% of these cannot be loaded by restrictive loading policies, and model scanners have both false positives and false negatives. Pickle remains the de facto standard for model exchange, and the ML community lacks a tool that offers transparent safe loading.
  We present PickleBall to help machine learning engineers load pickle-based models safely. PickleBall statically analyzes the source code of a given machine learning library and computes a custom policy that specifies a safe load-time behavior for benign models. PickleBall then dynamically enforces the policy during load time as a drop-in replacement for the pickle module. PickleBall generates policies that correctly load 79.8% of benign pickle-based models in our dataset, while rejecting all (100%) malicious examples in our dataset. In comparison, evaluated model scanners fail to identify known malicious models, and the state-of-art loader loads 22% fewer benign models than PickleBall. PickleBall removes the threat of arbitrary function invocation from malicious pickle-based models, raising the bar for attackers to depend on code reuse techniques.

http://arxiv.org/abs/2509.16060
SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection. (1%)
Maithili Joshi; Palash Nandi; Tanmoy Chakraborty
Large Language Models (LLMs) with safe-alignment training are powerful instruments with robust language comprehension capabilities. These models typically undergo meticulous alignment procedures involving human feedback to ensure the acceptance of safe inputs while rejecting harmful or unsafe ones. However, despite their massive scale and alignment efforts, LLMs remain vulnerable to jailbreak attacks, where malicious users manipulate the model to produce harmful outputs that it was explicitly trained to avoid. In this study, we find that the safety mechanisms in LLMs are predominantly embedded in the middle-to-late layers. Building on this insight, we introduce a novel white-box jailbreak method, SABER (Safety Alignment Bypass via Extra Residuals), which connects two intermediate layers $s$ and $e$ such that $s < e$, through a residual connection. Our approach achieves a 51% improvement over the best-performing baseline on the HarmBench test set. Furthermore, SABER induces only a marginal shift in perplexity when evaluated on the HarmBench validation set. The source code is publicly available at https://github.com/PalGitts/SABER.

http://arxiv.org/abs/2509.15572
Cuckoo Attack: Stealthy and Persistent Attacks Against AI-IDE. (1%)
Xinpeng Liu; Junming Liu; Peiyu Liu; Han Zheng; Qinying Wang; Mathias Payer; Shouling Ji; Wenhai Wang
Modern AI-powered Integrated Development Environments (AI-IDEs) are increasingly defined by an Agent-centric architecture, where an LLM-powered Agent is deeply integrated to autonomously execute complex tasks. This tight integration, however, also introduces a new and critical attack surface. Attackers can exploit these components by injecting malicious instructions into untrusted external sources, effectively hijacking the Agent to perform harmful operations beyond the user's intention or awareness. This emerging threat has quickly attracted research attention, leading to various proposed attack vectors, such as hijacking Model Context Protocol (MCP) Servers to access private data. However, most existing approaches lack stealth and persistence, limiting their practical impact.
  We propose the Cuckoo Attack, a novel attack that achieves stealthy and persistent command execution by embedding malicious payloads into configuration files. These files, commonly used in AI-IDEs, execute system commands during routine operations, without displaying execution details to the user. Once configured, such files are rarely revisited unless an obvious runtime error occurs, creating a blind spot for attackers to exploit. We formalize our attack paradigm into two stages, including initial infection and persistence. Based on these stages, we analyze the practicality of the attack execution process and identify the relevant exploitation techniques. Furthermore, we analyze the impact of Cuckoo Attack, which can not only invade the developer's local computer but also achieve supply chain attacks through the spread of configuration files. We contribute seven actionable checkpoints for vendors to evaluate their product security. The critical need for these checks is demonstrated by our end-to-end Proof of Concept, which validated the proposed attack across nine mainstream Agent and AI-IDE pairs.

http://arxiv.org/abs/2509.15437
Impact of Phonetics on Speaker Identity in Adversarial Voice Attack. (99%)
Daniyal Kabir Dar; Qiben Yan; Li Xiao; Arun Ross
Adversarial perturbations in speech pose a serious threat to automatic speech recognition (ASR) and speaker verification by introducing subtle waveform modifications that remain imperceptible to humans but can significantly alter system outputs. While targeted attacks on end-to-end ASR models have been widely studied, the phonetic basis of these perturbations and their effect on speaker identity remain underexplored. In this work, we analyze adversarial audio at the phonetic level and show that perturbations exploit systematic confusions such as vowel centralization and consonant substitutions. These distortions not only mislead transcription but also degrade phonetic cues critical for speaker verification, leading to identity drift. Using DeepSpeech as our ASR target, we generate targeted adversarial examples and evaluate their impact on speaker embeddings across genuine and impostor samples. Results across 16 phonetically diverse target phrases demonstrate that adversarial audio induces both transcription errors and identity drift, highlighting the need for phonetic-aware defenses to ensure the robustness of ASR and speaker recognition systems.

http://arxiv.org/abs/2509.15435
ORCA: Agentic Reasoning For Hallucination and Adversarial Robustness in Vision-Language Models. (97%)
Chung-En Johnny Yu; Hsuan-Chih; Chen; Brian Jalaian; Nathaniel D. Bastian
Large Vision-Language Models (LVLMs) exhibit strong multimodal capabilities but remain vulnerable to hallucinations from intrinsic errors and adversarial attacks from external exploitations, limiting their reliability in real-world applications. We present ORCA, an agentic reasoning framework that improves the factual accuracy and adversarial robustness of pretrained LVLMs through test-time structured inference reasoning with a suite of small vision models (less than 3B parameters). ORCA operates via an Observe--Reason--Critique--Act loop, querying multiple visual tools with evidential questions, validating cross-model inconsistencies, and refining predictions iteratively without access to model internals or retraining. ORCA also stores intermediate reasoning traces, which supports auditable decision-making. Though designed primarily to mitigate object-level hallucinations, ORCA also exhibits emergent adversarial robustness without requiring adversarial training or defense mechanisms. We evaluate ORCA across three settings: (1) clean images on hallucination benchmarks, (2) adversarially perturbed images without defense, and (3) adversarially perturbed images with defense applied. On the POPE hallucination benchmark, ORCA improves standalone LVLM performance by +3.64\% to +40.67\% across different subsets. Under adversarial perturbations on POPE, ORCA achieves an average accuracy gain of +20.11\% across LVLMs. When combined with defense techniques on adversarially perturbed AMBER images, ORCA further improves standalone LVLM performance, with gains ranging from +1.20\% to +48.00\% across evaluation metrics. These results demonstrate that ORCA offers a promising path toward building more reliable and robust multimodal systems.

http://arxiv.org/abs/2509.15370
Adversarial generalization of unfolding (model-based) networks. (82%)
Vicky Kouni
Unfolding networks are interpretable networks emerging from iterative algorithms, incorporate prior knowledge of data structure, and are designed to solve inverse problems like compressed sensing, which deals with recovering data from noisy, missing observations. Compressed sensing finds applications in critical domains, from medical imaging to cryptography, where adversarial robustness is crucial to prevent catastrophic failures. However, a solid theoretical understanding of the performance of unfolding networks in the presence of adversarial attacks is still in its infancy. In this paper, we study the adversarial generalization of unfolding networks when perturbed with $l_2$-norm constrained attacks, generated by the fast gradient sign method. Particularly, we choose a family of state-of-the-art overaparameterized unfolding networks and deploy a new framework to estimate their adversarial Rademacher complexity. Given this estimate, we provide adversarial generalization error bounds for the networks under study, which are tight with respect to the attack level. To our knowledge, this is the first theoretical analysis on the adversarial generalization of unfolding networks. We further present a series of experiments on real-world data, with results corroborating our derived theory, consistently for all data. Finally, we observe that the family's overparameterization can be exploited to promote adversarial robustness, shedding light on how to efficiently robustify neural networks.

http://arxiv.org/abs/2408.01964
Top K Enhanced Reinforcement Learning Attacks on Heterogeneous Graph Node Classification. (78%)
Honglin Gao; Xiang Li; Yajuan Sun; Gaoxi Xiao
Graph Neural Networks (GNNs) have attracted substantial interest due to their exceptional performance on graph-based data. However, their robustness, especially on heterogeneous graphs, remains underexplored, particularly against adversarial attacks. This paper proposes HeteroKRLAttack, a targeted evasion black-box attack method for heterogeneous graphs. By integrating reinforcement learning with a Top-K algorithm to reduce the action space, our method efficiently identifies effective attack strategies to disrupt node classification tasks. We validate the effectiveness of HeteroKRLAttack through experiments on multiple heterogeneous graph datasets, showing significant reductions in classification accuracy compared to baseline methods. An ablation study underscores the critical role of the Top-K algorithm in enhancing attack performance. Our findings highlight potential vulnerabilities in current models and provide guidance for future defense strategies against adversarial attacks on heterogeneous graphs.

http://arxiv.org/abs/2509.14959
Discrete optimal transport is a strong audio adversarial attack. (75%)
Anton Selitskiy; Akib Shahriyar; Jishnuraj Prakasan
In this paper, we show that discrete optimal transport (DOT) is an effective black-box adversarial attack against modern audio anti-spoofing countermeasures (CMs). Our attack operates as a post-processing, distribution-alignment step: frame-level WavLM embeddings of generated speech are aligned to an unpaired bona fide pool via entropic OT and a top-$k$ barycentric projection, then decoded with a neural vocoder. Evaluated on ASVspoof2019 and ASVspoof5 with AASIST baselines, DOT yields consistently high equal error rate (EER) across datasets and remains competitive after CM fine-tuning, outperforming several conventional attacks in cross-dataset transfer. Ablation analysis highlights the practical impact of vocoder overlap. Results indicate that distribution-level alignment is a powerful and stable attack surface for deployed CMs.

http://arxiv.org/abs/2509.15159
AIP: Subverting Retrieval-Augmented Generation via Adversarial Instructional Prompt. (75%)
Saket S. Chaturvedi; Gaurav Bagwe; Lan Zhang; Xiaoyong Yuan
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by retrieving relevant documents from external sources to improve factual accuracy and verifiability. However, this reliance introduces new attack surfaces within the retrieval pipeline, beyond the LLM itself. While prior RAG attacks have exposed such vulnerabilities, they largely rely on manipulating user queries, which is often infeasible in practice due to fixed or protected user inputs. This narrow focus overlooks a more realistic and stealthy vector: instructional prompts, which are widely reused, publicly shared, and rarely audited. Their implicit trust makes them a compelling target for adversaries to manipulate RAG behavior covertly.
  We introduce a novel attack for Adversarial Instructional Prompt (AIP) that exploits adversarial instructional prompts to manipulate RAG outputs by subtly altering retrieval behavior. By shifting the attack surface to the instructional prompts, AIP reveals how trusted yet seemingly benign interface components can be weaponized to degrade system integrity. The attack is crafted to achieve three goals: (1) naturalness, to evade user detection; (2) utility, to encourage use of prompts; and (3) robustness, to remain effective across diverse query variations. We propose a diverse query generation strategy that simulates realistic linguistic variation in user queries, enabling the discovery of prompts that generalize across paraphrases and rephrasings. Building on this, a genetic algorithm-based joint optimization is developed to evolve adversarial prompts by balancing attack success, clean-task utility, and stealthiness. Experimental results show that AIP achieves up to 95.23% ASR while preserving benign functionality. These findings uncover a critical and previously overlooked vulnerability in RAG systems, emphasizing the need to reassess the shared instructional prompts.

http://arxiv.org/abs/2509.15202
Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction. (16%)
Yuanbo Xie; Yingjie Zhang; Tianyun Liu; Duohe Ma; Tingwen Liu
Jailbreak attacks pose persistent threats to large language models (LLMs). Current safety alignment methods have attempted to address these issues, but they experience two significant limitations: insufficient safety alignment depth and unrobust internal defense mechanisms. These limitations make them vulnerable to adversarial attacks such as prefilling and refusal direction manipulation. We introduce DeepRefusal, a robust safety alignment framework that overcomes these issues. DeepRefusal forces the model to dynamically rebuild its refusal mechanisms from jailbreak states. This is achieved by probabilistically ablating the refusal direction across layers and token depths during fine-tuning. Our method not only defends against prefilling and refusal direction attacks but also demonstrates strong resilience against other unseen jailbreak strategies. Extensive evaluations on four open-source LLM families and six representative attacks show that DeepRefusal reduces attack success rates by approximately 95%, while maintaining model capabilities with minimal performance degradation.

http://arxiv.org/abs/2509.19360
Semantic Representation Attack against Aligned Large Language Models. (1%)
Jiawei Lian; Jianhong Pan; Lefan Wang; Yi Wang; Shaohui Mei; Lap-Pui Chau
Large Language Models (LLMs) increasingly employ alignment techniques to prevent harmful outputs. Despite these safeguards, attackers can circumvent them by crafting prompts that induce LLMs to generate harmful content.
  Current methods typically target exact affirmative responses, such as ``Sure, here is...'', suffering from limited convergence, unnatural prompts, and high computational costs.
  We introduce Semantic Representation Attack, a novel paradigm that fundamentally reconceptualizes adversarial objectives against aligned LLMs.
  Rather than targeting exact textual patterns, our approach exploits the semantic representation space comprising diverse responses with equivalent harmful meanings.
  This innovation resolves the inherent trade-off between attack efficacy and prompt naturalness that plagues existing methods.
  The Semantic Representation Heuristic Search algorithm is proposed to efficiently generate semantically coherent and concise adversarial prompts by maintaining interpretability during incremental expansion.
  We establish rigorous theoretical guarantees for semantic convergence and demonstrate that our method achieves unprecedented attack success rates (89.41\% averaged across 18 LLMs, including 100\% on 11 models) while maintaining stealthiness and efficiency.
  Comprehensive experimental results confirm the overall superiority of our Semantic Representation Attack.
  The code will be publicly available.

http://arxiv.org/abs/2509.14801
STEP: Structured Training and Evaluation Platform for benchmarking trajectory prediction models. (1%)
Julian F. Schumann; Anna Mészáros; Jens Kober; Arkady Zgonnikov
While trajectory prediction plays a critical role in enabling safe and effective path-planning in automated vehicles, standardized practices for evaluating such models remain underdeveloped. Recent efforts have aimed to unify dataset formats and model interfaces for easier comparisons, yet existing frameworks often fall short in supporting heterogeneous traffic scenarios, joint prediction models, or user documentation. In this work, we introduce STEP -- a new benchmarking framework that addresses these limitations by providing a unified interface for multiple datasets, enforcing consistent training and evaluation conditions, and supporting a wide range of prediction models. We demonstrate the capabilities of STEP in a number of experiments which reveal 1) the limitations of widely-used testing procedures, 2) the importance of joint modeling of agents for better predictions of interactions, and 3) the vulnerability of current state-of-the-art models against both distribution shifts and targeted attacks by adversarial agents. With STEP, we aim to shift the focus from the ``leaderboard'' approach to deeper insights about model behavior and generalization in complex multi-agent settings.

http://arxiv.org/abs/2509.14571
VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models. (1%)
Huanchen Wang; Wencheng Zhang; Zhiqiang Wang; Zhicong Lu; Yuxin Ma
Vision-language (VL) models have shown transformative potential across various critical domains due to their capability to comprehend multi-modal information. However, their performance frequently degrades under distribution shifts, making it crucial to assess and improve robustness against real-world data corruption encountered in practical applications. While advancements in VL benchmark datasets and data augmentation (DA) have contributed to robustness evaluation and improvement, there remain challenges due to a lack of in-depth comprehension of model behavior as well as the need for expertise and iterative efforts to explore data patterns. Given the achievement of visualization in explaining complex models and exploring large-scale data, understanding the impact of various data corruption on VL models aligns naturally with a visual analytics approach. To address these challenges, we introduce VisMoDAl, a visual analytics framework designed to evaluate VL model robustness against various corruption types and identify underperformed samples to guide the development of effective DA strategies. Grounded in the literature review and expert discussions, VisMoDAl supports multi-level analysis, ranging from examining performance under specific corruptions to task-driven inspection of model behavior and corresponding data slice. Unlike conventional works, VisMoDAl enables users to reason about the effects of corruption on VL models, facilitating both model behavior understanding and DA strategy formulation. The utility of our system is demonstrated through case studies and quantitative evaluations focused on corruption robustness in the image captioning task.

http://arxiv.org/abs/2509.13922
Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification. (64%)
Wenkui Yang; Jie Cao; Junxian Duan; Ran He
Diffusion models like Stable Diffusion have become prominent in visual synthesis tasks due to their powerful customization capabilities, which also introduce significant security risks, including deepfakes and copyright infringement. In response, a class of methods known as protective perturbation emerged, which mitigates image misuse by injecting imperceptible adversarial noise. However, purification can remove protective perturbations, thereby exposing images again to the risk of malicious forgery. In this work, we formalize the anti-purification task, highlighting challenges that hinder existing approaches, and propose a simple diagnostic protective perturbation named AntiPure. AntiPure exposes vulnerabilities of purification within the "purification-customization" workflow, owing to two guidance mechanisms: 1) Patch-wise Frequency Guidance, which reduces the model's influence over high-frequency components in the purified image, and 2) Erroneous Timestep Guidance, which disrupts the model's denoising strategy across different timesteps. With additional guidance, AntiPure embeds imperceptible perturbations that persist under representative purification settings, achieving effective post-customization distortion. Experiments show that, as a stress test for purification, AntiPure achieves minimal perceptual discrepancy and maximal distortion, outperforming other protective perturbation methods within the purification-customization workflow.

http://arxiv.org/abs/2509.14383
RLBind: Adversarial-Invariant Cross-Modal Alignment for Unified Robust Embeddings. (15%)
Yuhong Lu
Unified multi-modal encoders that bind vision, audio, and other sensors into a shared embedding space are attractive building blocks for robot perception and decision-making. However, on-robot deployment exposes the vision branch to adversarial and natural corruptions, making robustness a prerequisite for safety. Prior defenses typically align clean and adversarial features within CLIP-style encoders and overlook broader cross-modal correspondence, yielding modest gains and often degrading zero-shot transfer. We introduce RLBind, a two-stage adversarial-invariant cross-modal alignment framework for robust unified embeddings. Stage 1 performs unsupervised fine-tuning on clean-adversarial pairs to harden the visual encoder. Stage 2 leverages cross-modal correspondence by minimizing the discrepancy between clean/adversarial features and a text anchor, while enforcing class-wise distributional alignment across modalities. Extensive experiments on Image, Audio, Thermal, and Video data show that RLBind consistently outperforms the LanguageBind backbone and standard fine-tuning baselines in both clean accuracy and norm-bounded adversarial robustness. By improving resilience without sacrificing generalization, RLBind provides a practical path toward safer multi-sensor perception stacks for embodied robots in navigation, manipulation, and other autonomy settings.

http://arxiv.org/abs/2509.13772
Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generation. (9%)
Baolei Zhang; Haoran Xin; Yuxi Chen; Zhuqing Liu; Biao Yi; Tong Li; Lihai Nie; Zheli Liu; Minghong Fang
Retrieval-Augmented Generation (RAG) integrates external knowledge into large language models to improve response quality. However, recent work has shown that RAG systems are highly vulnerable to poisoning attacks, where malicious texts are inserted into the knowledge database to influence model outputs. While several defenses have been proposed, they are often circumvented by more adaptive or sophisticated attacks.
  This paper presents RAGOrigin, a black-box responsibility attribution framework designed to identify which texts in the knowledge database are responsible for misleading or incorrect generations. Our method constructs a focused attribution scope tailored to each misgeneration event and assigns a responsibility score to each candidate text by evaluating its retrieval ranking, semantic relevance, and influence on the generated response. The system then isolates poisoned texts using an unsupervised clustering method. We evaluate RAGOrigin across seven datasets and fifteen poisoning attacks, including newly developed adaptive poisoning strategies and multi-attacker scenarios. Our approach outperforms existing baselines in identifying poisoned content and remains robust under dynamic and noisy conditions. These results suggest that RAGOrigin provides a practical and effective solution for tracing the origins of corrupted knowledge in RAG systems.

http://arxiv.org/abs/2509.14297
A Simple and Efficient Jailbreak Method Exploiting LLMs' Helpfulness. (9%)
Xuan Luo; Yue Wang; Zefeng He; Geng Tu; Jing Li; Ruifeng Xu
Safety alignment aims to prevent Large Language Models (LLMs) from responding to harmful queries. To strengthen safety protections, jailbreak methods are developed to simulate malicious attacks and uncover vulnerabilities. In this paper, we introduce HILL (Hiding Intention by Learning from LLMs), a novel jailbreak approach that systematically transforms imperative harmful requests into learning-style questions with only straightforward hypotheticality indicators. Further, we introduce two new metrics to thoroughly evaluate the utility of jailbreak methods. Experiments on the AdvBench dataset across a wide range of models demonstrate HILL's strong effectiveness, generalizability, and harmfulness. It achieves top attack success rates on the majority of models and across malicious categories while maintaining high efficiency with concise prompts. Results of various defense methods show the robustness of HILL, with most defenses having mediocre effects or even increasing the attack success rates. Moreover, the assessment on our constructed safe prompts reveals inherent limitations of LLMs' safety mechanisms and flaws in defense methods. This work exposes significant vulnerabilities of safety measures against learning-style elicitation, highlighting a critical challenge of balancing helpfulness and safety alignments.

http://arxiv.org/abs/2503.17534
MetaSel: A Test Selection Approach for Fine-tuned DNN Models. (2%)
Amin Abbasishahkoo; Mahboubeh Dadkhah; Lionel Briand; Dayi Lin
Deep Neural Networks (DNNs) face challenges during deployment due to covariate shift, i.e., data distribution shifts between development and deployment contexts. Fine-tuning adapts pre-trained models to new contexts requiring smaller labeled sets. However, testing fine-tuned models under constrained labeling budgets remains a critical challenge. This paper introduces MetaSel, a new approach tailored for DNN models that have been fine-tuned to address covariate shift, to select tests from unlabeled inputs. MetaSel assumes that fine-tuned and pre-trained models share related data distributions and exhibit similar behaviors for many inputs. However, their behaviors diverge within the input subspace where fine-tuning alters decision boundaries, making those inputs more prone to misclassification. Unlike general approaches that rely solely on the DNN model and its input set, MetaSel leverages information from both the fine-tuned and pre-trained models and their behavioral differences to estimate misclassification probability for unlabeled test inputs, enabling more effective test selection. Our extensive empirical evaluation, comparing MetaSel against 11 state-of-the-art approaches and involving 68 fine-tuned models across weak, medium, and strong distribution shifts, demonstrates that MetaSel consistently delivers significant improvements in Test Relative Coverage (TRC) over existing baselines, particularly under highly constrained labeling budgets. MetaSel shows average TRC improvements of 28.46% to 56.18% over the most frequent second-best baselines while maintaining a high TRC median and low variability. Our results confirm MetaSel's practicality, robustness, and cost-effectiveness for test selection in the context of fine-tuned models.

http://arxiv.org/abs/2509.13813
Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs. (1%)
Edward Phillips; Sean Wu; Soheila Molaei; Danielle Belgrave; Anshul Thakur; David Clifton
Large language models demonstrate impressive results across diverse tasks but are still known to hallucinate, generating linguistically plausible but incorrect answers to questions. Uncertainty quantification has been proposed as a strategy for hallucination detection, but no existing black-box approach provides estimates for both global and local uncertainty. The former attributes uncertainty to a batch of responses, while the latter attributes uncertainty to individual responses. Current local methods typically rely on white-box access to internal model states, whilst black-box methods only provide global uncertainty estimates. We introduce a geometric framework to address this, based on archetypal analysis of batches of responses sampled with only black-box model access. At the global level, we propose Geometric Volume, which measures the convex hull volume of archetypes derived from response embeddings. At the local level, we propose Geometric Suspicion, which ranks responses by reliability and enables hallucination reduction through preferential response selection. Unlike prior dispersion methods which yield only a single global score, our approach provides semantic boundary points which have utility for attributing reliability to individual responses. Experiments show that our framework performs comparably to or better than prior methods on short form question-answering datasets, and achieves superior results on medical datasets where hallucinations carry particularly critical risks. We also provide theoretical justification by proving a link between convex hull volume and entropy.

http://arxiv.org/abs/2509.13982
CLMTracing: Black-box User-level Watermarking for Code Language Model Tracing. (1%)
Boyu Zhang; Ping He; Tianyu Du; Xuhong Zhang; Lei Yun; Kingsum Chow; Jianwei Yin
With the widespread adoption of open-source code language models (code LMs), intellectual property (IP) protection has become an increasingly critical concern. While current watermarking techniques have the potential to identify the code LM to protect its IP, they have limitations when facing the more practical and complex demand, i.e., offering the individual user-level tracing in the black-box setting. This work presents CLMTracing, a black-box code LM watermarking framework employing the rule-based watermarks and utility-preserving injection method for user-level model tracing. CLMTracing further incorporates a parameter selection algorithm sensitive to the robust watermark and adversarial training to enhance the robustness against watermark removal attacks. Comprehensive evaluations demonstrate CLMTracing is effective across multiple state-of-the-art (SOTA) code LMs, showing significant harmless improvements compared to existing SOTA baselines and strong robustness against various removal attacks.

http://arxiv.org/abs/2412.20634
Graph Neural Networks for Next-Generation-IoT: Recent Advances and Open Challenges. (1%)
Nguyen Xuan Tung; Le Tung Giang; Bui Duc Son; Seon Geun Jeong; Chien Trinh Van; Won Joo Hwang; Lajos Hanzo
Graph Neural Networks (GNNs) have emerged as a powerful framework for modeling complex interconnected systems, hence making them particularly well-suited to address the growing challenges of next-generation Internet of Things (NG-IoT) networks. Existing studies remain fragmented, and there is a lack of comprehensive guidance on how GNNs can be systematically applied to NG-IoT systems. As NG-IoT systems evolve toward 6G, they incorporate diverse technologies. These advances promise unprecedented connectivity, sensing, and automation but also introduce significant complexity, requiring new approaches for scalable learning, dynamic optimization, and secure, decentralized decision-making. This survey provides a comprehensive and forward-looking exploration of how GNNs can empower NG-IoT environments. We commence by exploring the fundamental paradigms of GNNs and articulating the motivation for their use in NG-IoT networks. Besides, we intrinsically connect GNNs with the family of low-density parity-check codes, modeling the NG-IoT as dynamic constrained graphs. We highlight the distinct roles of node-, edge-, and graph-level tasks in tackling key challenges and demonstrate the GNNs' ability to overcome the limitations of traditional optimization. We examine the application of GNNs across core NG-enabling technologies and their integration with distributed frameworks to support privacy-preservation and distributed intelligence. We then delve into the challenges posed by adversarial attacks, offering insights into defense mechanisms. Lastly, we examine how GNNs can be integrated with emerging technologies. Our findings highlight the transformative potential of GNNs in improving efficiency, scalability, and security. Finally, we summarize the key lessons learned and outline promising future research directions, along with a set of design guidelines tailored for NG-IoT applications.

http://arxiv.org/abs/2502.05760
MADAR: Efficient Continual Learning for Malware Analysis with Distribution-Aware Replay. (1%)
Mohammad Saidur Rahman; Scott Coull; Qi Yu; Matthew Wright
Millions of new pieces of malicious software (i.e., malware) are introduced each year. This poses significant challenges for antivirus vendors, who use machine learning to detect and analyze malware, and must keep up with changes in the distribution while retaining knowledge of older variants. Continual learning (CL) holds the potential to address this challenge by reducing the storage and computational costs of regularly retraining over all the collected data. Prior work, however, shows that CL techniques, which are designed primarily for computer vision tasks, fare poorly when applied to malware classification. To address these issues, we begin with an exploratory analysis of a typical malware dataset, which reveals that malware families are diverse and difficult to characterize, requiring a wide variety of samples to learn a robust representation. Based on these findings, we propose $\underline{M}$alware $\underline{A}$nalysis with $\underline{D}$istribution-$\underline{A}$ware $\underline{R}$eplay (MADAR), a CL framework that accounts for the unique properties and challenges of the malware data distribution. Through extensive evaluation on large-scale Windows and Android malware datasets, we show that MADAR significantly outperforms prior work. This highlights the importance of understanding domain characteristics when designing CL techniques and demonstrates a path forward for the malware classification domain.

http://arxiv.org/abs/2509.14335
Beyond Classification: Evaluating LLMs for Fine-Grained Automatic Malware Behavior Auditing. (1%)
Xinran Zheng; Xingzhi Qian; Yiling He; Shuo Yang; Lorenzo Cavallaro
Automated malware classification has achieved strong detection performance. Yet, malware behavior auditing seeks causal and verifiable explanations of malicious activities -- essential not only to reveal what malware does but also to substantiate such claims with evidence. This task is challenging, as adversarial intent is often hidden within complex, framework-heavy applications, making manual auditing slow and costly. Large Language Models (LLMs) could help address this gap, but their auditing potential remains largely unexplored due to three limitations: (1) scarce fine-grained annotations for fair assessment; (2) abundant benign code obscuring malicious signals; and (3) unverifiable, hallucination-prone outputs undermining attribution credibility. To close this gap, we introduce MalEval, a comprehensive framework for fine-grained Android malware auditing, designed to evaluate how effectively LLMs support auditing under real-world constraints. MalEval provides expert-verified reports and an updated sensitive API list to mitigate ground truth scarcity and reduce noise via static reachability analysis. Function-level structural representations serve as intermediate attribution units for verifiable evaluation. Building on this, we define four analyst-aligned tasks -- function prioritization, evidence attribution, behavior synthesis, and sample discrimination -- together with domain-specific metrics and a unified workload-oriented score. We evaluate seven widely used LLMs on a curated dataset of recent malware and misclassified benign apps, offering the first systematic assessment of their auditing capabilities. MalEval reveals both promising potential and critical limitations across audit stages, providing a reproducible benchmark and foundation for future research on LLM-enhanced malware behavior auditing. MalEval is publicly available at https://github.com/ZhengXR930/MalEval.git

http://arxiv.org/abs/2310.11850
Revisiting Transferable Adversarial Images: Systemization, Evaluation, and New Insights. (99%)
Zhengyu Zhao; Hanwei Zhang; Renjue Li; Ronan Sicre; Laurent Amsaleg; Michael Backes; Qi Li; Qian Wang; Chao Shen
Transferable adversarial images raise critical security concerns for computer vision systems in real-world, black-box attack scenarios. Although many transfer attacks have been proposed, existing research lacks a systematic and comprehensive evaluation. In this paper, we systemize transfer attacks into five categories around the general machine learning pipeline and provide the first comprehensive evaluation, with 23 representative attacks against 11 representative defenses, including the recent, transfer-oriented defense and the real-world Google Cloud Vision. In particular, we identify two main problems of existing evaluations: (1) for attack transferability, lack of intra-category analyses with fair hyperparameter settings, and (2) for attack stealthiness, lack of diverse measures. Our evaluation results validate that these problems have indeed caused misleading conclusions and missing points, and addressing them leads to new, \textit{consensus-challenging} insights, such as (1) an early attack, DI, even outperforms all similar follow-up ones, (2) the state-of-the-art (white-box) defense, DiffPure, is even vulnerable to (black-box) transfer attacks, and (3) even under the same $L_p$ constraint, different attacks yield dramatically different stealthiness results regarding diverse imperceptibility metrics, finer-grained measures, and a user study. We hope that our analyses will serve as guidance on properly evaluating transferable adversarial images and advance the design of attacks and defenses. Code is available at https://github.com/ZhengyuZhao/TransferAttackEval.

http://arxiv.org/abs/2509.12633
CIARD: Cyclic Iterative Adversarial Robustness Distillation. (99%)
Liming Lu; Shuchao Pang; Xu Zheng; Xiang Gu; Anan Du; Yunhuai Liu; Yongbin Zhou
Adversarial robustness distillation (ARD) aims to transfer both performance and robustness from teacher model to lightweight student model, enabling resilient performance on resource-constrained scenarios. Though existing ARD approaches enhance student model's robustness, the inevitable by-product leads to the degraded performance on clean examples. We summarize the causes of this problem inherent in existing methods with dual-teacher framework as: 1. The divergent optimization objectives of dual-teacher models, i.e., the clean and robust teachers, impede effective knowledge transfer to the student model, and 2. The iteratively generated adversarial examples during training lead to performance deterioration of the robust teacher model. To address these challenges, we propose a novel Cyclic Iterative ARD (CIARD) method with two key innovations: a. A multi-teacher framework with contrastive push-loss alignment to resolve conflicts in dual-teacher optimization objectives, and b. Continuous adversarial retraining to maintain dynamic teacher robustness against performance degradation from the varying adversarial examples. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CIARD achieves remarkable performance with an average 3.53 improvement in adversarial defense rates across various attack scenarios and a 5.87 increase in clean sample accuracy, establishing a new benchmark for balancing model robustness and generalization. Our code is available at https://github.com/eminentgu/CIARD

http://arxiv.org/abs/2509.12939
Sy-FAR: Symmetry-based Fair Adversarial Robustness. (99%)
Haneen Najjar; Eyal Ronen; Mahmood Sharif
Security-critical machine-learning (ML) systems, such as face-recognition systems, are susceptible to adversarial examples, including real-world physically realizable attacks. Various means to boost ML's adversarial robustness have been proposed; however, they typically induce unfair robustness: It is often easier to attack from certain classes or groups than from others. Several techniques have been developed to improve adversarial robustness while seeking perfect fairness between classes. Yet, prior work has focused on settings where security and fairness are less critical. Our insight is that achieving perfect parity in realistic fairness-critical tasks, such as face recognition, is often infeasible -- some classes may be highly similar, leading to more misclassifications between them. Instead, we suggest that seeking symmetry -- i.e., attacks from class $i$ to $j$ would be as successful as from $j$ to $i$ -- is more tractable. Intuitively, symmetry is a desirable because class resemblance is a symmetric relation in most domains. Additionally, as we prove theoretically, symmetry between individuals induces symmetry between any set of sub-groups, in contrast to other fairness notions where group-fairness is often elusive. We develop Sy-FAR, a technique to encourage symmetry while also optimizing adversarial robustness and extensively evaluate it using five datasets, with three model architectures, including against targeted and untargeted realistic attacks. The results show Sy-FAR significantly improves fair adversarial robustness compared to state-of-the-art methods. Moreover, we find that Sy-FAR is faster and more consistent across runs. Notably, Sy-FAR also ameliorates another type of unfairness we discover in this work -- target classes that adversarial examples are likely to be classified into become significantly less vulnerable after inducing symmetry.

http://arxiv.org/abs/2509.12595
DisorientLiDAR: Physical Attacks on LiDAR-based Localization. (99%)
Yizhen Lao; Yu Zhang; Ziting Wang; Chengbo Wang; Yifei Xue; Wanpeng Shao
Deep learning models have been shown to be susceptible to adversarial attacks with visually imperceptible perturbations. Even this poses a serious security challenge for the localization of self-driving cars, there has been very little exploration of attack on it, as most of adversarial attacks have been applied to 3D perception. In this work, we propose a novel adversarial attack framework called DisorientLiDAR targeting LiDAR-based localization. By reverse-engineering localization models (e.g., feature extraction networks), adversaries can identify critical keypoints and strategically remove them, thereby disrupting LiDAR-based localization. Our proposal is first evaluated on three state-of-the-art point-cloud registration models (HRegNet, D3Feat, and GeoTransformer) using the KITTI dataset. Experimental results demonstrate that removing regions containing Top-K keypoints significantly degrades their registration accuracy. We further validate the attack's impact on the Autoware autonomous driving platform, where hiding merely a few critical regions induces noticeable localization drift. Finally, we extended our attacks to the physical world by hiding critical regions with near-infrared absorptive materials, thereby successfully replicate the attack effects observed in KITTI data. This step has been closer toward the realistic physical-world attack that demonstrate the veracity and generality of our proposal.

http://arxiv.org/abs/2501.13336
Gradient-Free Adversarial Purification with Diffusion Models. (99%)
Xuelong Dai; Dong Wang; Xiuzhen Cheng; Bin Xiao
Adversarial training and adversarial purification are two widely used defense strategies for enhancing model robustness against adversarial attacks. However, adversarial training requires costly retraining, while adversarial purification often suffers from low efficiency. More critically, existing defenses are primarily designed under the perturbation-based adversarial threat model, which is ineffective against recently introduced unrestricted adversarial attacks. In this paper, we propose an effective and efficient defense framework that counters both perturbation-based and unrestricted adversarial attacks. Our approach is motivated by the observation that adversarial examples typically lie near the decision boundary and are highly sensitive to pixel-level perturbations. To address this, we introduce adversarial anti-aliasing, a preprocessing technique that mitigates adversarial noise by reducing the magnitude of pixel-level perturbations. In addition, we propose adversarial super-resolution, which leverages prior knowledge from clean datasets to benignly restore high-quality images from adversarially degraded ones. Unlike image synthesis methods that generate entirely new images, adversarial super-resolution focuses on image restoration, making it more suitable for purification. Importantly, both techniques require no additional training and are computationally efficient since they do not rely on gradient computations. To further improve robustness across diverse datasets, we introduce a contrastive learning-based adversarial deblurring fine-tuning method. By incorporating adversarial priors during fine-tuning on the target dataset, this method enhances purification effectiveness without the need to retrain diffusion models.

http://arxiv.org/abs/2509.12824
DiffHash: Text-Guided Targeted Attack via Diffusion Models against Deep Hashing Image Retrieval. (98%)
Zechao Liu; Zheng Zhou; Xiangkun Chen; Tao Liang; Dapeng Lang
Deep hashing models have been widely adopted to tackle the challenges of large-scale image retrieval. However, these approaches face serious security risks due to their vulnerability to adversarial examples. Despite the increasing exploration of targeted attacks on deep hashing models, existing approaches still suffer from a lack of multimodal guidance, reliance on labeling information and dependence on pixel-level operations for attacks. To address these limitations, we proposed DiffHash, a novel diffusion-based targeted attack for deep hashing. Unlike traditional pixel-based attacks that directly modify specific pixels and lack multimodal guidance, our approach focuses on optimizing the latent representations of images, guided by text information generated by a Large Language Model (LLM) for the target image. Furthermore, we designed a multi-space hash alignment network to align the high-dimension image space and text space to the low-dimension binary hash space. During reconstruction, we also incorporated text-guided attention mechanisms to refine adversarial examples, ensuring them aligned with the target semantics while maintaining visual plausibility. Extensive experiments have demonstrated that our method outperforms state-of-the-art (SOTA) targeted attack methods, achieving better black-box transferability and offering more excellent stability across datasets.

http://arxiv.org/abs/2509.12724
Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models. (92%)
Yunhan Zhao; Xiang Zheng; Xingjun Ma
Despite their superb capabilities, Vision-Language Models (VLMs) have been shown to be vulnerable to jailbreak attacks. While recent jailbreaks have achieved notable progress, their effectiveness and efficiency can still be improved. In this work, we reveal an interesting phenomenon: incorporating weak defense into the attack pipeline can significantly enhance both the effectiveness and the efficiency of jailbreaks on VLMs. Building on this insight, we propose Defense2Attack, a novel jailbreak method that bypasses the safety guardrails of VLMs by leveraging defensive patterns to guide jailbreak prompt design. Specifically, Defense2Attack consists of three key components: (1) a visual optimizer that embeds universal adversarial perturbations with affirmative and encouraging semantics; (2) a textual optimizer that refines the input using a defense-styled prompt; and (3) a red-team suffix generator that enhances the jailbreak through reinforcement fine-tuning. We empirically evaluate our method on four VLMs and four safety benchmarks. The results demonstrate that Defense2Attack achieves superior jailbreak performance in a single attempt, outperforming state-of-the-art attack methods that often require multiple tries. Our work offers a new perspective on jailbreaking VLMs.

http://arxiv.org/abs/2509.12672
Towards Inclusive Toxic Content Moderation: Addressing Vulnerabilities to Adversarial Attacks in Toxicity Classifiers Tackling LLM-generated Content. (92%)
Shaz Furniturewala; Arkaitz Zubiaga
The volume of machine-generated content online has grown dramatically due to the widespread use of Large Language Models (LLMs), leading to new challenges for content moderation systems. Conventional content moderation classifiers, which are usually trained on text produced by humans, suffer from misclassifications due to LLM-generated text deviating from their training data and adversarial attacks that aim to avoid detection. Present-day defence tactics are reactive rather than proactive, since they rely on adversarial training or external detection models to identify attacks. In this work, we aim to identify the vulnerable components of toxicity classifiers that contribute to misclassification, proposing a novel strategy based on mechanistic interpretability techniques. Our study focuses on fine-tuned BERT and RoBERTa classifiers, testing on diverse datasets spanning a variety of minority groups. We use adversarial attacking techniques to identify vulnerable circuits. Finally, we suppress these vulnerable circuits, improving performance against adversarial attacks. We also provide demographic-level insights into these vulnerable circuits, exposing fairness and robustness gaps in model training. We find that models have distinct heads that are either crucial for performance or vulnerable to attack and suppressing the vulnerable heads improves performance on adversarial input. We also find that different heads are responsible for vulnerability across different demographic groups, which can inform more inclusive development of toxicity detection models.

http://arxiv.org/abs/2411.15244
Adversarial Prompt Distillation for Vision-Language Models. (75%)
Lin Luo; Xin Wang; Bojia Zi; Shihao Zhao; Xingjun Ma; Yu-Gang Jiang
Large pre-trained Vision-Language Models (VLMs) such as Contrastive Language-Image Pre-training (CLIP) have been shown to be susceptible to adversarial attacks, raising concerns about their deployment in safety-critical applications like autonomous driving and medical diagnosis. One promising approach for robustifying pre-trained VLMs is Adversarial Prompt Tuning (APT), which applies adversarial training during the process of prompt tuning. However, existing APT methods are mostly single-modal methods that design prompt(s) for only the visual or textual modality, limiting their effectiveness in either robustness or clean accuracy. In this work, we propose Adversarial Prompt Distillation (APD), a bimodal knowledge distillation framework that enhances APT by integrating it with multi-modal knowledge transfer. APD optimizes prompts for both visual and textual modalities while distilling knowledge from a clean pre-trained teacher CLIP model. Extensive experiments on multiple benchmark datasets demonstrate the superiority of our APD method over the current state-of-the-art APT methods in terms of both adversarial robustness and clean accuracy. The effectiveness of APD also validates the possibility of using a non-robust teacher to improve the generalization and robustness of fine-tuned VLMs.

http://arxiv.org/abs/2508.00552
DBLP: Noise Bridge Consistency Distillation For Efficient And Reliable Adversarial Purification. (68%)
Chihan Huang; Belal Alsinglawi; Islam Al-qudah
Recent advances in deep neural networks (DNNs) have led to remarkable success across a wide range of tasks. However, their susceptibility to adversarial perturbations remains a critical vulnerability. Existing diffusion-based adversarial purification methods often require intensive iterative denoising, severely limiting their practical deployment. In this paper, we propose Diffusion Bridge Distillation for Purification (DBLP), a novel and efficient diffusion-based framework for adversarial purification. Central to our approach is a new objective, noise bridge distillation, which constructs a principled alignment between the adversarial noise distribution and the clean data distribution within a latent consistency model (LCM). To further enhance semantic fidelity, we introduce adaptive semantic enhancement, which fuses multi-scale pyramid edge maps as conditioning input to guide the purification process. Extensive experiments across multiple datasets demonstrate that DBLP achieves state-of-the-art (SOTA) robust accuracy, superior image quality, and around 0.2s inference time, marking a significant step toward real-time adversarial purification.

http://arxiv.org/abs/2509.12964
BAPFL: Exploring Backdoor Attacks Against Prototype-based Federated Learning. (45%)
Honghong Zeng; Jiong Lou; Zhe Wang; Hefeng Zhou; Chentao Wu; Wei Zhao; Jie Li
Prototype-based federated learning (PFL) has emerged as a promising paradigm to address data heterogeneity problems in federated learning, as it leverages mean feature vectors as prototypes to enhance model generalization. However, its robustness against backdoor attacks remains largely unexplored. In this paper, we identify that PFL is inherently resistant to existing backdoor attacks due to its unique prototype learning mechanism and local data heterogeneity. To further explore the security of PFL, we propose BAPFL, the first backdoor attack method specifically designed for PFL frameworks. BAPFL integrates a prototype poisoning strategy with a trigger optimization mechanism. The prototype poisoning strategy manipulates the trajectories of global prototypes to mislead the prototype training of benign clients, pushing their local prototypes of clean samples away from the prototypes of trigger-embedded samples. Meanwhile, the trigger optimization mechanism learns a unique and stealthy trigger for each potential target label, and guides the prototypes of trigger-embedded samples to align closely with the global prototype of the target label. Experimental results across multiple datasets and PFL variants demonstrate that BAPFL achieves a 35\%-75\% improvement in attack success rate compared to traditional backdoor attacks, while preserving main task accuracy. These results highlight the effectiveness, stealthiness, and adaptability of BAPFL in PFL.

http://arxiv.org/abs/2509.11173
Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers. (45%)
Simin Chen; Jinjun Peng; Yixin He; Junfeng Yang; Baishakhi Ray
Deep learning (DL) compilers are core infrastructure in modern DL systems, offering flexibility and scalability beyond vendor-specific libraries. This work uncovers a fundamental vulnerability in their design: can an official, unmodified compiler alter a model's semantics during compilation and introduce hidden backdoors? We study both adversarial and natural settings. In the adversarial case, we craft benign models where triggers have no effect pre-compilation but become effective backdoors after compilation. Tested on six models, three commercial compilers, and two hardware platforms, our attack yields 100% success on triggered inputs while preserving normal accuracy and remaining undetected by state-of-the-art detectors. The attack generalizes across compilers, hardware, and floating-point settings. In the natural setting, we analyze the top 100 HuggingFace models (including one with 220M+ downloads) and find natural triggers in 31 models. This shows that compilers can introduce risks even without adversarial manipulation.
  Our results reveal an overlooked threat: unmodified DL compilers can silently alter model semantics. To our knowledge, this is the first work to expose inherent security risks in DL compiler design, opening a new direction for secure and trustworthy ML.

http://arxiv.org/abs/2509.13219
On the Out-of-Distribution Backdoor Attack for Federated Learning. (41%)
Jiahao Xu; Zikai Zhang; Rui Hu
Traditional backdoor attacks in federated learning (FL) operate within constrained attack scenarios, as they depend on visible triggers and require physical modifications to the target object, which limits their practicality. To address this limitation, we introduce a novel backdoor attack prototype for FL called the out-of-distribution (OOD) backdoor attack ($\mathtt{OBA}$), which uses OOD data as both poisoned samples and triggers simultaneously. Our approach significantly broadens the scope of backdoor attack scenarios in FL. To improve the stealthiness of $\mathtt{OBA}$, we propose $\mathtt{SoDa}$, which regularizes both the magnitude and direction of malicious local models during local training, aligning them closely with their benign versions to evade detection. Empirical results demonstrate that $\mathtt{OBA}$ effectively circumvents state-of-the-art defenses while maintaining high accuracy on the main task.
  To address this security vulnerability in the FL system, we introduce $\mathtt{BNGuard}$, a new server-side defense method tailored against $\mathtt{SoDa}$. $\mathtt{BNGuard}$ leverages the observation that OOD data causes significant deviations in the running statistics of batch normalization layers. This allows $\mathtt{BNGuard}$ to identify malicious model updates and exclude them from aggregation, thereby enhancing the backdoor robustness of FL. Extensive experiments across various settings show the effectiveness of $\mathtt{BNGuard}$ on defending against $\mathtt{SoDa}$. The code is available at https://github.com/JiiahaoXU/SoDa-BNGuard.

http://arxiv.org/abs/2508.09456
IAG: Input-aware Backdoor Attack on VLMs for Visual Grounding. (33%)
Junxian Li; Beining Xu; Di Zhang
Vision-language models (VLMs) have shown significant advancements in tasks such as visual grounding, where they localize specific objects in images based on natural language queries and images. However, security issues in visual grounding tasks for VLMs remain underexplored, especially in the context of backdoor attacks. In this paper, we introduce a novel input-aware backdoor attack method, IAG, designed to manipulate the grounding behavior of VLMs. This attack forces the model to ground a specific target object in the input image, regardless of the user's query. We propose an adaptive trigger generator that embeds the semantic information of the attack target's description into the original image using a text-conditional U-Net, thereby overcoming the open-vocabulary attack challenge. To ensure the attack's stealthiness, we utilize a reconstruction loss to minimize visual discrepancies between poisoned and clean images. Additionally, we introduce a unified method for generating attack data. IAG is evaluated theoretically and empirically, demonstrating its feasibility and effectiveness. Notably, our ASR@0.5 on InternVL-2.5-8B reaches over 65\% on various testing sets. IAG also shows promising potential on manipulating Ferret-7B and LlaVA-1.5-7B with very little accuracy decrease on clean samples. Extensive specific experiments, such as ablation study and potential defense, also indicate the robustness and transferability of our attack.

http://arxiv.org/abs/2508.20890
PromptSleuth: Detecting Prompt Injection via Semantic Intent Invariance. (33%)
Mengxiao Wang; Yuxuan Zhang; Guofei Gu
Large Language Models (LLMs) are increasingly integrated into real-world applications, from virtual assistants to autonomous agents. However, their flexibility also introduces new attack vectors-particularly Prompt Injection (PI), where adversaries manipulate model behavior through crafted inputs. As attackers continuously evolve with paraphrased, obfuscated, and even multi-task injection strategies, existing benchmarks are no longer sufficient to capture the full spectrum of emerging threats.
  To address this gap, we construct a new benchmark that systematically extends prior efforts. Our benchmark subsumes the two widely-used existing ones while introducing new manipulation techniques and multi-task scenarios, thereby providing a more comprehensive evaluation setting. We find that existing defenses, though effective on their original benchmarks, show clear weaknesses under our benchmark, underscoring the need for more robust solutions. Our key insight is that while attack forms may vary, the adversary's intent-injecting an unauthorized task-remains invariant. Building on this observation, we propose PromptSleuth, a semantic-oriented defense framework that detects prompt injection by reasoning over task-level intent rather than surface features. Evaluated across state-of-the-art benchmarks, PromptSleuth consistently outperforms existing defense while maintaining comparable runtime and cost efficiency. These results demonstrate that intent-based semantic reasoning offers a robust, efficient, and generalizable strategy for defending LLMs against evolving prompt injection threats.

http://arxiv.org/abs/2509.11120
SoK: How Sensor Attacks Disrupt Autonomous Vehicles: An End-to-end Analysis, Challenges, and Missed Threats. (11%)
Qingzhao Zhang; Shaocheng Luo; Z. Morley Mao; Miroslav Pajic; Michael K. Reiter
Autonomous vehicles, including self-driving cars, robotic ground vehicles, and drones, rely on complex sensor pipelines to ensure safe and reliable operation. However, these safety-critical systems remain vulnerable to adversarial sensor attacks that can compromise their performance and mission success. While extensive research has demonstrated various sensor attack techniques, critical gaps remain in understanding their feasibility in real-world, end-to-end systems. This gap largely stems from the lack of a systematic perspective on how sensor errors propagate through interconnected modules in autonomous systems when autonomous vehicles interact with the physical world.
  To bridge this gap, we present a comprehensive survey of autonomous vehicle sensor attacks across platforms, sensor modalities, and attack methods. Central to our analysis is the System Error Propagation Graph (SEPG), a structured demonstration tool that illustrates how sensor attacks propagate through system pipelines, exposing the conditions and dependencies that determine attack feasibility. With the aid of SEPG, our study distills seven key findings that highlight the feasibility challenges of sensor attacks and uncovers eleven previously overlooked attack vectors exploiting inter-module interactions, several of which we validate through proof-of-concept experiments. Additionally, we demonstrate how large language models (LLMs) can automate aspects of SEPG construction and cross-validate expert analysis, showcasing the promise of AI-assisted security evaluation.

http://arxiv.org/abs/2509.13266
JANUS: A Dual-Constraint Generative Framework for Stealthy Node Injection Attacks. (11%)
Jiahao Zhang; Xiaobing Pei; Zhaokun Zhong; Wenqiang Hao; Zhenghao Tang
Graph Neural Networks (GNNs) have demonstrated remarkable performance across various applications, yet they are vulnerable to sophisticated adversarial attacks, particularly node injection attacks. The success of such attacks heavily relies on their stealthiness, the ability to blend in with the original graph and evade detection. However, existing methods often achieve stealthiness by relying on indirect proxy metrics, lacking consideration for the fundamental characteristics of the injected content, or focusing only on imitating local structures, which leads to the problem of local myopia. To overcome these limitations, we propose a dual-constraint stealthy node injection framework, called Joint Alignment of Nodal and Universal Structures (JANUS). At the local level, we introduce a local feature manifold alignment strategy to achieve geometric consistency in the feature space. At the global level, we incorporate structured latent variables and maximize the mutual information with the generated structures, ensuring the injected structures are consistent with the semantic patterns of the original graph. We model the injection attack as a sequential decision process, which is optimized by a reinforcement learning agent. Experiments on multiple standard datasets demonstrate that the JANUS framework significantly outperforms existing methods in terms of both attack effectiveness and stealthiness.

http://arxiv.org/abs/2509.13514
AQUA-LLM: Evaluating Accuracy, Quantization, and Adversarial Robustness Trade-offs in LLMs for Cybersecurity Question Answering. (4%)
Onat Gungor; Roshan Sood; Harold Wang; Tajana Rosing
Large Language Models (LLMs) have recently demonstrated strong potential for cybersecurity question answering (QA), supporting decision-making in real-time threat detection and response workflows. However, their substantial computational demands pose significant challenges for deployment on resource-constrained edge devices. Quantization, a widely adopted model compression technique, can alleviate these constraints. Nevertheless, quantization may degrade model accuracy and increase susceptibility to adversarial attacks. Fine-tuning offers a potential means to mitigate these limitations, but its effectiveness when combined with quantization remains insufficiently explored. Hence, it is essential to understand the trade-offs among accuracy, efficiency, and robustness. We propose AQUA-LLM, an evaluation framework designed to benchmark several state-of-the-art small LLMs under four distinct configurations: base, quantized-only, fine-tuned, and fine-tuned combined with quantization, specifically for cybersecurity QA. Our results demonstrate that quantization alone yields the lowest accuracy and robustness despite improving efficiency. In contrast, combining quantization with fine-tuning enhances both LLM robustness and predictive performance, achieving an optimal balance of accuracy, robustness, and efficiency. These findings highlight the critical need for quantization-aware, robustness-preserving fine-tuning methodologies to enable the robust and efficient deployment of LLMs for cybersecurity QA.

http://arxiv.org/abs/2509.13046
MIA-EPT: Membership Inference Attack via Error Prediction for Tabular Data. (2%)
Eyal German; Daniel Samira; Yuval Elovici; Asaf Shabtai
Synthetic data generation plays an important role in enabling data sharing, particularly in sensitive domains like healthcare and finance. Recent advances in diffusion models have made it possible to generate realistic, high-quality tabular data, but they may also memorize training records and leak sensitive information. Membership inference attacks (MIAs) exploit this vulnerability by determining whether a record was used in training. While MIAs have been studied in images and text, their use against tabular diffusion models remains underexplored despite the unique risks of structured attributes and limited record diversity. In this paper, we introduce MIAEPT, Membership Inference Attack via Error Prediction for Tabular Data, a novel black-box attack specifically designed to target tabular diffusion models. MIA-EPT constructs errorbased feature vectors by masking and reconstructing attributes of target records, disclosing membership signals based on how well these attributes are predicted. MIA-EPT operates without access to the internal components of the generative model, relying only on its synthetic data output, and was shown to generalize across multiple state-of-the-art diffusion models. We validate MIA-EPT on three diffusion-based synthesizers, achieving AUC-ROC scores of up to 0.599 and TPR@10% FPR values of 22.0% in our internal tests. Under the MIDST 2025 competition conditions, MIA-EPT achieved second place in the Black-box Multi-Table track (TPR@10% FPR = 20.0%). These results demonstrate that our method can uncover substantial membership leakage in synthetic tabular data, challenging the assumption that synthetic data is inherently privacy-preserving. Our code is publicly available at https://github.com/eyalgerman/MIA-EPT.

http://arxiv.org/abs/2509.02072
Abex-rat: Synergizing Abstractive Augmentation and Adversarial Training for Classification of Occupational Accident Reports. (2%)
Jian Chen; Jiabao Dou; Jinbao Tian; Yunqi Yang; Zhou Li
The automatic classification of occupational accident reports is a critical research area for enhancing workplace safety and enabling large-scale risk analysis. However, the severe class imbalance inherent in these real-world datasets often compromises the performance of analytical models, particularly for rare but severe incident types, hindering the development of reliable automated systems. To address this challenge, we propose ABEX-RAT, a novel and efficient framework that synergizes generative data augmentation with robust adversarial training. Our approach first employs a twostep abstractive-expansive (ABEX) pipeline, which leverages a large language model to distill core incident semantics and then uses a generative model to create diverse, highquality synthetic samples for underrepresented classes. Subsequently, a lightweight classifier is trained on the augmented data using a computationally efficient random adversarial training (RAT) protocol, which stochastically applies perturbations to enhance model generalization and robustness without significant overhead. Experimental results on the public OSHA dataset demonstrate that our method achieves new state-of-the-art performance, reaching a macro-F1 score of 90.32% and significantly outperforming previous SOTA and fine-tuned large model baselines. Our work validates that this synergistic strategy is a highly effective and efficient alternative to brute-force fine-tuning for specialized, imbalanced classification tasks. The code is publicly available at:https://github.com/nxcc-lab/ABEX-RAT.

http://arxiv.org/abs/2502.13061
Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection. (1%)
Jingbiao Mei; Jinghong Chen; Guangyu Yang; Weizhe Lin; Bill Byrne
Hateful memes have become a significant concern on the Internet, necessitating robust automated detection systems. While Large Multimodal Models (LMMs) have shown promise in hateful meme detection, they face notable challenges like sub-optimal performance and limited out-of-domain generalization capabilities. Recent studies further reveal the limitations of both supervised fine-tuning (SFT) and in-context learning when applied to LMMs in this setting. To address these issues, we propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs. Analysis reveals that our approach achieves improved robustness under adversarial attacks compared to SFT models. Experiments on six meme classification datasets show that our approach achieves state-of-the-art performance, outperforming larger agentic systems. Moreover, our method generates higher-quality rationales for explaining hateful content compared to standard SFT, enhancing model interpretability. Code available at https://github.com/JingbiaoMei/RGCL

http://arxiv.org/abs/2509.13048
SLasH-DSA: Breaking SLH-DSA Using an Extensible End-To-End Rowhammer Framework. (1%)
Jeremy Boy; Antoon Purnal; Anna Pätschke; Luca Wilke; Thomas Eisenbarth
As quantum computing advances, PQC schemes are adopted to replace classical algorithms. Among them is the SLH-DSA that was recently standardized by NIST and is favored for its conservative security foundations.
  In this work, we present the first software-only universal forgery attack on SLH-DSA, leveraging Rowhammer-induced bit flips to corrupt the internal state and forge signatures. While prior work targeted embedded systems and required physical access, our attack is software-only, targeting commodity desktop and server hardware, significantly broadening the threat model. We demonstrate a full end-to-end attack against all security levels of SLH-DSA in OpenSSL 3.5.1, achieving universal forgery for the highest security level after eight hours of hammering and 36 seconds of post-processing. Our post-processing is informed by a novel complexity analysis that, given a concrete set of faulty signatures, identifies the most promising computational path to pursue.
  To enable the attack, we introduce Swage, a modular and extensible framework for implementing end-to-end Rowhammer-based fault attacks. Swage abstracts and automates key components of practical Rowhammer attacks. Unlike prior tooling, Swage is untangled from the attacked code, making it reusable and suitable for frictionless analysis of different targets. Our findings highlight that even theoretically sound PQC schemes can fail under real-world conditions, underscoring the need for additional implementation hardening or hardware defenses against Rowhammer.

http://arxiv.org/abs/2509.21344
Towards mitigating information leakage when evaluating safety monitors. (1%)
Gerard Boxo; Aman Neelappa; Shivam Raval
White box monitors that analyze model internals offer promising advantages for detecting potentially harmful behaviors in large language models, including lower computational costs and integration into layered defense systems.However, training and evaluating these monitors requires response exemplars that exhibit the target behaviors, typically elicited through prompting or fine-tuning. This presents a challenge when the information used to elicit behaviors inevitably leaks into the data that monitors ingest, inflating their effectiveness. We present a systematic framework for evaluating a monitor's performance in terms of its ability to detect genuine model behavior rather than superficial elicitation artifacts. Furthermore, we propose three novel strategies to evaluate the monitor: content filtering (removing deception-related text from inputs), score filtering (aggregating only over task-relevant tokens), and prompt distilled fine-tuned model organisms (models trained to exhibit deceptive behavior without explicit prompting). Using deception detection as a representative case study, we identify two forms of leakage that inflate monitor performance: elicitation leakage from prompts that explicitly request harmful behavior, and reasoning leakage from models that verbalize their deceptive actions. Through experiments on multiple deception benchmarks, we apply our proposed mitigation strategies and measure performance retention. Our evaluation of the monitors reveal three crucial findings: (1) Content filtering is a good mitigation strategy that allows for a smooth removal of elicitation signal and can decrease probe AUROC by 30\% (2) Score filtering was found to reduce AUROC by 15\% but is not as straightforward to attribute to (3) A finetuned model organism improves monitor evaluations but reduces their performance by upto 40\%, even when re-trained.

http://arxiv.org/abs/2509.11836
A Practical Adversarial Attack against Sequence-based Deep Learning Malware Classifiers. (99%)
Kai Tan; Dongyang Zhan; Lin Ye; Hongli Zhang; Binxing Fang
Sequence-based deep learning models (e.g., RNNs), can detect malware by analyzing its behavioral sequences. Meanwhile, these models are susceptible to adversarial attacks. Attackers can create adversarial samples that alter the sequence characteristics of behavior sequences to deceive malware classifiers. The existing methods for generating adversarial samples typically involve deleting or replacing crucial behaviors in the original data sequences, or inserting benign behaviors that may violate the behavior constraints. However, these methods that directly manipulate sequences make adversarial samples difficult to implement or apply in practice. In this paper, we propose an adversarial attack approach based on Deep Q-Network and a heuristic backtracking search strategy, which can generate perturbation sequences that satisfy practical conditions for successful attacks. Subsequently, we utilize a novel transformation approach that maps modifications back to the source code, thereby avoiding the need to directly modify the behavior log sequences. We conduct an evaluation of our approach, and the results confirm its effectiveness in generating adversarial samples from real-world malware behavior sequences, which have a high success rate in evading anomaly detection models. Furthermore, our approach is practical and can generate adversarial samples while maintaining the functionality of the modified software.

http://arxiv.org/abs/2509.11525
DARD: Dice Adversarial Robustness Distillation against Adversarial Attacks. (99%)
Jing Zou; Shungeng Zhang; Meikang Qiu; Chong Li
Deep learning models are vulnerable to adversarial examples, posing critical security challenges in real-world applications. While Adversarial Training (AT ) is a widely adopted defense mechanism to enhance robustness, it often incurs a trade-off by degrading performance on unperturbed, natural data. Recent efforts have highlighted that larger models exhibit enhanced robustness over their smaller counterparts. In this paper, we empirically demonstrate that such robustness can be systematically distilled from large teacher models into compact student models. To achieve better performance, we introduce Dice Adversarial Robustness Distillation (DARD), a novel method designed to transfer robustness through a tailored knowledge distillation paradigm. Additionally, we propose Dice Projected Gradient Descent (DPGD), an adversarial example generalization method optimized for effective attack. Our extensive experiments demonstrate that the DARD approach consistently outperforms adversarially trained networks with the same architecture, achieving superior robustness and standard accuracy.

http://arxiv.org/abs/2509.11864
NeuroStrike: Neuron-Level Attacks on Aligned LLMs. (98%)
Lichao Wu; Sasha Behrouzi; Mohamadreza Rostami; Maximilian Thang; Stjepan Picek; Ahmad-Reza Sadeghi
Safety alignment is critical for the ethical deployment of large language models (LLMs), guiding them to avoid generating harmful or unethical content. Current alignment techniques, such as supervised fine-tuning and reinforcement learning from human feedback, remain fragile and can be bypassed by carefully crafted adversarial prompts. Unfortunately, such attacks rely on trial and error, lack generalizability across models, and are constrained by scalability and reliability.
  This paper presents NeuroStrike, a novel and generalizable attack framework that exploits a fundamental vulnerability introduced by alignment techniques: the reliance on sparse, specialized safety neurons responsible for detecting and suppressing harmful inputs. We apply NeuroStrike to both white-box and black-box settings: In the white-box setting, NeuroStrike identifies safety neurons through feedforward activation analysis and prunes them during inference to disable safety mechanisms. In the black-box setting, we propose the first LLM profiling attack, which leverages safety neuron transferability by training adversarial prompt generators on open-weight surrogate models and then deploying them against black-box and proprietary targets. We evaluate NeuroStrike on over 20 open-weight LLMs from major LLM developers. By removing less than 0.6% of neurons in targeted layers, NeuroStrike achieves an average attack success rate (ASR) of 76.9% using only vanilla malicious prompts. Moreover, Neurostrike generalizes to four multimodal LLMs with 100% ASR on unsafe image inputs. Safety neurons transfer effectively across architectures, raising ASR to 78.5% on 11 fine-tuned models and 77.7% on five distilled models. The black-box LLM profiling attack achieves an average ASR of 63.7% across five black-box models, including the Google Gemini family.

http://arxiv.org/abs/2212.06123
Security of Deep Reinforcement Learning for Autonomous Driving: A Survey. (96%)
Ambra Demontis; Srishti Gupta; Maura Pintor; Luca Demetrio; Kathrin Grosse; Hsiao-Ying Lin; Chengfang Fang; Battista Biggio; Fabio Roli
Reinforcement learning (RL) enables agents to learn optimal behaviors through interaction with their environment and has been increasingly deployed in safety-critical applications, including autonomous driving. Despite its promise, RL is susceptible to attacks designed either to compromise policy learning or to induce erroneous decisions by trained agents. Although the literature on RL security has grown rapidly and several surveys exist, existing categorizations often fall short in guiding the selection of appropriate defenses for specific systems. In this work, we present a comprehensive survey of 86 recent studies on RL security, addressing these limitations by systematically categorizing attacks and defenses according to defined threat models and single- versus multi-agent settings. Furthermore, we examine the relevance and applicability of state-of-the-art attacks and defense mechanisms within the context of autonomous driving, providing insights to inform the design of robust RL systems.

http://arxiv.org/abs/2507.12932
Enkidu: Universal Frequential Perturbation for Real-Time Audio Privacy Protection against Voice Deepfakes. (89%)
Zhou Feng; Jiahao Chen; Chunyi Zhou; Yuwen Pu; Qingming Li; Tianyu Du; Shouling Ji
The rapid advancement of voice deepfake technologies has raised serious concerns about user audio privacy, as attackers increasingly exploit publicly available voice data to generate convincing fake audio for malicious purposes such as identity theft, financial fraud, and misinformation campaigns. While existing defense methods offer partial protection, they face critical limitations, including weak adaptability to unseen user data, poor scalability to long audio, rigid reliance on white-box knowledge, and high computational and temporal costs during the encryption process. To address these challenges and defend against personalized voice deepfake threats, we propose Enkidu, a novel user-oriented privacy-preserving framework that leverages universal frequential perturbations generated through black-box knowledge and few-shot training on a small amount of user data. These highly malleable frequency-domain noise patches enable real-time, lightweight protection with strong generalization across variable-length audio and robust resistance to voice deepfake attacks, all while preserving perceptual quality and speech intelligibility. Notably, Enkidu achieves over 50 to 200 times processing memory efficiency (as low as 0.004 gigabytes) and 3 to 7000 times runtime efficiency (real-time coefficient as low as 0.004) compared to six state-of-the-art countermeasures. Extensive experiments across six mainstream text-to-speech models and five cutting-edge automated speaker verification models demonstrate the effectiveness, transferability, and practicality of Enkidu in defending against both vanilla and adaptive voice deepfake attacks. Our code is currently available.

http://arxiv.org/abs/2509.14271
Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models. (88%)
Gustavo Sandoval; Denys Fenchenko; Junyao Chen
This paper documents early research conducted in 2022 on defending against prompt injection attacks in large language models, providing historical context for the evolution of this critical security domain. This research focuses on two adversarial attacks against Large Language Models (LLMs): prompt injection and goal hijacking. We examine how to construct these attacks, test them on various LLMs, and compare their effectiveness. We propose and evaluate a novel defense technique called Adversarial Fine-Tuning. Our results show that, without this defense, the attacks succeeded 31\% of the time on GPT-3 series models. When using our Adversarial Fine-Tuning approach, attack success rates were reduced to near zero for smaller GPT-3 variants (Ada, Babbage, Curie), though we note that subsequent research has revealed limitations of fine-tuning-based defenses. We also find that more flexible models exhibit greater vulnerability to these attacks. Consequently, large models such as GPT-3 Davinci are more vulnerable than smaller models like GPT-2. While the specific models tested are now superseded, the core methodology and empirical findings contributed to the foundation of modern prompt injection defense research, including instruction hierarchy systems and constitutional AI approaches.

http://arxiv.org/abs/2509.11745
Removal Attack and Defense on AI-generated Content Latent-based Watermarking. (74%)
De Zhang Lee; Han Fang; Hanyi Wang; Ee-Chien Chang
Digital watermarks can be embedded into AI-generated content (AIGC) by initializing the generation process with starting points sampled from a secret distribution. When combined with pseudorandom error-correcting codes, such watermarked outputs can remain indistinguishable from unwatermarked objects, while maintaining robustness under whitenoise. In this paper, we go beyond indistinguishability and investigate security under removal attacks. We demonstrate that indistinguishability alone does not necessarily guarantee resistance to adversarial removal. Specifically, we propose a novel attack that exploits boundary information leaked by the locations of watermarked objects. This attack significantly reduces the distortion required to remove watermarks -- by up to a factor of $15 \times$ compared to a baseline whitenoise attack under certain settings. To mitigate such attacks, we introduce a defense mechanism that applies a secret transformation to hide the boundary, and prove that the secret transformation effectively rendering any attacker's perturbations equivalent to those of a naive whitenoise adversary. Our empirical evaluations, conducted on multiple versions of Stable Diffusion, validate the effectiveness of both the attack and the proposed defense, highlighting the importance of addressing boundary leakage in latent-based watermarking schemes.

http://arxiv.org/abs/2410.14827
Enhancing Prompt Injection Attacks to LLMs via Poisoning Alignment. (22%)
Zedian Shao; Hongbin Liu; Jaden Mu; Neil Zhenqiang Gong
Prompt injection attack, where an attacker injects a prompt into the original one, aiming to make an Large Language Model (LLM) follow the injected prompt to perform an attacker-chosen task, represent a critical security threat. Existing attacks primarily focus on crafting these injections at inference time, treating the LLM itself as a static target. Our experiments show that these attacks achieve some success, but there is still significant room for improvement. In this work, we introduces a more foundational attack vector: poisoning the LLM's alignment process to amplify the success of future prompt injection attacks. Specifically, we propose PoisonedAlign, a method that strategically creates poisoned alignment samples to poison an LLM's alignment dataset. Our experiments across five LLMs and two alignment datasets show that when even a small fraction of the alignment data is poisoned, the resulting model becomes substantially more vulnerable to a wide range of prompt injection attacks. Crucially, this vulnerability is instilled while the LLM's performance on standard capability benchmarks remains largely unchanged, making the manipulation difficult to detect through automated, general-purpose performance evaluations. The code for implementing the attack is available at https://github.com/Sadcardation/PoisonedAlign.

http://arxiv.org/abs/2509.01909
Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models. (22%)
Ranjie Duan; Jiexi Liu; Xiaojun Jia; Shiji Zhao; Ruoxi Cheng; Fengxiang Wang; Cheng Wei; Yong Xie; Chang Liu; Defeng Li; Yinpeng Dong; Yichi Zhang; Yuefeng Chen; Chongwen Wang; Xingjun Ma; Xingxing Wei; Yang Liu; Hang Su; Jun Zhu; Xinfeng Li; Yitong Sun; Jie Zhang; Jinzhao Hu; Sha Xu; Wenchao Yang; Yitong Yang; Jialing Tao; Hui Xue
Large language models (LLMs) typically deploy safety mechanisms to prevent harmful content generation. Most current approaches focus narrowly on risks posed by malicious actors, often framing risks as adversarial events and relying on defensive refusals. However, in real-world settings, risks also come from non-malicious users seeking help while under psychological distress (e.g., self-harm intentions). In such cases, the model's response can strongly influence the user's next actions. Simple refusals may lead them to repeat, escalate, or move to unsafe platforms, creating worse outcomes. We introduce Constructive Safety Alignment (CSA), a human-centric paradigm that protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results. Implemented in Oyster-I (Oy1), CSA combines game-theoretic anticipation of user reactions, fine-grained risk boundary discovery, and interpretable reasoning control, turning safety into a trust-building process. Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities. On our Constructive Benchmark, it shows strong constructive engagement, close to GPT-5, and unmatched robustness on the Strata-Sword jailbreak dataset, nearing GPT-o1 levels. By shifting from refusal-first to guidance-first safety, CSA redefines the model-user relationship, aiming for systems that are not just safe, but meaningfully helpful. We release Oy1, code, and the benchmark to support responsible, user-centered AI.

http://arxiv.org/abs/2406.15444
Cutting Through the Noise: Boosting LLM Performance on Math Word Problems. (16%)
Ujjwala Anantheswaran; Himanshu Gupta; Kevin Scaria; Shreyas Verma; Chitta Baral; Swaroop Mishra
Large Language Models (LLMs) excel at various tasks, including solving math word problems (MWPs), but struggle with real-world problems containing irrelevant information. To address this, we propose a prompting framework that generates adversarial variants of MWPs by adding irrelevant variables. We introduce a dataset, PROBLEMATHIC, containing both adversarial and non-adversarial MWPs. Our experiments reveal that LLMs are susceptible to distraction by numerical noise, resulting in an average relative performance drop of ~26% on adversarial MWPs. To mitigate this, we fine-tune LLMs (Llama-2, Mistral) on the adversarial samples from our dataset. Fine-tuning on adversarial training instances improves performance on adversarial MWPs by ~8%, indicating increased robustness to noise and improved ability to identify relevant data for reasoning. Finally, to assess the generalizability of our prompting framework, we introduce GSM-8K-Adv, an adversarial variant of the GSM-8K benchmark. LLMs continue to struggle when faced with adversarial information, reducing performance by up to 6%.

http://arxiv.org/abs/2509.11683
An Unsupervised Learning Approach For A Reliable Profiling Of Cyber Threat Actors Reported Globally Based On Complete Contextual Information Of Cyber Attacks. (10%)
Sawera Shahid; Umara Noor; Zahid Rashid
Cyber attacks are rapidly increasing with the advancement of technology and there is no protection for our information. To prevent future cyberattacks it is critical to promptly recognize cyberattacks and establish strong defense mechanisms against them. To respond to cybersecurity threats immediately, it is essential to examine the attackers skills, knowledge, and behaviors with the goal of evaluating their impact on the system and comprehending the traits associated with these attacks. Creating a profile of cyber threat actors based on their traits or patterns of behavior can help to create effective defenses against cyberattacks in advance. In the current literature, multiple supervised machine learning based approaches considered a smaller number of features for attacker profiling that are reported in textual cyber threat incident documents although these profiles have been developed based on the security experts own perception, we cannot rely on them. Supervised machine learning approaches strictly depend upon the structure data set. This usually leads to a two step process where we first have to establish a structured data set before we can analyze it and then employ it to construct defense mechanisms, which takes time. In this paper, an unsupervised efficient agglomerative hierarchal clustering technique is proposed for profiling cybercriminal groups based on their comprehensive contextual threat information in order to address the aforementioned issues. The main objective of this report is to identify the relationship between cyber threat actors based on their common features, aggregate them, and also profile cyber criminal groups.

http://arxiv.org/abs/2509.11615
Cyber Threat Hunting: Non-Parametric Mining of Attack Patterns from Cyber Threat Intelligence for Precise Threats Attribution. (1%)
Rimsha Kanwal; Umara Noor; Zafar Iqbal; Zahid Rashid
With the ever-changing landscape of cyber threats, identifying their origin has become paramount, surpassing the simple task of attack classification. Cyber threat attribution gives security analysts the insights they need to device effective threat mitigation strategies. Such strategies empower enterprises to proactively detect and defend against future cyber-attacks. However, existing approaches exhibit limitations in accurately identifying threat actors, leading to low precision and a significant occurrence of false positives. Machine learning offers the potential to automate certain aspects of cyber threat attribution. The distributed nature of information regarding cyber threat actors and their intricate attack methodologies has hindered substantial progress in this domain. Cybersecurity analysts deal with an ever-expanding collection of cyber threat intelligence documents. While these documents hold valuable insights, their sheer volume challenges efficient organization and retrieval of pertinent information. To assist the cybersecurity analyst activities, we propose a machine learning based approach featuring visually interactive analytics tool named the Cyber-Attack Pattern Explorer (CAPE), designed to facilitate efficient information discovery by employing interactive visualization and mining techniques. In the proposed system, a non-parametric mining technique is proposed to create a dataset for identifying the attack patterns within cyber threat intelligence documents. These attack patterns align semantically with commonly employed themes ensuring ease of interpretation. The extracted dataset is used for training of proposed machine learning algorithms that enables the attribution of cyber threats with respective to the actors.

http://arxiv.org/abs/2509.11187
DMLDroid: Deep Multimodal Fusion Framework for Android Malware Detection with Resilience to Code Obfuscation and Adversarial Perturbations. (99%)
Doan Minh Trung; Tien Duc Anh Hao; Luong Hoang Minh; Nghi Hoang Khoa; Nguyen Tan Cam; Van-Hau Pham; Phan The Duy
In recent years, learning-based Android malware detection has seen significant advancements, with detectors generally falling into three categories: string-based, image-based, and graph-based approaches. While these methods have shown strong detection performance, they often struggle to sustain robustness in real-world settings, particularly when facing code obfuscation and adversarial examples (AEs). Deep multimodal learning has emerged as a promising solution, leveraging the strengths of multiple feature types to enhance robustness and generalization. However, a systematic investigation of multimodal fusion for both accuracy and resilience remains underexplored. In this study, we propose DMLDroid, an Android malware detection based on multimodal fusion that leverages three different representations of malware features, including permissions & intents (tabular-based), DEX file representations (image-based), and API calls (graph-derived sequence-based). We conduct exhaustive experiments independently on each feature, as well as in combination, using different fusion strategies. Experimental results on the CICMalDroid 2020 dataset demonstrate that our multimodal approach with the dynamic weighted fusion mechanism achieves high performance, reaching 97.98% accuracy and 98.67% F1-score on original malware detection. Notably, the proposed method maintains strong robustness, sustaining over 98% accuracy and 98% F1-score under both obfuscation and adversarial attack scenarios. Our findings highlight the benefits of multimodal fusion in improving both detection accuracy and robustness against evolving Android malware threats.

http://arxiv.org/abs/2509.09112
Character-Level Perturbations Disrupt LLM Watermarks. (92%)
Zhaoxi Zhang; Xiaomei Zhang; Yanjun Zhang; He Zhang; Shirui Pan; Bo Liu; Asif Qumer Gill; Leo Yu Zhang
Large Language Model (LLM) watermarking embeds detectable signals into generated text for copyright protection, misuse prevention, and content detection. While prior studies evaluate robustness using watermark removal attacks, these methods are often suboptimal, creating the misconception that effective removal requires large perturbations or powerful adversaries.
  To bridge the gap, we first formalize the system model for LLM watermark, and characterize two realistic threat models constrained on limited access to the watermark detector. We then analyze how different types of perturbation vary in their attack range, i.e., the number of tokens they can affect with a single edit. We observe that character-level perturbations (e.g., typos, swaps, deletions, homoglyphs) can influence multiple tokens simultaneously by disrupting the tokenization process. We demonstrate that character-level perturbations are significantly more effective for watermark removal under the most restrictive threat model. We further propose guided removal attacks based on the Genetic Algorithm (GA) that uses a reference detector for optimization. Under a practical threat model with limited black-box queries to the watermark detector, our method demonstrates strong removal performance. Experiments confirm the superiority of character-level perturbations and the effectiveness of the GA in removing watermarks under realistic constraints. Additionally, we argue there is an adversarial dilemma when considering potential defenses: any fixed defense can be bypassed by a suitable perturbation strategy. Motivated by this principle, we propose an adaptive compound character-level attack. Experimental results show that this approach can effectively defeat the defenses. Our findings highlight significant vulnerabilities in existing LLM watermark schemes and underline the urgency for the development of new robust mechanisms.

http://arxiv.org/abs/2509.11220
ANROT-HELANet: Adverserially and Naturally Robust Attention-Based Aggregation Network via The Hellinger Distance for Few-Shot Classification. (76%)
Gao Yu Lee; Tanmoy Dam; Md Meftahul Ferdaus; Daniel Puiu Poenar; Vu N. Duong
Few-Shot Learning (FSL), which involves learning to generalize using only a few data samples, has demonstrated promising and superior performances to ordinary CNN methods. While Bayesian based estimation approaches using Kullback-Leibler (KL) divergence have shown improvements, they remain vulnerable to adversarial attacks and natural noises. We introduce ANROT-HELANet, an Adversarially and Naturally RObusT Hellinger Aggregation Network that significantly advances the state-of-the-art in FSL robustness and performance. Our approach implements an adversarially and naturally robust Hellinger distance-based feature class aggregation scheme, demonstrating resilience to adversarial perturbations up to $ε=0.30$ and Gaussian noise up to $σ=0.30$. The network achieves substantial improvements across benchmark datasets, including gains of 1.20\% and 1.40\% for 1-shot and 5-shot scenarios on miniImageNet respectively. We introduce a novel Hellinger Similarity contrastive loss function that generalizes cosine similarity contrastive loss for variational few-shot inference scenarios. Our approach also achieves superior image reconstruction quality with a FID score of 2.75, outperforming traditional VAE (3.43) and WAE (3.38) approaches. Extensive experiments conducted on four few-shot benchmarked datasets verify that ANROT-HELANet's combination of Hellinger distance-based feature aggregation, attention mechanisms, and our novel loss function establishes new state-of-the-art performance while maintaining robustness against both adversarial and natural perturbations. Our code repository will be available at https://github.com/GreedYLearner1146/ANROT-HELANet/tree/main.

http://arxiv.org/abs/2509.11128
ENJ: Optimizing Noise with Genetic Algorithms to Jailbreak LSMs. (69%)
Yibo Zhang; Liang Lin
The widespread application of Large Speech Models (LSMs) has made their security risks increasingly prominent. Traditional speech adversarial attack methods face challenges in balancing effectiveness and stealth. This paper proposes Evolutionary Noise Jailbreak (ENJ), which utilizes a genetic algorithm to transform environmental noise from a passive interference into an actively optimizable attack carrier for jailbreaking LSMs. Through operations such as population initialization, crossover fusion, and probabilistic mutation, this method iteratively evolves a series of audio samples that fuse malicious instructions with background noise. These samples sound like harmless noise to humans but can induce the model to parse and execute harmful commands. Extensive experiments on multiple mainstream speech models show that ENJ's attack effectiveness is significantly superior to existing baseline methods. This research reveals the dual role of noise in speech security and provides new critical insights for model security defense in complex acoustic environments.

http://arxiv.org/abs/2504.11358
DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks. (61%)
Yupei Liu; Yuqi Jia; Jinyuan Jia; Dawn Song; Neil Zhenqiang Gong
LLM-integrated applications and agents are vulnerable to prompt injection attacks, where an attacker injects prompts into their inputs to induce attacker-desired outputs. A detection method aims to determine whether a given input is contaminated by an injected prompt. However, existing detection methods have limited effectiveness against state-of-the-art attacks, let alone adaptive ones. In this work, we propose DataSentinel, a game-theoretic method to detect prompt injection attacks. Specifically, DataSentinel fine-tunes an LLM to detect inputs contaminated with injected prompts that are strategically adapted to evade detection. We formulate this as a minimax optimization problem, with the objective of fine-tuning the LLM to detect strong adaptive attacks. Furthermore, we propose a gradient-based method to solve the minimax optimization problem by alternating between the inner max and outer min problems. Our evaluation results on multiple benchmark datasets and LLMs show that DataSentinel effectively detects both existing and adaptive prompt injection attacks.

http://arxiv.org/abs/2509.11159
Stabilizing Data-Free Model Extraction. (33%)
Dat-Thinh Nguyen; Kim-Hung Le; Nhien-An Le-Khac
Model extraction is a severe threat to Machine Learning-as-a-Service systems, especially through data-free approaches, where dishonest users can replicate the functionality of a black-box target model without access to realistic data. Despite recent advancements, existing data-free model extraction methods suffer from the oscillating accuracy of the substitute model. This oscillation, which could be attributed to the constant shift in the generated data distribution during the attack, makes the attack impractical since the optimal substitute model cannot be determined without access to the target model's in-distribution data. Hence, we propose MetaDFME, a novel data-free model extraction method that employs meta-learning in the generator training to reduce the distribution shift, aiming to mitigate the substitute model's accuracy oscillation. In detail, we train our generator to iteratively capture the meta-representations of the synthetic data during the attack. These meta-representations can be adapted with a few steps to produce data that facilitates the substitute model to learn from the target model while reducing the effect of distribution shifts. Our experiments on popular baseline image datasets, MNIST, SVHN, CIFAR-10, and CIFAR-100, demonstrate that MetaDFME outperforms the current state-of-the-art data-free model extraction method while exhibiting a more stable substitute model's accuracy during the attack.

http://arxiv.org/abs/2506.02040
Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystem. (22%)
Hao Song; Yiming Shen; Wenxuan Luo; Leixin Guo; Ting Chen; Jiashui Wang; Beibei Li; Xiaosong Zhang; Jiachi Chen
The Model Context Protocol (MCP) is an emerging standard designed to enable seamless interaction between Large Language Model (LLM) applications and external tools or resources. Within a short period, thousands of MCP services have been developed and deployed. However, the client-server integration architecture inherent in MCP may expand the attack surface against LLM Agent systems, introducing new vulnerabilities that allow attackers to exploit by designing malicious MCP servers. In this paper, we present the first end-to-end empirical evaluation of attack vectors targeting the MCP ecosystem. We identify four categories of attacks, i.e., Tool Poisoning Attacks, Puppet Attacks, Rug Pull Attacks, and Exploitation via Malicious External Resources. To evaluate their feasibility, we conduct experiments following the typical steps of launching an attack through malicious MCP servers: upload -> download -> attack. Specifically, we first construct malicious MCP servers and successfully upload them to three widely used MCP aggregation platforms. The results indicate that current audit mechanisms are insufficient to identify and prevent these threats. Next, through a user study and interview with 20 participants, we demonstrate that users struggle to identify malicious MCP servers and often unknowingly install them from aggregator platforms. Finally, we empirically demonstrate that these attacks can trigger harmful actions within the user's local environment, such as accessing private files or controlling devices to transfer digital assets. Additionally, based on interview results, we discuss four key challenges faced by the current MCP security ecosystem. These findings underscore the urgent need for robust security mechanisms to defend against malicious MCP servers and ensure the safe deployment of increasingly autonomous LLM agents.

http://arxiv.org/abs/2509.13353
Hybrid Quantum-Classical Model for Image Classification. (16%)
Muhammad Adnan Shahzad
This study presents a systematic comparison between hybrid quantum-classical neural networks and purely classical models across three benchmark datasets (MNIST, CIFAR100, and STL10) to evaluate their performance, efficiency, and robustness. The hybrid models integrate parameterized quantum circuits with classical deep learning architectures, while the classical counterparts use conventional convolutional neural networks (CNNs). Experiments were conducted over 50 training epochs for each dataset, with evaluations on validation accuracy, test accuracy, training time, computational resource usage, and adversarial robustness (tested with $ε=0.1$ perturbations).Key findings demonstrate that hybrid models consistently outperform classical models in final accuracy, achieving {99.38\% (MNIST), 41.69\% (CIFAR100), and 74.05\% (STL10) validation accuracy, compared to classical benchmarks of 98.21\%, 32.25\%, and 63.76\%, respectively. Notably, the hybrid advantage scales with dataset complexity, showing the most significant gains on CIFAR100 (+9.44\%) and STL10 (+10.29\%). Hybrid models also train 5--12$\times$ faster (e.g., 21.23s vs. 108.44s per epoch on MNIST) and use 6--32\% fewer parameters} while maintaining superior generalization to unseen test data.Adversarial robustness tests reveal that hybrid models are significantly more resilient on simpler datasets (e.g., 45.27\% robust accuracy on MNIST vs. 10.80\% for classical) but show comparable fragility on complex datasets like CIFAR100 ($\sim$1\% robustness for both). Resource efficiency analyses indicate that hybrid models consume less memory (4--5GB vs. 5--6GB for classical) and lower CPU utilization (9.5\% vs. 23.2\% on average).These results suggest that hybrid quantum-classical architectures offer compelling advantages in accuracy, training efficiency, and parameter scalability, particularly for complex vision tasks.

http://arxiv.org/abs/2509.11250
Realistic Environmental Injection Attacks on GUI Agents. (16%)
Yitong Zhang; Ximo Li; Liyi Cai; Jia Li
GUI agents built on LVLMs are increasingly used to interact with websites. However, their exposure to open-world content makes them vulnerable to Environmental Injection Attacks (EIAs) that hijack agent behavior via webpage elements. Many recent studies assume the attacker to be a regular user who can only upload a single trigger image, which is more realistic than earlier assumptions of website-level administrative control. However, these works still fall short of realism: (1) the trigger's position and surrounding context remain largely fixed between training and testing, failing to capture the dynamic nature of real webpages and (2) the trigger often occupies an unrealistically large area, whereas real-world images are typically small. To better reflect real-world scenarios, we introduce a more realistic threat model where the attacker is a regular user and the trigger image is small and embedded within a dynamically changing environment. As a result, existing attacks prove largely ineffective under this threat model.
  To better expose the vulnerabilities of GUI agents, we propose Chameleon, an attack framework with two main novelties. The first is LLM-Driven Environment Simulation, which automatically generates diverse and high-fidelity webpage simulations. The second is Attention Black Hole, which transforms attention weights into explicit supervisory signals that guide the agent's focus toward the trigger region. We evaluate Chameleon on 6 realistic websites and 4 representative LVLM-powered GUI agents, where it significantly outperforms existing methods. Ablation studies confirm that both novelties are critical to performance. Our findings reveal underexplored vulnerabilities in modern GUI agents and establish a robust foundation for future research on defense in open-world GUI agent systems. The code is publicly available at https://github.com/zhangyitonggg/attack2gui.

http://arxiv.org/abs/2509.11337
On the Escaping Efficiency of Distributed Adversarial Training Algorithms. (13%)
Ying Cao; Kun Yuan; Ali H. Sayed
Adversarial training has been widely studied in recent years due to its role in improving model robustness against adversarial attacks. This paper focuses on comparing different distributed adversarial training algorithms--including centralized and decentralized strategies--within multi-agent learning environments. Previous studies have highlighted the importance of model flatness in determining robustness. To this end, we develop a general theoretical framework to study the escaping efficiency of these algorithms from local minima, which is closely related to the flatness of the resulting models. We show that when the perturbation bound is sufficiently small (i.e., when the attack strength is relatively mild) and a large batch size is used, decentralized adversarial training algorithms--including consensus and diffusion--are guaranteed to escape faster from local minima than the centralized strategy, thereby favoring flatter minima. However, as the perturbation bound increases, this trend may no longer hold. In the simulation results, we illustrate our theoretical findings and systematically compare the performance of models obtained through decentralized and centralized adversarial training algorithms. The results highlight the potential of decentralized strategies to enhance the robustness of models in distributed settings.

http://arxiv.org/abs/2509.11080
Membership Inference Attacks on Recommender System: A Survey. (5%)
Jiajie He; Yuechun Gu; Keke Chen; Xintong Chen
Recommender systems (RecSys) have been widely applied to various applications, including E-commerce, finance, healthcare, social media and have become increasingly influential in shaping user behavior and decision-making, highlighting their growing impact in various domains. However, recent studies have shown that RecSys are vulnerable to membership inference attacks (MIAs), which aim to infer whether user interaction record was used to train a target model or not. MIAs on RecSys models can directly lead to a privacy breach. For example, via identifying the fact that a purchase record that has been used to train a RecSys associated with a specific user, an attacker can infer that user's special quirks. In recent years, MIAs have been shown to be effective on other ML tasks, e.g., classification models and natural language processing. However, traditional MIAs are ill-suited for RecSys due to the unseen posterior probability. Although MIAs on RecSys form a newly emerging and rapidly growing research area, there has been no systematic survey on this topic yet. In this article, we conduct the first comprehensive survey on RecSys MIAs. This survey offers a comprehensive review of the latest advancements in RecSys MIAs, exploring the design principles, challenges, attack and defense associated with this emerging field. We provide a unified taxonomy that categorizes different RecSys MIAs based on their characterizations and discuss their pros and cons. Based on the limitations and gaps identified in this survey, we point out several promising future research directions to inspire the researchers who wish to follow this area. This survey not only serves as a reference for the research community but also provides a clear description for researchers outside this research domain.

http://arxiv.org/abs/2508.02881
Optimizing Preventive and Reactive Defense Resource Allocation with Uncertain Sensor Signals. (4%)
Faezeh Shojaeighadikolaei; Shouhuai Xu; Keith Paarporn
Cyber attacks continue to be a cause of concern despite advances in cyber defense techniques. Although cyber attacks cannot be fully prevented, standard decision-making frameworks typically focus on how to prevent them from succeeding, without considering the cost of cleaning up the damages incurred by successful attacks. This motivates us to investigate a new resource allocation problem formulated in this paper: The defender must decide how to split its investment between preventive defenses, which aim to harden nodes from attacks, and reactive defenses, which aim to quickly clean up the compromised nodes. This encounters a challenge imposed by the uncertainty associated with the observation, or sensor signal, whether a node is truly compromised or not; this uncertainty is real because attack detectors are not perfect. We investigate how the quality of sensor signals impacts the defender's strategic investment in the two types of defense, and ultimately the level of security that can be achieved. In particular, we show that the optimal investment in preventive resources increases, and thus reactive resource investment decreases, with higher sensor quality. We also show that the defender's performance improvement, relative to a baseline of no sensors employed, is maximal when the attacker can only achieve low attack success probabilities.

http://arxiv.org/abs/2509.11249
Make Identity Unextractable yet Perceptible: Synthesis-Based Privacy Protection for Subject Faces in Photos. (2%)
Tao Wang; Yushu Zhang; Xiangli Xiao; Kun Xu; Lin Yuan; Wenying Wen; Yuming Fang
Deep learning-based face recognition (FR) technology exacerbates privacy concerns in photo sharing. In response, the research community developed a suite of anti-FR methods to block identity extraction by unauthorized FR systems. Benefiting from quasi-imperceptible alteration, perturbation-based methods are well-suited for privacy protection of subject faces in photos, as they allow familiar persons to recognize subjects via naked eyes. However, we reveal that perturbation-based methods provide a false sense of privacy through theoretical analysis and experimental validation.
  Therefore, new alternative solutions should be found to protect subject faces. In this paper, we explore synthesis-based methods as a promising solution, whose challenge is to enable familiar persons to recognize subjects. To solve the challenge, we present a key insight: In most photo sharing scenarios, familiar persons recognize subjects through identity perception rather than meticulous face analysis. Based on the insight, we propose the first synthesis-based method dedicated to subject faces, i.e., PerceptFace, which can make identity unextractable yet perceptible. To enhance identity perception, a new perceptual similarity loss is designed for faces, reducing the alteration in regions of high sensitivity to human vision.
  As a synthesis-based method, PerceptFace can inherently provide reliable identity protection. Meanwhile, out of the confine of meticulous face analysis, PerceptFace focuses on identity perception from a more practical scenario, which is also enhanced by the designed perceptual similarity loss. Sufficient experiments show that PerceptFace achieves a superior trade-off between identity protection and identity perception compared to existing methods. We provide a public API of PerceptFace and believe that it has great potential to become a practical anti-FR tool.

http://arxiv.org/abs/2509.11191
RanAT4BIE: Random Adversarial Training for Biomedical Information Extraction. (1%)
Jian Chen; Shengyi Lv; Leilei Su
We introduce random adversarial training (RAT), a novel framework successfully applied to biomedical information extraction (BioIE) tasks. Building on PubMedBERT as the foundational architecture, our study first validates the effectiveness of conventional adversarial training in enhancing pre-trained language models' performance on BioIE tasks. While adversarial training yields significant improvements across various performance metrics, it also introduces considerable computational overhead. To address this limitation, we propose RAT as an efficiency solution for biomedical information extraction. This framework strategically integrates random sampling mechanisms with adversarial training principles, achieving dual objectives: enhanced model generalization and robustness while significantly reducing computational costs. Through comprehensive evaluations, RAT demonstrates superior performance compared to baseline models in BioIE tasks. The results highlight RAT's potential as a transformative framework for biomedical natural language processing, offering a balanced solution to the model performance and computational efficiency.

http://arxiv.org/abs/2509.11265
SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixing. (1%)
Qiuhao Liu; Ling Li; Yao Lu; Qi Xuan; Zhaowei Zhu; Jiaheng Wei
Deep neural networks tend to memorize noisy labels, severely degrading their generalization performance. Although Mixup has demonstrated effectiveness in improving generalization and robustness, existing Mixup-based methods typically perform indiscriminate mixing without principled guidance on sample selection and mixing strategy, inadvertently propagating noisy supervision. To overcome these limitations, we propose SelectMix, a confidence-guided mixing framework explicitly tailored for noisy labels. SelectMix first identifies potentially noisy or ambiguous samples through confidence based mismatch analysis using K-fold cross-validation, then selectively blends identified uncertain samples with confidently predicted peers from their potential classes. Furthermore, SelectMix employs soft labels derived from all classes involved in the mixing process, ensuring the labels accurately represent the composition of the mixed samples, thus aligning supervision signals closely with the actual mixed inputs. Through extensive theoretical analysis and empirical evaluations on multiple synthetic (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100) and real-world benchmark datasets (CIFAR-N, MNIST and Clothing1M), we demonstrate that SelectMix consistently outperforms strong baseline methods, validating its effectiveness and robustness in learning with noisy labels.

http://arxiv.org/abs/2508.17456
Adversarial Examples Are Not Bugs, They Are Superposition. (98%)
Liv Gorton; Owen Lewis
Adversarial examples -- inputs with imperceptible perturbations that fool neural networks -- remain one of deep learning's most perplexing phenomena despite nearly a decade of research. While numerous defenses and explanations have been proposed, there is no consensus on the fundamental mechanism. One underexplored hypothesis is that superposition, a concept from mechanistic interpretability, may be a major contributing factor, or even the primary cause. We present four lines of evidence in support of this hypothesis, greatly extending prior arguments by Elhage et al. (2022): (1) superposition can theoretically explain a range of adversarial phenomena, (2) in toy models, intervening on superposition controls robustness, (3) in toy models, intervening on robustness (via adversarial training) controls superposition, and (4) in ResNet18, intervening on robustness (via adversarial training) controls superposition.

http://arxiv.org/abs/2509.10913
Robustifying Diffusion-Denoised Smoothing Against Covariate Shift. (67%)
Ali Hedayatnia; Mostafa Tavassolipour; Babak Nadjar Araabi; Abdol-Hossein Vahabie
Randomized smoothing is a well-established method for achieving certified robustness against l2-adversarial perturbations. By incorporating a denoiser before the base classifier, pretrained classifiers can be seamlessly integrated into randomized smoothing without significant performance degradation. Among existing methods, Diffusion Denoised Smoothing - where a pretrained denoising diffusion model serves as the denoiser - has produced state-of-the-art results. However, we show that employing a denoising diffusion model introduces a covariate shift via misestimation of the added noise, ultimately degrading the smoothed classifier's performance. To address this issue, we propose a novel adversarial objective function focused on the added noise of the denoising diffusion model. This approach is inspired by our understanding of the origin of the covariate shift. Our goal is to train the base classifier to ensure it is robust against the covariate shift introduced by the denoiser. Our method significantly improves certified accuracy across three standard classification benchmarks - MNIST, CIFAR-10, and ImageNet - achieving new state-of-the-art performance in l2-adversarial perturbations. Our implementation is publicly available at https://github.com/ahedayat/Robustifying-DDS-Against-Covariate-Shift

http://arxiv.org/abs/2503.14284
Entente: Cross-silo Intrusion Detection on Network Log Graphs with Federated Learning. (1%)
Jiacen Xu; Chenang Li; Yu Zheng; Zhou Li
Graph-based Network Intrusion Detection Systems (GNIDS) have gained significant momentum in detecting sophisticated cyber-attacks, such as Advanced Persistent Threats (APTs), within and across organizational boundaries. Though achieving satisfying detection accuracy and demonstrating adaptability to ever-changing attacks and normal patterns, existing GNIDS predominantly assume a centralized data setting. However, flexible data collection is not always realistic or achievable due to increasing constraints from privacy regulations and operational limitations. We argue that the practical development of GNIDS requires accounting for distributed collection settings and we leverage Federated Learning (FL) as a viable paradigm to address this prominent challenge. We observe that naively applying FL to GNIDS is unlikely to be effective, due to issues like graph heterogeneity over clients and the diverse design choices taken by different GNIDS. We address these issues with a set of novel techniques tailored to the graph datasets, including reference graph synthesis, graph sketching and adaptive contribution scaling, eventually developing a new system Entente. By leveraging the domain knowledge, Entente can achieve effectiveness, scalability and robustness simultaneously. Empirical evaluation on the large-scale LANL, OpTC and Pivoting datasets shows that Entente outperforms the SOTA FL baselines. We also evaluate Entente under FL poisoning attacks tailored to the GNIDS setting, showing the robustness by bounding the attack success rate to low values. Overall, our study suggests a promising direction to build cross-silo GNIDS.

http://arxiv.org/abs/2307.08327
Analyzing the Impact of Adversarial Examples on Explainable Machine Learning. (99%)
Prathyusha Devabhakthini; Sasmita Parida; Raj Mani Shukla; Suvendu Chandan Nayak; Tapadhir Das
Adversarial attacks are a type of attack on machine learning models where an attacker deliberately modifies the inputs to cause the model to make incorrect predictions. Adversarial attacks can have serious consequences, particularly in applications such as autonomous vehicles, medical diagnosis, and security systems. Work on the vulnerability of deep learning models to adversarial attacks has shown that it is very easy to make samples that make a model predict things that it doesn't want to. In this work, we analyze the impact of model interpretability due to adversarial attacks on text classification problems. We develop an ML-based classification model for text data. Then, we introduce the adversarial perturbations on the text data to understand the classification performance after the attack. Subsequently, we analyze and interpret the model's explainability before and after the attack

http://arxiv.org/abs/2509.10298
Adversarial robustness through Lipschitz-Guided Stochastic Depth in Neural Networks. (78%)
Laith Nayal; Mahmoud Mousatat; Bader Rasheed
Deep neural networks and Vision Transformers achieve state-of-the-art performance in computer vision but are highly vulnerable to adversarial perturbations. Standard defenses often incur high computational cost or lack formal guarantees. We propose a Lipschitz-guided stochastic depth (DropPath) method, where drop probabilities increase with depth to control the effective Lipschitz constant of the network. This approach regularizes deeper layers, improving robustness while preserving clean accuracy and reducing computation. Experiments on CIFAR-10 with ViT-Tiny show that our custom depth-dependent schedule maintains near-baseline clean accuracy, enhances robustness under FGSM, PGD-20, and AutoAttack, and significantly reduces FLOPs compared to baseline and linear DropPath schedules.

http://arxiv.org/abs/2509.05367
Between a Rock and a Hard Place: Exploiting Ethical Reasoning to Jailbreak LLMs. (50%)
Shei Pern Chua; Zhen Leng Thai; Teh Kai Jun; Xiao Li; Xiaolin Hu
Large language models (LLMs) have undergone safety alignment efforts to mitigate harmful outputs. However, as LLMs become more sophisticated in reasoning, their intelligence may introduce new security risks. While traditional jailbreak attacks relied on singlestep attacks, multi-turn jailbreak strategies that adapt dynamically to context remain underexplored. In this work, we introduce TRIAL (Trolley-problem Reasoning for Interactive Attack Logic), a framework that leverages LLMs ethical reasoning to bypass their safeguards. TRIAL embeds adversarial goals within ethical dilemmas modeled on the trolley problem. TRIAL demonstrates high jailbreak success rates towards both open and close-source models. Our findings underscore a fundamental limitation in AI safety: as models gain advanced reasoning abilities, the nature of their alignment may inadvertently allow for more covert security vulnerabilities to be exploited. TRIAL raises an urgent need in reevaluating safety alignment oversight strategies, as current safeguards may prove insufficient against context-aware adversarial attack.

http://arxiv.org/abs/2509.10359
Immunizing Images from Text to Image Editing via Adversarial Cross-Attention. (11%)
Matteo Trippodo; Federico Becattini; Lorenzo Seidenari
Recent advances in text-based image editing have enabled fine-grained manipulation of visual content guided by natural language. However, such methods are susceptible to adversarial attacks. In this work, we propose a novel attack that targets the visual component of editing methods. We introduce Attention Attack, which disrupts the cross-attention between a textual prompt and the visual representation of the image by using an automatically generated caption of the source image as a proxy for the edit prompt. This breaks the alignment between the contents of the image and their textual description, without requiring knowledge of the editing method or the editing prompt. Reflecting on the reliability of existing metrics for immunization success, we propose two novel evaluation strategies: Caption Similarity, which quantifies semantic consistency between original and adversarial edits, and semantic Intersection over Union (IoU), which measures spatial layout disruption via segmentation masks. Experiments conducted on the TEDBench++ benchmark demonstrate that our attack significantly degrades editing performance while remaining imperceptible.

http://arxiv.org/abs/2402.05675
Is Adversarial Training with Compressed Datasets Effective? (10%)
Tong Chen; Raghavendra Selvan
Dataset Condensation (DC) refers to the recent class of dataset compression methods that generate a smaller, synthetic, dataset from a larger dataset. This synthetic dataset aims to retain the essential information of the original dataset, enabling models trained on it to achieve performance levels comparable to those trained on the full dataset. Most current DC methods have mainly concerned with achieving high test performance with limited data budget, and have not directly addressed the question of adversarial robustness. In this work, we investigate the impact of adversarial robustness on models trained with compressed datasets. We show that the compressed datasets obtained from DC methods are not effective in transferring adversarial robustness to models. As a solution to improve dataset compression efficiency and adversarial robustness simultaneously, we present a robustness-aware dataset compression method based on finding the Minimal Finite Covering (MFC) of the dataset. The proposed method is (1) provably robust by minimizing the generalized adversarial loss, (2) more effective than DC methods when applying adversarial training over MFC, (3) obtained by a one-time computation and is applicable for any model.

http://arxiv.org/abs/2505.19260
ALRPHFS: Adversarially Learned Risk Patterns with Hierarchical Fast \& Slow Reasoning for Robust Agent Defense. (8%)
Shiyu Xiang; Tong Zhang; Ronghao Chen
LLM Agents are becoming central to intelligent systems. However, their deployment raises serious safety concerns. Existing defenses largely rely on "Safety Checks", which struggle to capture the complex semantic risks posed by harmful user inputs or unsafe agent behaviors - creating a significant semantic gap between safety checks and real-world risks. To bridge this gap, we propose a novel defense framework, ALRPHFS (Adversarially Learned Risk Patterns with Hierarchical Fast & Slow Reasoning). ALRPHFS consists of two core components: (1) an offline adversarial self-learning loop to iteratively refine a generalizable and balanced library of risk patterns, substantially enhancing robustness without retraining the base LLM, and (2) an online hierarchical fast & slow reasoning engine that balances detection effectiveness with computational efficiency. Experimental results demonstrate that our approach achieves superior overall performance compared to existing baselines, achieving a best-in-class average accuracy of 80% and exhibiting strong generalizability across agents and tasks.

http://arxiv.org/abs/2503.12188
Multi-Agent Systems Execute Arbitrary Malicious Code. (2%)
Harold Triedman; Rishi Jha; Vitaly Shmatikov
Multi-agent systems coordinate LLM-based agents to perform tasks on users' behalf. In real-world applications, multi-agent systems will inevitably interact with untrusted inputs, such as malicious Web content, files, email attachments, and more.
  Using several recently proposed multi-agent frameworks as concrete examples, we demonstrate that adversarial content can hijack control and communication within the system to invoke unsafe agents and functionalities. This results in a complete security breach, up to execution of arbitrary malicious code on the user's device or exfiltration of sensitive data from the user's containerized environment. For example, when agents are instantiated with GPT-4o, Web-based attacks successfully cause the multi-agent system execute arbitrary malicious code in 58-90\% of trials (depending on the orchestrator). In some model-orchestrator configurations, the attack success rate is 100\%. We also demonstrate that these attacks succeed even if individual agents are not susceptible to direct or indirect prompt injection, and even if they refuse to perform harmful actions. We hope that these results will motivate development of trust and security models for multi-agent systems before they are widely deployed.

http://arxiv.org/abs/2505.19613
TESSER: Transfer-Enhancing Adversarial Attacks from Vision Transformers via Spectral and Semantic Regularization. (99%)
Amira Guesmi; Bassem Ouni; Muhammad Shafique
Adversarial transferability remains a critical challenge in evaluating the robustness of deep neural networks. In security-critical applications, transferability enables black-box attacks without access to model internals, making it a key concern for real-world adversarial threat assessment. While Vision Transformers (ViTs) have demonstrated strong adversarial performance, existing attacks often fail to transfer effectively across architectures, especially from ViTs to Convolutional Neural Networks (CNNs) or hybrid models. In this paper, we introduce \textbf{TESSER} -- a novel adversarial attack framework that enhances transferability via two key strategies: (1) \textit{Feature-Sensitive Gradient Scaling (FSGS)}, which modulates gradients based on token-wise importance derived from intermediate feature activations, and (2) \textit{Spectral Smoothness Regularization (SSR)}, which suppresses high-frequency noise in perturbations using a differentiable Gaussian prior. These components work in tandem to generate perturbations that are both semantically meaningful and spectrally smooth. Extensive experiments on ImageNet across 12 diverse architectures demonstrate that TESSER achieves +10.9\% higher attack succes rate (ASR) on CNNs and +7.2\% on ViTs compared to the state-of-the-art Adaptive Token Tuning (ATT) method. Moreover, TESSER significantly improves robustness against defended models, achieving 53.55\% ASR on adversarially trained CNNs. Qualitative analysis shows strong alignment between TESSER's perturbations and salient visual regions identified via Grad-CAM, while frequency-domain analysis reveals a 12\% reduction in high-frequency energy, confirming the effectiveness of spectral regularization.

http://arxiv.org/abs/2505.16402
AdvReal: Physical Adversarial Patch Generation Framework for Security Evaluation of Object Detection Systems. (99%)
Yuanhao Huang; Yilong Ren; Jinlei Wang; Lujia Huo; Xuesong Bai; Jinchuan Zhang; Haiyan Yu
Autonomous vehicles are typical complex intelligent systems with artificial intelligence at their core. However, perception methods based on deep learning are extremely vulnerable to adversarial samples, resulting in security accidents. How to generate effective adversarial examples in the physical world and evaluate object detection systems is a huge challenge. In this study, we propose a unified joint adversarial training framework for both 2D and 3D domains, which simultaneously optimizes texture maps in 2D image and 3D mesh spaces to better address intra-class diversity and real-world environmental variations. The framework includes a novel realistic enhanced adversarial module, with time-space and relighting mapping pipeline that adjusts illumination consistency between adversarial patches and target garments under varied viewpoints. Building upon this, we develop a realism enhancement mechanism that incorporates non-rigid deformation modeling and texture remapping to ensure alignment with the human body's non-rigid surfaces in 3D scenes. Extensive experiment results in digital and physical environments demonstrate that the adversarial textures generated by our method can effectively mislead the target detection model. Specifically, our method achieves an average attack success rate (ASR) of 70.13% on YOLOv12 in physical scenarios, significantly outperforming existing methods such as T-SEA (21.65%) and AdvTexture (19.70%). Moreover, the proposed method maintains stable ASR across multiple viewpoints and distances, with an average attack success rate exceeding 90% under both frontal and oblique views at a distance of 4 meters. This confirms the method's strong robustness and transferability under multi-angle attacks, varying lighting conditions, and real-world distances. The demo video and code can be obtained at https://github.com/Huangyh98/AdvReal.git.

http://arxiv.org/abs/2207.03400
On the Relationship Between Adversarial Robustness and Decision Region in Deep Neural Networks. (99%)
Seongjin Park; Haedong Jeong; Tair Djanibekov; Giyoung Jeon; Jinseok Seol; Jaesik Choi
In general, Deep Neural Networks (DNNs) are evaluated by the generalization performance measured on unseen data excluded from the training phase. Along with the development of DNNs, the generalization performance converges to the state-of-the-art and it becomes difficult to evaluate DNNs solely based on this metric. The robustness against adversarial attack has been used as an additional metric to evaluate DNNs by measuring their vulnerability. However, few studies have been performed to analyze the adversarial robustness in terms of the geometry in DNNs. In this work, we perform an empirical study to analyze the internal properties of DNNs that affect model robustness under adversarial attacks. In particular, we propose the novel concept of the Populated Region Set (PRS), where training samples are populated more frequently, to represent the internal properties of DNNs in a practical setting. From systematic experiments with the proposed concept, we provide empirical evidence to validate that a low PRS ratio has a strong relationship with the adversarial robustness of DNNs. We also devise PRS regularizer leveraging the characteristics of PRS to improve the adversarial robustness without adversarial training.

http://arxiv.org/abs/2509.09787
ZORRO: Zero-Knowledge Robustness and Privacy for Split Learning (Full Version). (68%)
Nojan Sheybani; Alessandro Pegoraro; Jonathan Knauer; Phillip Rieger; Elissa Mollakuqe; Farinaz Koushanfar; Ahmad-Reza Sadeghi
Split Learning (SL) is a distributed learning approach that enables resource-constrained clients to collaboratively train deep neural networks (DNNs) by offloading most layers to a central server while keeping in- and output layers on the client-side. This setup enables SL to leverage server computation capacities without sharing data, making it highly effective in resource-constrained environments dealing with sensitive data. However, the distributed nature enables malicious clients to manipulate the training process. By sending poisoned intermediate gradients, they can inject backdoors into the shared DNN. Existing defenses are limited by often focusing on server-side protection and introducing additional overhead for the server. A significant challenge for client-side defenses is enforcing malicious clients to correctly execute the defense algorithm.
  We present ZORRO, a private, verifiable, and robust SL defense scheme. Through our novel design and application of interactive zero-knowledge proofs (ZKPs), clients prove their correct execution of a client-located defense algorithm, resulting in proofs of computational integrity attesting to the benign nature of locally trained DNN portions. Leveraging the frequency representation of model partitions enables ZORRO to conduct an in-depth inspection of the locally trained models in an untrusted environment, ensuring that each client forwards a benign checkpoint to its succeeding client. In our extensive evaluation, covering different model architectures as well as various attack strategies and data scenarios, we show ZORRO's effectiveness, as it reduces the attack success rate to less than 6\% while causing even for models storing \numprint{1000000} parameters on the client-side an overhead of less than 10 seconds.

http://arxiv.org/abs/2509.09660
Steering MoE LLMs via Expert (De)Activation. (31%)
Mohsen Fayyaz; Ali Modarressi; Hanieh Deilamsalehy; Franck Dernoncourt; Ryan Rossi; Trung Bui; Hinrich Schütze; Nanyun Peng
Mixture-of-Experts (MoE) in Large Language Models (LLMs) routes each token through a subset of specialized Feed-Forward Networks (FFN), known as experts. We present SteerMoE, a framework for steering MoE models by detecting and controlling behavior-linked experts. Our detection method identifies experts with distinct activation patterns across paired inputs exhibiting contrasting behaviors. By selectively (de)activating such experts during inference, we control behaviors like faithfulness and safety without retraining or modifying weights. Across 11 benchmarks and 6 LLMs, our steering raises safety by up to +20% and faithfulness by +27%. In adversarial attack mode, it drops safety by -41% alone, and -100% when combined with existing jailbreak methods, bypassing all safety guardrails and exposing a new dimension of alignment faking hidden within experts.

http://arxiv.org/abs/2509.09534
ProDiGy: Proximity- and Dissimilarity-Based Byzantine-Robust Federated Learning. (16%)
Sena Ergisi; Luis Maßny; Rawad Bitar
Federated Learning (FL) emerged as a widely studied paradigm for distributed learning. Despite its many advantages, FL remains vulnerable to adversarial attacks, especially under data heterogeneity. We propose a new Byzantine-robust FL algorithm called ProDiGy. The key novelty lies in evaluating the client gradients using a joint dual scoring system based on the gradients' proximity and dissimilarity. We demonstrate through extensive numerical experiments that ProDiGy outperforms existing defenses in various scenarios. In particular, when the clients' data do not follow an IID distribution, while other defense mechanisms fail, ProDiGy maintains strong defense capabilities and model accuracy. These findings highlight the effectiveness of a dual perspective approach that promotes natural similarity among honest clients while detecting suspicious uniformity as a potential indicator of an attack.

http://arxiv.org/abs/2503.20884
Byzantine-Robust Federated Learning Using Generative Adversarial Networks. (13%)
Usama Zafar; André M. H. Teixeira; Salman Toor
Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, but its robustness is threatened by Byzantine behaviors such as data and model poisoning. Existing defenses face fundamental limitations: robust aggregation rules incur error lower bounds that grow with client heterogeneity, while detection-based methods often rely on heuristics (e.g., a fixed number of malicious clients) or require trusted external datasets for validation. We present a defense framework that addresses these challenges by leveraging a conditional generative adversarial network (cGAN) at the server to synthesize representative data for validating client updates. This approach eliminates reliance on external datasets, adapts to diverse attack strategies, and integrates seamlessly into standard FL workflows. Extensive experiments on benchmark datasets demonstrate that our framework accurately distinguishes malicious from benign clients while maintaining overall model accuracy. Beyond Byzantine robustness, we also examine the representativeness of synthesized data, computational costs of cGAN training, and the transparency and scalability of our approach.

http://arxiv.org/abs/2503.16183
Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations. (12%)
Xiao Wang; Hendrik Borras; Bernhard Klein; Holger Fröning
The disparity between the computational demands of deep learning and the capabilities of compute hardware is expanding drastically. Although deep learning achieves remarkable performance in countless tasks, its escalating requirements for computational power and energy consumption surpass the sustainable limits of even specialized neural processing units, including the Apple Neural Engine and NVIDIA TensorCores. This challenge is intensified by the slowdown in CMOS scaling.
  Analog computing presents a promising alternative, offering substantial improvements in energy efficiency by directly manipulating physical quantities such as current, voltage, charge, or photons. However, it is inherently vulnerable to manufacturing variations, nonlinearities, and noise, leading to degraded prediction accuracy. One of the most effective techniques for enhancing robustness, Noisy Training, introduces noise during the training phase to reinforce the model against disturbances encountered during inference. Although highly effective, its performance degrades in real-world environments where noise characteristics fluctuate due to external factors such as temperature variations and temporal drift.
  This study underscores the necessity of Noisy Training while revealing its fundamental limitations in the presence of dynamic noise. To address these challenges, we propose Variance-Aware Noisy Training, a novel approach that mitigates performance degradation by incorporating noise schedules which emulate the evolving noise conditions encountered during inference. Our method substantially improves model robustness, without training overhead. We demonstrate a significant increase in robustness, from 79.3\% with conventional Noisy Training to 97.6\% with Variance-Aware Noisy Training on CIFAR-10 and from 32.4\% to 99.7\% on Tiny ImageNet.

http://arxiv.org/abs/2504.21846
Combating Falsification of Speech Videos with Live Optical Signatures (Extended Version). (5%)
Hadleigh Schwartz; Xiaofeng Yan; Charles J. Carver; Xia Zhou
High-profile speech videos are prime targets for falsification, owing to their accessibility and influence. This work proposes VeriLight, a low-overhead and unobtrusive system for protecting speech videos from visual manipulations of speaker identity and lip and facial motion. Unlike the predominant purely digital falsification detection methods, VeriLight creates dynamic physical signatures at the event site and embeds them into all video recordings via imperceptible modulated light. These physical signatures encode semantically-meaningful features unique to the speech event, including the speaker's identity and facial motion, and are cryptographically-secured to prevent spoofing. The signatures can be extracted from any video downstream and validated against the portrayed speech content to check its integrity. Key elements of VeriLight include (1) a framework for generating extremely compact (i.e., 150-bit), pose-invariant speech video features, based on locality-sensitive hashing; and (2) an optical modulation scheme that embeds $>$200 bps into video while remaining imperceptible both in video and live. Experiments on extensive video datasets show VeriLight achieves AUCs $\geq$ 0.99 and a true positive rate of 100% in detecting falsified videos. Further, VeriLight is highly robust across recording conditions, video post-processing techniques, and white-box adversarial attacks on its feature extraction methods. A demonstration of VeriLight is available at https://mobilex.cs.columbia.edu/verilight.

http://arxiv.org/abs/2504.07875
QubitHammer: Remotely Inducing Qubit State Change on Superconducting Quantum Computers. (1%)
Yizhuo Tan; Navnil Choudhury; Kanad Basu; Jakub Szefer
To address the rapidly growing demand for cloud-based quantum computing, various researchers are proposing shifting from the existing single-tenant model to a multi-tenant model that expands resource utilization and improves accessibility. However, while multi-tenancy enables multiple users to access the same quantum computer, it introduces potential for security and reliability vulnerabilities. It therefore becomes important to investigate these vulnerabilities, especially considering realistic attackers who operate without elevated privileges relative to ordinary users. To address this research need, this paper presents and evaluates QubitHammer, the first attack to demonstrate that an adversary can remotely induce unauthorized changes to a victim's quantum circuit's qubit's state within a multi-tenant model by using custom qubit control pulses that are generated within constraints of the public interfaces and without elevated privileges. Through extensive evaluation on real-world superconducting devices from IBM and Rigetti, this work demonstrates that QubitHammer allows an adversary to significantly change the output distribution of a victim quantum circuit. In the experimentation, variational distance is used to evaluate the magnitude of the changes, and variational distance as high as 0.938 is observed. Cross-platform analysis of QubitHammer on a number of quantum computing devices exposes a fundamental susceptibility in superconducting hardware. Further, QubitHammer was also found to evade all currently proposed defenses aimed at ensuring reliable execution in multi-tenant superconducting quantum systems.

http://arxiv.org/abs/2509.07673
Nearest Neighbor Projection Removal Adversarial Training. (98%)
Himanshu Singh; A. V. Subramanyam; Shivank Rajput; Mohan Kankanhalli
Deep neural networks have exhibited impressive performance in image classification tasks but remain vulnerable to adversarial examples. Standard adversarial training enhances robustness but typically fails to explicitly address inter-class feature overlap, a significant contributor to adversarial susceptibility. In this work, we introduce a novel adversarial training framework that actively mitigates inter-class proximity by projecting out inter-class dependencies from adversarial and clean samples in the feature space. Specifically, our approach first identifies the nearest inter-class neighbors for each adversarial sample and subsequently removes projections onto these neighbors to enforce stronger feature separability. Theoretically, we demonstrate that our proposed logits correction reduces the Lipschitz constant of neural networks, thereby lowering the Rademacher complexity, which directly contributes to improved generalization and robustness. Extensive experiments across standard benchmarks including CIFAR-10, CIFAR-100, and SVHN show that our method demonstrates strong performance that is competitive with leading adversarial training techniques, highlighting significant achievements in both robust and clean accuracy. Our findings reveal the importance of addressing inter-class feature proximity explicitly to bolster adversarial robustness in DNNs.

http://arxiv.org/abs/2509.08463
Adversarial Attacks Against Automated Fact-Checking: A Survey. (86%)
Fanzhen Liu; Alsharif Abuadbba; Kristen Moore; Surya Nepal; Cecile Paris; Jia Wu; Jian Yang; Quan Z. Sheng
In an era where misinformation spreads freely, fact-checking (FC) plays a crucial role in verifying claims and promoting reliable information. While automated fact-checking (AFC) has advanced significantly, existing systems remain vulnerable to adversarial attacks that manipulate or generate claims, evidence, or claim-evidence pairs. These attacks can distort the truth, mislead decision-makers, and ultimately undermine the reliability of FC models. Despite growing research interest in adversarial attacks against AFC systems, a comprehensive, holistic overview of key challenges remains lacking. These challenges include understanding attack strategies, assessing the resilience of current models, and identifying ways to enhance robustness. This survey provides the first in-depth review of adversarial attacks targeting FC, categorizing existing attack methodologies and evaluating their impact on AFC systems. Additionally, we examine recent advancements in adversary-aware defenses and highlight open research questions that require further exploration. Our findings underscore the urgent need for resilient FC frameworks capable of withstanding adversarial manipulations in pursuit of preserving high verification accuracy.

http://arxiv.org/abs/2407.09447
ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Prompts. (13%)
Amelia F. Hardy; Houjun Liu; Allie Griffith; Bernard Lange; Duncan Eddy; Mykel J. Kochenderfer
Existing LLM red-teaming approaches prioritize high attack success rate, often resulting in high-perplexity prompts. This focus overlooks low-perplexity attacks that are more difficult to filter, more likely to arise during benign usage, and more impactful as negative downstream training examples. In response, we introduce ASTPrompter, a single-step optimization method that uses contrastive preference learning to train an attacker to maintain low perplexity while achieving a high attack success rate (ASR). ASTPrompter achieves an attack success rate 5.1 times higher on Llama-8.1B while using inputs that are 2.1 times more likely to occur according to the frozen LLM. Furthermore, our attack transfers to Mistral-7B, Qwen-7B, and TinyLlama in both black- and white-box settings. Lastly, by tuning a single hyperparameter in our method, we discover successful attack prefixes along an efficient frontier between ASR and perplexity, highlighting perplexity as a previously under-considered factor in red-teaming.

http://arxiv.org/abs/2503.14111
Towards properties of adversarial image perturbations. (5%)
Egor Kuznetsov; Kirill Aistov; Maxim Koroteev
Using stochastic gradient approach we study the properties of adversarial perturbations resulting in noticeable growth of VMAF image quality metric. The structure of the perturbations is investigated depending on the acceptable PSNR values and based on the Fourier power spectrum computations for the perturbations. It is demonstrated that moderate variation of image brightness ($\sim 10$ pixel units in a restricted region of an image can result in VMAF growth by $\sim 60\%$). Unlike some other methods demonstrating similar VMAF growth, the subjective quality of an image remains almost unchanged. It is also shown that the adversarial perturbations may demonstrate approximately linear dependence of perturbation amplitudes on the image brightness. The perturbations are studied based on the direct VMAF optimization in PyTorch. The significant discrepancies between the metric values and subjective judgements are also demonstrated when image restoration from noise is carried out using the same direct VMAF optimization.

http://arxiv.org/abs/2509.08436
Beyond Distribution Shifts: Adaptive Hyperspectral Image Classification at Test Time. (1%)
Xia Yue; Anfeng Liu; Ning Chen; Chenjia Huang; Hui Liu; Zhou Huang; Leyuan Fang
Hyperspectral image (HSI) classification models are highly sensitive to distribution shifts caused by various real-world degradations such as noise, blur, compression, and atmospheric effects. To address this challenge, we propose HyperTTA, a unified framework designed to enhance model robustness under diverse degradation conditions. Specifically, we first construct a multi-degradation hyperspectral dataset that systematically simulates nine representative types of degradations, providing a comprehensive benchmark for robust classification evaluation. Based on this, we design a spectral-spatial transformer classifier (SSTC) enhanced with a multi-level receptive field mechanism and label smoothing regularization to jointly capture multi-scale spatial context and improve generalization. Furthermore, HyperTTA incorporates a lightweight test-time adaptation (TTA) strategy, the confidence-aware entropy-minimized LayerNorm adapter (CELA), which updates only the affine parameters of LayerNorm layers by minimizing prediction entropy on high-confidence unlabeled target samples. This confidence-aware adaptation prevents unreliable updates from noisy predictions, enabling robust and dynamic adaptation without access to source data or target annotations. Extensive experiments on two benchmark datasets demonstrate that HyperTTA outperforms existing baselines across a wide range of degradation scenarios, validating the effectiveness of both its classification backbone and the proposed TTA scheme. Code will be made available publicly.

http://arxiv.org/abs/2509.10563
Enhancing IoMT Security with Explainable Machine Learning: A Case Study on the CICIOMT2024 Dataset. (1%)
Mohammed Yacoubi; Omar Moussaoui; C. Drocourt
Explainable Artificial Intelligence (XAI) enhances the transparency and interpretability of AI models, addressing their inherent opacity. In cybersecurity, particularly within the Internet of Medical Things (IoMT), the black-box nature of AI-driven threat detection poses a significant challenge. Cybersecurity professionals must not only detect attacks but also understand the reasoning behind AI decisions to ensure trust and accountability. The rapid increase in cyberattacks targeting connected medical devices threatens patient safety and data privacy, necessitating advanced AI-driven solutions. This study compares two ensemble learning techniques, bagging and boosting, for cyber-attack classification in IoMT environments. We selected Random Forest for bagging and CatBoost for boosting. Random Forest helps reduce variance, while CatBoost improves bias by combining weak classifiers into a strong ensemble model, making them effective for detecting sophisticated attacks. However, their complexity often reduces transparency, making it difficult for cybersecurity professionals to interpret and trust their decisions. To address this issue, we apply XAI models to generate local and global explanations, providing insights into AI decision-making. Using techniques like SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations), we highlight feature importance to help stakeholders understand the key factors driving cyber threat detection.

http://arxiv.org/abs/2509.07677
Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems. (99%)
Kamel Kamel; Hridoy Sankar Dutta; Keshav Sood; Sunil Aryal
Voice Authentication Systems (VAS) use unique vocal characteristics for verification. They are increasingly integrated into high-security sectors such as banking and healthcare. Despite their improvements using deep learning, they face severe vulnerabilities from sophisticated threats like deepfakes and adversarial attacks. The emergence of realistic voice cloning complicates detection, as systems struggle to distinguish authentic from synthetic audio. While anti-spoofing countermeasures (CMs) exist to mitigate these risks, many rely on static detection models that can be bypassed by novel adversarial methods, leaving a critical security gap. To demonstrate this vulnerability, we propose the Spectral Masking and Interpolation Attack (SMIA), a novel method that strategically manipulates inaudible frequency regions of AI-generated audio. By altering the voice in imperceptible zones to the human ear, SMIA creates adversarial samples that sound authentic while deceiving CMs. We conducted a comprehensive evaluation of our attack against state-of-the-art (SOTA) models across multiple tasks, under simulated real-world conditions. SMIA achieved a strong attack success rate (ASR) of at least 82% against combined VAS/CM systems, at least 97.5% against standalone speaker verification systems, and 100% against countermeasures. These findings conclusively demonstrate that current security postures are insufficient against adaptive adversarial attacks. This work highlights the urgent need for a paradigm shift toward next-generation defenses that employ dynamic, context-aware frameworks capable of evolving with the threat landscape.

http://arxiv.org/abs/2509.07495
Generating Transferrable Adversarial Examples via Local Mixing and Logits Optimization for Remote Sensing Object Recognition. (99%)
Chun Liu; Hailong Wang; Bingqian Zhu; Panpan Ding; Zheng Zheng; Tao Xu; Zhigang Han; Jiayao Wang
Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, posing significant security threats to their deployment in remote sensing applications. Research on adversarial attacks not only reveals model vulnerabilities but also provides critical insights for enhancing robustness. Although current mixing-based strategies have been proposed to increase the transferability of adversarial examples, they either perform global blending or directly exchange a region in the images, which may destroy global semantic features and mislead the optimization of adversarial examples. Furthermore, their reliance on cross-entropy loss for perturbation optimization leads to gradient diminishing during iterative updates, compromising adversarial example quality. To address these limitations, we focus on non-targeted attacks and propose a novel framework via local mixing and logits optimization. First, we present a local mixing strategy to generate diverse yet semantically consistent inputs. Different from MixUp, which globally blends two images, and MixCut, which stitches images together, our method merely blends local regions to preserve global semantic information. Second, we adapt the logit loss from targeted attacks to non-targeted scenarios, mitigating the gradient vanishing problem of cross-entropy loss. Third, a perturbation smoothing loss is applied to suppress high-frequency noise and enhance transferability. Extensive experiments on FGSCR-42 and MTARSI datasets demonstrate superior performance over 12 state-of-the-art methods across 6 surrogate models. Notably, with ResNet as the surrogate on MTARSI, our method achieves a 17.28% average improvement in black-box attack success rate.

http://arxiv.org/abs/2509.08091
SAGE: Sample-Aware Guarding Engine for Robust Intrusion Detection Against Adversarial Attacks. (99%)
Jing Chen; Onat Gungor; Zhengli Shang; Tajana Rosing
The rapid proliferation of the Internet of Things (IoT) continues to expose critical security vulnerabilities, necessitating the development of efficient and robust intrusion detection systems (IDS). Machine learning-based intrusion detection systems (ML-IDS) have significantly improved threat detection capabilities; however, they remain highly susceptible to adversarial attacks. While numerous defense mechanisms have been proposed to enhance ML-IDS resilience, a systematic approach for selecting the most effective defense against a specific adversarial attack remains absent. To address this challenge, we previously proposed DYNAMITE, a dynamic defense selection approach that identifies the most suitable defense against adversarial attacks through an ML-driven selection mechanism. Building on this foundation, we propose SAGE (Sample-Aware Guarding Engine), a substantially improved defense algorithm that integrates active learning with targeted data reduction. It employs an active learning mechanism to selectively identify the most informative input samples and their corresponding optimal defense labels, which are then used to train a second-level learner responsible for selecting the most effective defense. This targeted sampling improves computational efficiency, exposes the model to diverse adversarial strategies during training, and enhances robustness, stability, and generalizability. As a result, SAGE demonstrates strong predictive performance across multiple intrusion detection datasets, achieving an average F1-score improvement of 201% over the state-of-the-art defenses. Notably, SAGE narrows the performance gap to the Oracle to just 3.8%, while reducing computational overhead by up to 29x.

http://arxiv.org/abs/2509.07617
Transferable Direct Prompt Injection via Activation-Guided MCMC Sampling. (75%)
Minghui Li; Hao Zhang; Yechao Zhang; Wei Wan; Shengshan Hu; pei Xiaobing; Jing Wang
Direct Prompt Injection (DPI) attacks pose a critical security threat to Large Language Models (LLMs) due to their low barrier of execution and high potential damage. To address the impracticality of existing white-box/gray-box methods and the poor transferability of black-box methods, we propose an activations-guided prompt injection attack framework. We first construct an Energy-based Model (EBM) using activations from a surrogate model to evaluate the quality of adversarial prompts. Guided by the trained EBM, we employ the token-level Markov Chain Monte Carlo (MCMC) sampling to adaptively optimize adversarial prompts, thereby enabling gradient-free black-box attacks. Experimental results demonstrate our superior cross-model transferability, achieving 49.6% attack success rate (ASR) across five mainstream LLMs and 34.6% improvement over human-crafted prompts, and maintaining 36.6% ASR on unseen task scenarios. Interpretability analysis reveals a correlation between activations and attack effectiveness, highlighting the critical role of semantic patterns in transferable vulnerability exploitation.

http://arxiv.org/abs/2509.08089
Hammer and Anvil: A Principled Defense Against Backdoors in Federated Learning. (47%)
Lucas Fenaux; Zheng Wang; Jacob Yan; Nathan Chung; Florian Kerschbaum
Federated Learning is a distributed learning technique in which multiple clients cooperate to train a machine learning model. Distributed settings facilitate backdoor attacks by malicious clients, who can embed malicious behaviors into the model during their participation in the training process. These malicious behaviors are activated during inference by a specific trigger. No defense against backdoor attacks has stood the test of time, especially against adaptive attackers, a powerful but not fully explored category of attackers. In this work, we first devise a new adaptive adversary that surpasses existing adversaries in capabilities, yielding attacks that only require one or two malicious clients out of 20 to break existing state-of-the-art defenses. Then, we present Hammer and Anvil, a principled defense approach that combines two defenses orthogonal in their underlying principle to produce a combined defense that, given the right set of parameters, must succeed against any attack. We show that our best combined defense, Krum+, is successful against our new adaptive adversary and state-of-the-art attacks.

http://arxiv.org/abs/2509.07504
Backdoor Attacks and Defenses in Computer Vision Domain: A Survey. (4%)
Bilal Hussain Abbasi; Yanjun Zhang; Leo Zhang; Shang Gao
Backdoor (trojan) attacks embed hidden, controllable behaviors into machine-learning models so that models behave normally on benign inputs but produce attacker-chosen outputs when a trigger is present. This survey reviews the rapidly growing literature on backdoor attacks and defenses in the computer-vision domain. We introduce a multi-dimensional taxonomy that organizes attacks and defenses by injection stage (dataset poisoning, model/parameter modification, inference-time injection), trigger type (patch, blended/frequency, semantic, transformation), labeling strategy (dirty-label vs. clean-label / feature-collision), representation stage (instance-specific, manifold/class-level, neuron/parameter hijacking, distributed encodings), and target task (classification, detection, segmentation, video, multimodal). For each axis we summarize representative methods, highlight evaluation practices, and discuss where defenses succeed or fail. For example, many classical sanitization and reverse-engineering tools are effective against reusable patch attacks but struggle with input-aware, sample-specific, or parameter-space backdoors and with transfer via compromised pre-trained encoders or hardware bit-flips. We synthesize trends, identify persistent gaps (supply-chain and hardware threats, certifiable defenses, cross-task benchmarks), and propose practical guidelines for threat-aware evaluation and layered defenses. This survey aims to orient researchers and practitioners to the current threat landscape and pressing research directions in secure computer vision.

http://arxiv.org/abs/2509.07941
ImportSnare: Directed "Code Manual" Hijacking in Retrieval-Augmented Code Generation. (1%)
Kai Ye; Liangcai Su; Chenxiong Qian
Code generation has emerged as a pivotal capability of Large Language Models(LLMs), revolutionizing development efficiency for programmers of all skill levels. However, the complexity of data structures and algorithmic logic often results in functional deficiencies and security vulnerabilities in generated code, reducing it to a prototype requiring extensive manual debugging. While Retrieval-Augmented Generation (RAG) can enhance correctness and security by leveraging external code manuals, it simultaneously introduces new attack surfaces.
  In this paper, we pioneer the exploration of attack surfaces in Retrieval-Augmented Code Generation (RACG), focusing on malicious dependency hijacking. We demonstrate how poisoned documentation containing hidden malicious dependencies (e.g., matplotlib_safe) can subvert RACG, exploiting dual trust chains: LLM reliance on RAG and developers' blind trust in LLM suggestions. To construct poisoned documents, we propose ImportSnare, a novel attack framework employing two synergistic strategies: 1)Position-aware beam search optimizes hidden ranking sequences to elevate poisoned documents in retrieval results, and 2)Multilingual inductive suggestions generate jailbreaking sequences to manipulate LLMs into recommending malicious dependencies. Through extensive experiments across Python, Rust, and JavaScript, ImportSnare achieves significant attack success rates (over 50% for popular libraries such as matplotlib and seaborn) in general, and is also able to succeed even when the poisoning ratio is as low as 0.01%, targeting both custom and real-world malicious packages. Our findings reveal critical supply chain risks in LLM-powered development, highlighting inadequate security alignment for code generation tasks. To support future research, we will release the multilingual benchmark suite and datasets. The project homepage is https://importsnare.github.io.

http://arxiv.org/abs/2508.04669
Cybersecurity of Quantum Key Distribution Implementations. (1%)
Ittay Alfassi; Ran Gelles; Rotem Liss; Tal Mor
Practical implementations of Quantum Key Distribution (QKD) often deviate from the theoretical protocols, exposing the implementations to various attacks even when the underlying (ideal) protocol is proven secure. We present new analysis tools and methodologies for quantum cybersecurity, adapting the concepts of vulnerabilities, attack surfaces, and exploits from classical cybersecurity to QKD implementation attacks. We also present three additional concepts, derived from the connection between classical and quantum cybersecurity: "Quantum Fuzzing", which is the first tool for black-box vulnerability research on QKD implementations; "Reversed-Space Attacks", which are a generic exploit method using the attack surface of imperfect receivers; and concrete quantum-mechanical definitions of "Quantum Side-Channel Attacks" and "Quantum State-Channel Attacks", meaningfully distinguishing them from each other and from other attacks. Using our tools, we analyze multiple existing QKD attacks and show that the "Bright Illumination" attack could have been found even with minimal knowledge of the device implementation. This work begins to bridge the gap between current analysis methods for experimental attacks on QKD implementations and the decades-long research in the field of classical cybersecurity, improving the practical security of QKD products and enhancing their usefulness in real-world systems.

http://arxiv.org/abs/2509.06835
Evaluating the Impact of Adversarial Attacks on Traffic Sign Classification using the LISA Dataset. (99%)
Nabeyou Tadessa; Balaji Iyangar; Mashrur Chowdhury
Adversarial attacks pose significant threats to machine learning models by introducing carefully crafted perturbations that cause misclassification. While prior work has primarily focused on MNIST and similar datasets, this paper investigates the vulnerability of traffic sign classifiers using the LISA Traffic Sign dataset. We train a convolutional neural network to classify 47 different traffic signs and evaluate its robustness against Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. Our results show a sharp decline in classification accuracy as the perturbation magnitude increases, highlighting the models susceptibility to adversarial examples. This study lays the groundwork for future exploration into defense mechanisms tailored for real-world traffic sign recognition systems.

http://arxiv.org/abs/2509.06459
IGAff: Benchmarking Adversarial Iterative and Genetic Affine Algorithms on Deep Neural Networks. (98%)
Sebastian-Vasile Echim; Andrei-Alexandru Preda; Dumitru-Clementin Cercel; Florin Pop
Deep neural networks currently dominate many fields of the artificial intelligence landscape, achieving state-of-the-art results on numerous tasks while remaining hard to understand and exhibiting surprising weaknesses. An active area of research focuses on adversarial attacks, which aim to generate inputs that uncover these weaknesses. However, this proves challenging, especially in the black-box scenario where model details are inaccessible. This paper explores in detail the impact of such adversarial algorithms on ResNet-18, DenseNet-121, Swin Transformer V2, and Vision Transformer network architectures. Leveraging the Tiny ImageNet, Caltech-256, and Food-101 datasets, we benchmark two novel black-box iterative adversarial algorithms based on affine transformations and genetic algorithms: 1) Affine Transformation Attack (ATA), an iterative algorithm maximizing our attack score function using random affine transformations, and 2) Affine Genetic Attack (AGA), a genetic algorithm that involves random noise and affine transformations. We evaluate the performance of the models in the algorithm parameter variation, data augmentation, and global and targeted attack configurations. We also compare our algorithms with two black-box adversarial algorithms, Pixle and Square Attack. Our experiments yield better results on the image classification task than similar methods in the literature, achieving an accuracy improvement of up to 8.82%. We provide noteworthy insights into successful adversarial defenses and attacks at both global and targeted levels, and demonstrate adversarial robustness through algorithm parameter variation.

http://arxiv.org/abs/2508.09994
Whisper Smarter, not Harder: Adversarial Attack on Partial Suppression. (80%)
Zheng Jie Wong; Bingquan Shen
Currently, Automatic Speech Recognition (ASR) models are deployed in an extensive range of applications. However, recent studies have demonstrated the possibility of adversarial attack on these models which could potentially suppress or disrupt model output. We investigate and verify the robustness of these attacks and explore if it is possible to increase their imperceptibility. We additionally find that by relaxing the optimisation objective from complete suppression to partial suppression, we can further decrease the imperceptibility of the attack. We also explore possible defences against these attacks and show a low-pass filter defence could potentially serve as an effective defence.

http://arxiv.org/abs/2509.07132
Adversarial Attacks on Audio Deepfake Detection: A Benchmark and Comparative Study. (75%)
Kutub Uddin; Muhammad Umar Farooq; Awais Khan; Khalid Mahmood Malik
The widespread use of generative AI has shown remarkable success in producing highly realistic deepfakes, posing a serious threat to various voice biometric applications, including speaker verification, voice biometrics, audio conferencing, and criminal investigations. To counteract this, several state-of-the-art (SoTA) audio deepfake detection (ADD) methods have been proposed to identify generative AI signatures to distinguish between real and deepfake audio. However, the effectiveness of these methods is severely undermined by anti-forensic (AF) attacks that conceal generative signatures. These AF attacks span a wide range of techniques, including statistical modifications (e.g., pitch shifting, filtering, noise addition, and quantization) and optimization-based attacks (e.g., FGSM, PGD, C \& W, and DeepFool). In this paper, we investigate the SoTA ADD methods and provide a comparative analysis to highlight their effectiveness in exposing deepfake signatures, as well as their vulnerabilities under adversarial conditions. We conducted an extensive evaluation of ADD methods on five deepfake benchmark datasets using two categories: raw and spectrogram-based approaches. This comparative analysis enables a deeper understanding of the strengths and limitations of SoTA ADD methods against diverse AF attacks. It does not only highlight vulnerabilities of ADD methods, but also informs the design of more robust and generalized detectors for real-world voice biometrics. It will further guide future research in developing adaptive defense strategies that can effectively counter evolving AF techniques.

http://arxiv.org/abs/2509.06338
Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift. (69%)
Shuai Yuan; Zhibo Zhang; Yuxi Li; Guangdong Bai; Wang Kailong
The widespread distribution of Large Language Models (LLMs) through public platforms like Hugging Face introduces significant security challenges. While these platforms perform basic security scans, they often fail to detect subtle manipulations within the embedding layer. This work identifies a novel class of deployment phase attacks that exploit this vulnerability by injecting imperceptible perturbations directly into the embedding layer outputs without modifying model weights or input text. These perturbations, though statistically benign, systematically bypass safety alignment mechanisms and induce harmful behaviors during inference. We propose Search based Embedding Poisoning(SEP), a practical, model agnostic framework that introduces carefully optimized perturbations into embeddings associated with high risk tokens. SEP leverages a predictable linear transition in model responses, from refusal to harmful output to semantic deviation to identify a narrow perturbation window that evades alignment safeguards. Evaluated across six aligned LLMs, SEP achieves an average attack success rate of 96.43% while preserving benign task performance and evading conventional detection mechanisms. Our findings reveal a critical oversight in deployment security and emphasize the urgent need for embedding level integrity checks in future LLM defense strategies.

http://arxiv.org/abs/2509.06554
Robustness and accuracy of mean opinion scores with hard and soft outlier detection. (2%)
Dietmar Saupe; Tim Bleile
In subjective assessment of image and video quality, observers rate or compare selected stimuli. Before calculating the mean opinion scores (MOS) for these stimuli from the ratings, it is recommended to identify and deal with outliers that may have given unreliable ratings. Several methods are available for this purpose, some of which have been standardized. These methods are typically based on statistics and sometimes tested by introducing synthetic ratings from artificial outliers, such as random clickers. However, a reliable and comprehensive approach is lacking for comparative performance analysis of outlier detection methods. To fill this gap, this work proposes and applies an empirical worst-case analysis as a general solution. Our method involves evolutionary optimization of an adversarial black-box attack on outlier detection algorithms, where the adversary maximizes the distortion of scale values with respect to ground truth. We apply our analysis to several hard and soft outlier detection methods for absolute category ratings and show their differing performance in this stress test. In addition, we propose two new outlier detection methods with low complexity and excellent worst-case performance. Software for adversarial attacks and data analysis is available.

http://arxiv.org/abs/2509.06572
Mind Your Server: A Systematic Study of Parasitic Toolchain Attacks on the MCP Ecosystem. (1%)
Shuli Zhao; Qinsheng Hou; Zihan Zhan; Yanhao Wang; Yuchong Xie; Yu Guo; Libo Chen; Shenghong Li; Zhi Xue
Large language models (LLMs) are increasingly integrated with external systems through the Model Context Protocol (MCP), which standardizes tool invocation and has rapidly become a backbone for LLM-powered applications. While this paradigm enhances functionality, it also introduces a fundamental security shift: LLMs transition from passive information processors to autonomous orchestrators of task-oriented toolchains, expanding the attack surface, elevating adversarial goals from manipulating single outputs to hijacking entire execution flows. In this paper, we reveal a new class of attacks, Parasitic Toolchain Attacks, instantiated as MCP Unintended Privacy Disclosure (MCP-UPD). These attacks require no direct victim interaction; instead, adversaries embed malicious instructions into external data sources that LLMs access during legitimate tasks. The malicious logic infiltrates the toolchain and unfolds in three phases: Parasitic Ingestion, Privacy Collection, and Privacy Disclosure, culminating in stealthy exfiltration of private data. Our root cause analysis reveals that MCP lacks both context-tool isolation and least-privilege enforcement, enabling adversarial instructions to propagate unchecked into sensitive tool invocations. To assess the severity, we design MCP-SEC and conduct the first large-scale security census of the MCP ecosystem, analyzing 12,230 tools across 1,360 servers. Our findings show that the MCP ecosystem is rife with exploitable gadgets and diverse attack methods, underscoring systemic risks in MCP platforms and the urgent need for defense mechanisms in LLM-integrated environments.

http://arxiv.org/abs/2509.06896
Not All Samples Are Equal: Quantifying Instance-level Difficulty in Targeted Data Poisoning. (1%)
William Xu; Yiwei Lu; Yihan Wang; Matthew Y. R. Yang; Zuoqiu Liu; Gautam Kamath; Yaoliang Yu
Targeted data poisoning attacks pose an increasingly serious threat due to their ease of deployment and high success rates. These attacks aim to manipulate the prediction for a single test sample in classification models. Unlike indiscriminate attacks that aim to decrease overall test performance, targeted attacks present a unique threat to individual test instances. This threat model raises a fundamental question: what factors make certain test samples more susceptible to successful poisoning than others? We investigate how attack difficulty varies across different test instances and identify key characteristics that influence vulnerability. This paper introduces three predictive criteria for targeted data poisoning difficulty: ergodic prediction accuracy (analyzed through clean training dynamics), poison distance, and poison budget. Our experimental results demonstrate that these metrics effectively predict the varying difficulty of real-world targeted poisoning attacks across diverse scenarios, offering practitioners valuable insights for vulnerability assessment and understanding data poisoning attacks.

http://arxiv.org/abs/2509.06071
Asymmetry Vulnerability and Physical Attacks on Online Map Construction for Autonomous Driving. (75%)
Yang Lou; Haibo Hu; Qun Song; Qian Xu; Yi Zhu; Rui Tan; Wei-Bin Lee; Jianping Wang
High-definition maps provide precise environmental information essential for prediction and planning in autonomous driving systems. Due to the high cost of labeling and maintenance, recent research has turned to online HD map construction using onboard sensor data, offering wider coverage and more timely updates for autonomous vehicles. However, the robustness of online map construction under adversarial conditions remains underexplored. In this paper, we present a systematic vulnerability analysis of online map construction models, which reveals that these models exhibit an inherent bias toward predicting symmetric road structures. In asymmetric scenes like forks or merges, this bias often causes the model to mistakenly predict a straight boundary that mirrors the opposite side. We demonstrate that this vulnerability persists in the real-world and can be reliably triggered by obstruction or targeted interference. Leveraging this vulnerability, we propose a novel two-stage attack framework capable of manipulating online constructed maps. First, our method identifies vulnerable asymmetric scenes along the victim AV's potential route. Then, we optimize the location and pattern of camera-blinding attacks and adversarial patch attacks. Evaluations on a public AD dataset demonstrate that our attacks can degrade mapping accuracy by up to 9.9%, render up to 44% of targeted routes unreachable, and increase unsafe planned trajectory rates, colliding with real-world road boundaries, by up to 27%. These attacks are also validated on a real-world testbed vehicle. We further analyze root causes of the symmetry bias, attributing them to training data imbalance, model architecture, and map element representation. To the best of our knowledge, this study presents the first vulnerability assessment of online map construction models and introduces the first digital and physical attack against them.

http://arxiv.org/abs/2509.10543
Robust DDoS-Attack Classification with 3D CNNs Against Adversarial Methods. (22%)
Landon Bragg; Nathan Dorsey; Josh Prior; John Ajit; Ben Kim; Nate Willis; Pablo Rivas
Distributed Denial-of-Service (DDoS) attacks remain a serious threat to online infrastructure, often bypassing detection by altering traffic in subtle ways. We present a method using hive-plot sequences of network data and a 3D convolutional neural network (3D CNN) to classify DDoS traffic with high accuracy. Our system relies on three main ideas: (1) using spatio-temporal hive-plot encodings to set a pattern-recognition baseline, (2) applying adversarial training with FGSM and PGD alongside spatial noise and image shifts, and (3) analyzing frame-wise predictions to find early signals. On a benchmark dataset, our method lifts adversarial accuracy from 50-55% to over 93% while maintaining clean-sample performance. Frames 3-4 offer strong predictive signals, showing early-stage classification is possible.

http://arxiv.org/abs/2509.05921
Dataset Ownership in the Era of Large Language Models. (15%)
Kun Li; Cheng Wang; Minghui Xu; Yue Zhang; Xiuzhen Cheng
As datasets become critical assets in modern machine learning systems, ensuring robust copyright protection has emerged as an urgent challenge. Traditional legal mechanisms often fail to address the technical complexities of digital data replication and unauthorized use, particularly in opaque or decentralized environments. This survey provides a comprehensive review of technical approaches for dataset copyright protection, systematically categorizing them into three main classes: non-intrusive methods, which detect unauthorized use without modifying data; minimally-intrusive methods, which embed lightweight, reversible changes to enable ownership verification; and maximally-intrusive methods, which apply aggressive data alterations, such as reversible adversarial examples, to enforce usage restrictions. We synthesize key techniques, analyze their strengths and limitations, and highlight open research challenges. This work offers an organized perspective on the current landscape and suggests future directions for developing unified, scalable, and ethically sound solutions to protect datasets in increasingly complex machine learning ecosystems.

http://arxiv.org/abs/2509.07017
From Eigenmodes to Proofs: Integrating Graph Spectral Operators with Symbolic Interpretable Reasoning. (9%)
Andrew Kiruluta; Priscilla Burity
We introduce Spectral NSR, a fully spectral neuro-symbolic reasoning framework that embeds logical rules as spectral templates and performs inference directly in the graph spectral domain. By leveraging graph signal processing (GSP) and frequency-selective filters grounded in the Laplacian eigenstructure of knowledge graphs, the architecture unifies the interpretability of symbolic reasoning with the scalability and adaptability of spectral learning. Beyond the core formulation, we incorporate a comprehensive set of extensions, including dynamic graph and basis learning, rational and diffusion filters for sharper spectral selectivity, mixture-of-spectral-experts for modular specialization, proof-guided training with spectral curricula, and uncertainty quantification for calibrated confidence. Additional enhancements such as large language model coupling, co-spectral transfer alignment, adversarial robustness, efficient GPU kernels, generalized Laplacians, and causal interventions further expand the versatility of the framework.
  Empirical evaluation on state-of-the-art reasoning benchmarks such as ProofWriter and CLUTRR demonstrates that Spectral NSR achieves superior accuracy, faster inference, improved robustness to adversarial perturbations, and higher interpretability compared to leading baselines including transformers, message-passing neural networks, and neuro-symbolic logic programming systems. Spectral attribution and proof-band agreement analyses confirm that model decisions align closely with symbolic proof structures, while transfer experiments validate effective domain adaptation through co-spectral alignment. These results establish Spectral NSR as a scalable and principled foundation for the next generation of reasoning systems, offering transparency, robustness, and generalization beyond conventional approaches.

http://arxiv.org/abs/2509.05835
Yours or Mine? Overwriting Attacks against Neural Audio Watermarking. (87%)
Lingfeng Yao; Chenpei Huang; Shengyao Wang; Junpei Xue; Hanqing Guo; Jiang Liu; Phone Lin; Tomoaki Ohtsuki; Miao Pan
As generative audio models are rapidly evolving, AI-generated audios increasingly raise concerns about copyright infringement and misinformation spread. Audio watermarking, as a proactive defense, can embed secret messages into audio for copyright protection and source verification. However, current neural audio watermarking methods focus primarily on the imperceptibility and robustness of watermarking, while ignoring its vulnerability to security attacks. In this paper, we develop a simple yet powerful attack: the overwriting attack that overwrites the legitimate audio watermark with a forged one and makes the original legitimate watermark undetectable. Based on the audio watermarking information that the adversary has, we propose three categories of overwriting attacks, i.e., white-box, gray-box, and black-box attacks. We also thoroughly evaluate the proposed attacks on state-of-the-art neural audio watermarking methods. Experimental results demonstrate that the proposed overwriting attacks can effectively compromise existing watermarking schemes across various settings and achieve a nearly 100% attack success rate. The practicality and effectiveness of the proposed overwriting attacks expose security flaws in existing neural audio watermarking systems, underscoring the need to enhance security in future audio watermarking designs.

http://arxiv.org/abs/2509.08000
AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs. (38%)
Debdeep Sanyal; Manodeep Ray; Murari Mandal
The release of open-weight large language models (LLMs) creates a tension between advancing accessible research and preventing misuse, such as malicious fine-tuning to elicit harmful content. Current safety measures struggle to preserve the general capabilities of the LLM while resisting a determined adversary with full access to the model's weights and architecture, who can use full-parameter fine-tuning to erase existing safeguards. To address this, we introduce AntiDote, a bi-level optimization procedure for training LLMs to be resistant to such tampering. AntiDote involves an auxiliary adversary hypernetwork that learns to generate malicious Low-Rank Adaptation (LoRA) weights conditioned on the defender model's internal activations. The defender LLM is then trained with an objective to nullify the effect of these adversarial weight additions, forcing it to maintain its safety alignment. We validate this approach against a diverse suite of 52 red-teaming attacks, including jailbreak prompting, latent space manipulation, and direct weight-space attacks. AntiDote is upto 27.4\% more robust against adversarial attacks compared to both tamper-resistance and unlearning baselines. Crucially, this robustness is achieved with a minimal trade-off in utility, incurring a performance degradation of upto less than 0.5\% across capability benchmarks including MMLU, HellaSwag, and GSM8K. Our work offers a practical and compute efficient methodology for building open-weight models where safety is a more integral and resilient property.

http://arxiv.org/abs/2509.05831
Decoding Latent Attack Surfaces in LLMs: Prompt Injection via HTML in Web Summarization. (4%)
Ishaan Verma
Large Language Models (LLMs) are increasingly integrated into web-based systems for content summarization, yet their susceptibility to prompt injection attacks remains a pressing concern. In this study, we explore how non-visible HTML elements such as <meta>, aria-label, and alt attributes can be exploited to embed adversarial instructions without altering the visible content of a webpage. We introduce a novel dataset comprising 280 static web pages, evenly divided between clean and adversarial injected versions, crafted using diverse HTML-based strategies. These pages are processed through a browser automation pipeline to extract both raw HTML and rendered text, closely mimicking real-world LLM deployment scenarios. We evaluate two state-of-the-art open-source models, Llama 4 Scout (Meta) and Gemma 9B IT (Google), on their ability to summarize this content. Using both lexical (ROUGE-L) and semantic (SBERT cosine similarity) metrics, along with manual annotations, we assess the impact of these covert injections. Our findings reveal that over 29% of injected samples led to noticeable changes in the Llama 4 Scout summaries, while Gemma 9B IT showed a lower, yet non-trivial, success rate of 15%. These results highlight a critical and largely overlooked vulnerability in LLM driven web pipelines, where hidden adversarial content can subtly manipulate model outputs. Our work offers a reproducible framework and benchmark for evaluating HTML-based prompt injection and underscores the urgent need for robust mitigation strategies in LLM applications involving web content.

http://arxiv.org/abs/2509.05739
Reasoning Introduces New Poisoning Attacks Yet Makes Them More Complicated. (2%)
Hanna Foerster; Ilia Shumailov; Yiren Zhao; Harsh Chaudhari; Jamie Hayes; Robert Mullins; Yarin Gal
Early research into data poisoning attacks against Large Language Models (LLMs) demonstrated the ease with which backdoors could be injected. More recent LLMs add step-by-step reasoning, expanding the attack surface to include the intermediate chain-of-thought (CoT) and its inherent trait of decomposing problems into subproblems. Using these vectors for more stealthy poisoning, we introduce ``decomposed reasoning poison'', in which the attacker modifies only the reasoning path, leaving prompts and final answers clean, and splits the trigger across multiple, individually harmless components.
  Fascinatingly, while it remains possible to inject these decomposed poisons, reliably activating them to change final answers (rather than just the CoT) is surprisingly difficult. This difficulty arises because the models can often recover from backdoors that are activated within their thought processes. Ultimately, it appears that an emergent form of backdoor robustness is originating from the reasoning capabilities of these advanced LLMs, as well as from the architectural separation between reasoning and final answer generation.

http://arxiv.org/abs/2508.07556
Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning. (1%)
Stephan Rabanser
Machine learning (ML) systems are increasingly deployed in high-stakes domains where reliability is paramount. This thesis investigates how uncertainty estimation can enhance the safety and trustworthiness of ML, focusing on selective prediction -- where models abstain when confidence is low.
  We first show that a model's training trajectory contains rich uncertainty signals that can be exploited without altering its architecture or loss. By ensembling predictions from intermediate checkpoints, we propose a lightweight, post-hoc abstention method that works across tasks, avoids the cost of deep ensembles, and achieves state-of-the-art selective prediction performance. Crucially, this approach is fully compatible with differential privacy (DP), allowing us to study how privacy noise affects uncertainty quality. We find that while many methods degrade under DP, our trajectory-based approach remains robust, and we introduce a framework for isolating the privacy-uncertainty trade-off. Next, we then develop a finite-sample decomposition of the selective classification gap -- the deviation from the oracle accuracy-coverage curve -- identifying five interpretable error sources and clarifying which interventions can close the gap. This explains why calibration alone cannot fix ranking errors, motivating methods that improve uncertainty ordering. Finally, we show that uncertainty signals can be adversarially manipulated to hide errors or deny service while maintaining high accuracy, and we design defenses combining calibration audits with verifiable inference.
  Together, these contributions advance reliable ML by improving, evaluating, and safeguarding uncertainty estimation, enabling models that not only make accurate predictions -- but also know when to say "I do not know".

http://arxiv.org/abs/2509.04980
MAIA: An Inpainting-Based Approach for Music Adversarial Attacks. (99%)
Yuxuan Liu; Peihong Zhang; Rui Sang; Zhixin Li; Shengchen Li
Music adversarial attacks have garnered significant interest in the field of Music Information Retrieval (MIR). In this paper, we present Music Adversarial Inpainting Attack (MAIA), a novel adversarial attack framework that supports both white-box and black-box attack scenarios. MAIA begins with an importance analysis to identify critical audio segments, which are then targeted for modification. Utilizing generative inpainting models, these segments are reconstructed with guidance from the output of the attacked model, ensuring subtle and effective adversarial perturbations. We evaluate MAIA on multiple MIR tasks, demonstrating high attack success rates in both white-box and black-box settings while maintaining minimal perceptual distortion. Additionally, subjective listening tests confirm the high audio fidelity of the adversarial samples. Our findings highlight vulnerabilities in current MIR systems and emphasize the need for more robust and secure models.

http://arxiv.org/abs/2509.04985
Training a Perceptual Model for Evaluating Auditory Similarity in Music Adversarial Attack. (98%)
Yuxuan Liu; Rui Sang; Peihong Zhang; Zhixin Li; Shengchen Li
Music Information Retrieval (MIR) systems are highly vulnerable to adversarial attacks that are often imperceptible to humans, primarily due to a misalignment between model feature spaces and human auditory perception. Existing defenses and perceptual metrics frequently fail to adequately capture these auditory nuances, a limitation supported by our initial listening tests showing low correlation between common metrics and human judgments. To bridge this gap, we introduce Perceptually-Aligned MERT Transformer (PAMT), a novel framework for learning robust, perceptually-aligned music representations. Our core innovation lies in the psychoacoustically-conditioned sequential contrastive transformer, a lightweight projection head built atop a frozen MERT encoder. PAMT achieves a Spearman correlation coefficient of 0.65 with subjective scores, outperforming existing perceptual metrics. Our approach also achieves an average of 9.15\% improvement in robust accuracy on challenging MIR tasks, including Cover Song Identification and Music Genre Classification, under diverse perceptual adversarial attacks. This work pioneers architecturally-integrated psychoacoustic conditioning, yielding representations significantly more aligned with human perception and robust against music adversarial attacks.

http://arxiv.org/abs/2509.05192
On Hyperparameters and Backdoor-Resistance in Horizontal Federated Learning. (89%)
Simon Lachnit; Ghassan Karame
Horizontal Federated Learning (HFL) is particularly vulnerable to backdoor attacks as adversaries can easily manipulate both the training data and processes to execute sophisticated attacks. In this work, we study the impact of training hyperparameters on the effectiveness of backdoor attacks and defenses in HFL. More specifically, we show both analytically and by means of measurements that the choice of hyperparameters by benign clients does not only influence model accuracy but also significantly impacts backdoor attack success. This stands in sharp contrast with the multitude of contributions in the area of HFL security, which often rely on custom ad-hoc hyperparameter choices for benign clients$\unicode{x2013}$leading to more pronounced backdoor attack strength and diminished impact of defenses. Our results indicate that properly tuning benign clients' hyperparameters$\unicode{x2013}$such as learning rate, batch size, and number of local epochs$\unicode{x2013}$can significantly curb the effectiveness of backdoor attacks, regardless of the malicious clients' settings. We support this claim with an extensive robustness evaluation of state-of-the-art attack-defense combinations, showing that carefully chosen hyperparameters yield across-the-board improvements in robustness without sacrificing main task accuracy. For example, we show that the 50%-lifespan of the strong A3FL attack can be reduced by 98.6%, respectively$\unicode{x2013}$all without using any defense and while incurring only a 2.9 percentage points drop in clean task accuracy.

http://arxiv.org/abs/2509.09706
Differential Robustness in Transformer Language Models: Empirical Evaluation Under Adversarial Text Attacks. (80%)
Taniya Gidatkar; Oluwaseun Ajao; Matthew Shardlow
This study evaluates the resilience of large language models (LLMs) against adversarial attacks, specifically focusing on Flan-T5, BERT, and RoBERTa-Base. Using systematically designed adversarial tests through TextFooler and BERTAttack, we found significant variations in model robustness. RoBERTa-Base and FlanT5 demonstrated remarkable resilience, maintaining accuracy even when subjected to sophisticated attacks, with attack success rates of 0%. In contrast. BERT-Base showed considerable vulnerability, with TextFooler achieving a 93.75% success rate in reducing model accuracy from 48% to just 3%. Our research reveals that while certain LLMs have developed effective defensive mechanisms, these safeguards often require substantial computational resources. This study contributes to the understanding of LLM security by identifying existing strengths and weaknesses in current safeguarding approaches and proposes practical recommendations for developing more efficient and effective defensive strategies.

http://arxiv.org/abs/2506.13205
Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents. (78%)
Xuan Wang; Siyuan Liang; Zhe Liu; Yi Yu; Aishan Liu; Yuliang Lu; Xitong Gao; Ee-Chien Chang
Mobile agents powered by vision-language models (VLMs) are increasingly adopted for tasks such as UI automation and camera-based assistance. These agents are typically fine-tuned using small-scale, user-collected data, making them susceptible to stealthy training-time threats. This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents. The attack injects malicious behaviors into the model by modifying only the visual input while preserving textual prompts and instructions, achieving stealth through the complete absence of textual anomalies. Once the agent is fine-tuned on this poisoned data, adding a predefined visual pattern (trigger) at inference time activates the attacker-specified behavior (backdoor). Our attack aligns the training gradients of poisoned samples with those of an attacker-specified target instance, effectively embedding backdoor-specific features into the poisoned data. To ensure the robustness and stealthiness of the attack, we design three trigger variants that better resemble real-world scenarios: static patches, dynamic motion patterns, and low-opacity blended content. Extensive experiments on six Android applications and three mobile-compatible VLMs demonstrate that our attack achieves high success rates (ASR up to 94.67%) while preserving clean-task behavior (FSR up to 95.85%). We further conduct ablation studies to understand how key design factors impact attack reliability and stealth. These findings is the first to reveal the security vulnerabilities of mobile agents and their susceptibility to backdoor injection, underscoring the need for robust defenses in mobile agent adaptation pipelines.

http://arxiv.org/abs/2509.05265
On Evaluating the Poisoning Robustness of Federated Learning under Local Differential Privacy. (76%)
Zijian Wang; Wei Tong; Tingxuan Han; Haoyu Chen; Tianling Zhang; Yunlong Mao; Sheng Zhong
Federated learning (FL) combined with local differential privacy (LDP) enables privacy-preserving model training across decentralized data sources. However, the decentralized data-management paradigm leaves LDPFL vulnerable to participants with malicious intent. The robustness of LDPFL protocols, particularly against model poisoning attacks (MPA), where adversaries inject malicious updates to disrupt global model convergence, remains insufficiently studied. In this paper, we propose a novel and extensible model poisoning attack framework tailored for LDPFL settings. Our approach is driven by the objective of maximizing the global training loss while adhering to local privacy constraints. To counter robust aggregation mechanisms such as Multi-Krum and trimmed mean, we develop adaptive attacks that embed carefully crafted constraints into a reverse training process, enabling evasion of these defenses. We evaluate our framework across three representative LDPFL protocols, three benchmark datasets, and two types of deep neural networks. Additionally, we investigate the influence of data heterogeneity and privacy budgets on attack effectiveness. Experimental results demonstrate that our adaptive attacks can significantly degrade the performance of the global model, revealing critical vulnerabilities and highlighting the need for more robust LDPFL defense strategies against MPA. Our code is available at https://github.com/ZiJW/LDPFL-Attack

http://arxiv.org/abs/2509.05086
Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers. (54%)
Svetlana Pavlitska; Haixi Fan; Konstantin Ditschuneit; J. Marius Zöllner
Robustifying convolutional neural networks (CNNs) against adversarial attacks remains challenging and often requires resource-intensive countermeasures. We explore the use of sparse mixture-of-experts (MoE) layers to improve robustness by replacing selected residual blocks or convolutional layers, thereby increasing model capacity without additional inference cost. On ResNet architectures trained on CIFAR-100, we find that inserting a single MoE layer in the deeper stages leads to consistent improvements in robustness under PGD and AutoPGD attacks when combined with adversarial training. Furthermore, we discover that when switch loss is used for balancing, it causes routing to collapse onto a small set of overused experts, thereby concentrating adversarial training on these paths and inadvertently making them more robust. As a result, some individual experts outperform the gated MoE model in robustness, suggesting that robust subpaths emerge through specialization. Our code is available at https://github.com/KASTEL-MobilityLab/robust-sparse-moes.

http://arxiv.org/abs/2509.05429
Safeguarding Graph Neural Networks against Topology Inference Attacks. (45%)
Jie Fu; Hong Yuan; Zhili Chen; Wendy Hui Wang
Graph Neural Networks (GNNs) have emerged as powerful models for learning from graph-structured data. However, their widespread adoption has raised serious privacy concerns. While prior research has primarily focused on edge-level privacy, a critical yet underexplored threat lies in topology privacy - the confidentiality of the graph's overall structure. In this work, we present a comprehensive study on topology privacy risks in GNNs, revealing their vulnerability to graph-level inference attacks. To this end, we propose a suite of Topology Inference Attacks (TIAs) that can reconstruct the structure of a target training graph using only black-box access to a GNN model. Our findings show that GNNs are highly susceptible to these attacks, and that existing edge-level differential privacy mechanisms are insufficient as they either fail to mitigate the risk or severely compromise model accuracy. To address this challenge, we introduce Private Graph Reconstruction (PGR), a novel defense framework designed to protect topology privacy while maintaining model accuracy. PGR is formulated as a bi-level optimization problem, where a synthetic training graph is iteratively generated using meta-gradients, and the GNN model is concurrently updated based on the evolving graph. Extensive experiments demonstrate that PGR significantly reduces topology leakage with minimal impact on model accuracy. Our code is anonymously available at https://github.com/JeffffffFu/PGR.

http://arxiv.org/abs/2410.17351
Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense. (41%)
Aditya Vikram Singh; Ethan Rathbun; Emma Graham; Lisa Oakley; Simona Boboila; Alina Oprea; Peter Chin
Recent advances in multi-agent reinforcement learning (MARL) have created opportunities to solve complex real-world tasks. Cybersecurity is a notable application area, where defending networks against sophisticated adversaries remains a challenging task typically performed by teams of security operators. In this work, we explore novel MARL strategies for building autonomous cyber network defenses that address challenges such as large policy spaces, partial observability, and stealthy, deceptive adversarial strategies. To facilitate efficient and generalized learning, we propose a hierarchical Proximal Policy Optimization (PPO) architecture that decomposes the cyber defense task into specific sub-tasks like network investigation and host recovery. Our approach involves training sub-policies for each sub-task using PPO enhanced with cybersecurity domain expertise. These sub-policies are then leveraged by a master defense policy that coordinates their selection to solve complex network defense tasks. Furthermore, the sub-policies can be fine-tuned and transferred with minimal cost to defend against shifts in adversarial behavior or changes in network settings. We conduct extensive experiments using CybORG Cage 4, the state-of-the-art MARL environment for cyber defense. Comparisons with multiple baselines across different adversaries show that our hierarchical learning approach achieves top performance in terms of convergence speed, episodic return, and several interpretable metrics relevant to cybersecurity, including the fraction of clean machines on the network, precision, and false positives.

http://arxiv.org/abs/2509.05471
Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models. (5%)
Youjia Zheng; Mohammad Zandsalimy; Shanu Sushmita
Large Language Models (LLMs) are increasingly vulnerable to a sophisticated form of adversarial prompting known as camouflaged jailbreaking. This method embeds malicious intent within seemingly benign language to evade existing safety mechanisms. Unlike overt attacks, these subtle prompts exploit contextual ambiguity and the flexible nature of language, posing significant challenges to current defense systems. This paper investigates the construction and impact of camouflaged jailbreak prompts, emphasizing their deceptive characteristics and the limitations of traditional keyword-based detection methods. We introduce a novel benchmark dataset, Camouflaged Jailbreak Prompts, containing 500 curated examples (400 harmful and 100 benign prompts) designed to rigorously stress-test LLM safety protocols. In addition, we propose a multi-faceted evaluation framework that measures harmfulness across seven dimensions: Safety Awareness, Technical Feasibility, Implementation Safeguards, Harmful Potential, Educational Value, Content Quality, and Compliance Score. Our findings reveal a stark contrast in LLM behavior: while models demonstrate high safety and content quality with benign inputs, they exhibit a significant decline in performance and safety when confronted with camouflaged jailbreak attempts. This disparity underscores a pervasive vulnerability, highlighting the urgent need for more nuanced and adaptive security strategies to ensure the responsible and robust deployment of LLMs in real-world applications.

http://arxiv.org/abs/2509.09703
CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor. (2%)
Zhenhua Xu; Xixiang Zhao; Xubin Yue; Shengwei Tian; Changting Lin; Meng Han
The widespread deployment of large language models (LLMs) has intensified concerns around intellectual property (IP) protection, as model theft and unauthorized redistribution become increasingly feasible. To address this, model fingerprinting aims to embed verifiable ownership traces into LLMs. However, existing methods face inherent trade-offs between stealthness, robustness, and generalizability, being either detectable via distributional shifts, vulnerable to adversarial modifications, or easily invalidated once the fingerprint is revealed. In this work, we introduce CTCC, a novel rule-driven fingerprinting framework that encodes contextual correlations across multiple dialogue turns, such as counterfactual, rather than relying on token-level or single-turn triggers. CTCC enables fingerprint verification under black-box access while mitigating false positives and fingerprint leakage, supporting continuous construction under a shared semantic rule even if partial triggers are exposed. Extensive experiments across multiple LLM architectures demonstrate that CTCC consistently achieves stronger stealth and robustness than prior work. Our findings position CTCC as a reliable and practical solution for ownership verification in real-world LLM deployment scenarios. Our code and data are publicly available at <https://github.com/Xuzhenhua55/CTCC>.

http://arxiv.org/abs/2408.09600
Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning. (2%)
Tiansheng Huang; Gautam Bhattacharya; Pratik Joshi; Josh Kimball; Ling Liu
Safety aligned Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- a few harmful data mixed in the fine-tuning dataset can break the LLMs's safety alignment. While several defenses have been proposed, our evaluation shows that existing defenses fail \textit{when some specific training hyper-parameters are chosen} -- a large learning rate or a large number of training epochs in the fine-tuning stage can easily invalidate the defense. To this end, we propose Antidote, a post-fine-tuning stage solution, which remains \textbf{\textit{agnostic to the training hyper-parameters in the fine-tuning stage}}. Antidote relies on the philosophy that by removing the harmful parameters, the harmful model can be recovered from the harmful behaviors, regardless of how those harmful parameters are formed in the fine-tuning stage. With this philosophy, we introduce a one-shot pruning stage after harmful fine-tuning to remove the harmful weights that are responsible for the generation of harmful content. Despite its embarrassing simplicity, empirical results show that Antidote can reduce harmful score while maintaining accuracy on downstream tasks. Code is available at https://github.com/git-disl/Antidote.

http://arxiv.org/abs/2509.04914
RobQFL: Robust Quantum Federated Learning in Adversarial Environment. (2%)
Walid El Maouaki; Nouhaila Innan; Alberto Marchisio; Taoufik Said; Muhammad Shafique; Mohamed Bennai
Quantum Federated Learning (QFL) merges privacy-preserving federation with quantum computing gains, yet its resilience to adversarial noise is unknown. We first show that QFL is as fragile as centralized quantum learning. We propose Robust Quantum Federated Learning (RobQFL), embedding adversarial training directly into the federated loop. RobQFL exposes tunable axes: client coverage $γ$ (0-100\%), perturbation scheduling (fixed-$\varepsilon$ vs $\varepsilon$-mixes), and optimization (fine-tune vs scratch), and distils the resulting $γ\times \varepsilon$ surface into two metrics: Accuracy-Robustness Area and Robustness Volume. On 15-client simulations with MNIST and Fashion-MNIST, IID and Non-IID conditions, training only 20-50\% clients adversarially boosts $\varepsilon \leq 0.1$ accuracy $\sim$15 pp at $< 2$ pp clean-accuracy cost; fine-tuning adds 3-5 pp. With $\geq$75\% coverage, a moderate $\varepsilon$-mix is optimal, while high-$\varepsilon$ schedules help only at 100\% coverage. Label-sorted non-IID splits halve robustness, underscoring data heterogeneity as a dominant risk.

http://arxiv.org/abs/2503.02780
Quantitative Resilience Modeling for Autonomous Cyber Defense. (1%)
Xavier Cadet; Simona Boboila; Edward Koh; Peter Chin; Alina Oprea
Cyber resilience is the ability of a system to recover from an attack with minimal impact on system operations. However, characterizing a network's resilience under a cyber attack is challenging, as there are no formal definitions of resilience applicable to diverse network topologies and attack patterns. In this work, we propose a quantifiable formulation of resilience that considers multiple defender operational goals, the criticality of various network resources for daily operations, and provides interpretability to security operators about their system's resilience under attack. We evaluate our approach within the CybORG environment, a reinforcement learning (RL) framework for autonomous cyber defense, analyzing trade-offs between resilience, costs, and prioritization of operational goals. Furthermore, we introduce methods to aggregate resilience metrics across time-variable attack patterns and multiple network topologies, comprehensively characterizing system resilience. Using insights gained from our resilience metrics, we design RL autonomous defensive agents and compare them against several heuristic baselines, showing that proactive network hardening techniques and prompt recovery of compromised machines are critical for effective cyber defenses.

http://arxiv.org/abs/2509.04802
Mind the Gap: Evaluating Model- and Agentic-Level Vulnerabilities in LLMs with Action Graphs. (1%)
Ilham Wicaksono; Zekun Wu; Theo King; Adriano Koshiyama; Philip Treleaven
As large language models transition to agentic systems, current safety evaluation frameworks face critical gaps in assessing deployment-specific risks. We introduce AgentSeer, an observability-based evaluation framework that decomposes agentic executions into granular action and component graphs, enabling systematic agentic-situational assessment. Through cross-model validation on GPT-OSS-20B and Gemini-2.0-flash using HarmBench single turn and iterative refinement attacks, we demonstrate fundamental differences between model-level and agentic-level vulnerability profiles. Model-level evaluation reveals baseline differences: GPT-OSS-20B (39.47% ASR) versus Gemini-2.0-flash (50.00% ASR), with both models showing susceptibility to social engineering while maintaining logic-based attack resistance. However, agentic-level assessment exposes agent-specific risks invisible to traditional evaluation. We discover "agentic-only" vulnerabilities that emerge exclusively in agentic contexts, with tool-calling showing 24-60% higher ASR across both models. Cross-model analysis reveals universal agentic patterns, agent transfer operations as highest-risk tools, semantic rather than syntactic vulnerability mechanisms, and context-dependent attack effectiveness, alongside model-specific security profiles in absolute ASR levels and optimal injection strategies. Direct attack transfer from model-level to agentic contexts shows degraded performance (GPT-OSS-20B: 57% human injection ASR; Gemini-2.0-flash: 28%), while context-aware iterative attacks successfully compromise objectives that failed at model-level, confirming systematic evaluation gaps. These findings establish the urgent need for agentic-situation evaluation paradigms, with AgentSeer providing the standardized methodology and empirical validation.

http://arxiv.org/abs/2509.04887
RINSER: Accurate API Prediction Using Masked Language Models. (1%)
Muhammad Ejaz Ahmed; Christopher Cody; Muhammad Ikram; Sean Lamont; Alsharif Abuadbba; Seyit Camtepe; Surya Nepal; Muhammad Ali Kaafar
Malware authors commonly use obfuscation to hide API identities in binary files, making analysis difficult and time-consuming for a human expert to understand the behavior and intent of the program. Automatic API prediction tools are necessary to efficiently analyze unknown binaries, facilitating rapid malware triage while reducing the workload on human analysts. In this paper, we present RINSER (AccuRate API predictioN using maSked languagE model leaRning), an automated framework for predicting Windows API (WinAPI) function names. RINSER introduces the novel concept of API codeprints, a set of API-relevant assembly instructions, and supports x86 PE binaries. RINSER relies on BERT's masked language model (LM) to predict API names at scale, achieving 85.77% accuracy for normal binaries and 82.88% accuracy for stripped binaries. We evaluate RINSER on a large dataset of 4.7M API codeprints from 11,098 malware binaries, covering 4,123 unique Windows APIs, making it the largest publicly available dataset of this type. RINSER successfully discovered 65 obfuscated Windows APIs related to C2 communication, spying, and evasion in our dataset, which the commercial disassembler IDA failed to identify. Furthermore, we compared RINSER against three state-of-the-art approaches, showing over 20% higher prediction accuracy. We also demonstrated RINSER's resilience to adversarial attacks, including instruction randomization and code displacement, with a performance drop of no more than 3%.

http://arxiv.org/abs/2509.04597
DisPatch: Disarming Adversarial Patches in Object Detection with Diffusion Models. (99%)
Jin Ma; Mohammed Aldeen; Christopher Salas; Feng Luo; Mashrur Chowdhury; Mert Pesé; Long Cheng
Object detection is fundamental to various real-world applications, such as security monitoring and surveillance video analysis. Despite their advancements, state-of-theart object detectors are still vulnerable to adversarial patch attacks, which can be easily applied to real-world objects to either conceal actual items or create non-existent ones, leading to severe consequences. Given the current diversity of adversarial patch attacks and potential unknown threats, an ideal defense method should be effective, generalizable, and robust against adaptive attacks. In this work, we introduce DISPATCH, the first diffusion-based defense framework for object detection. Unlike previous works that aim to "detect and remove" adversarial patches, DISPATCH adopts a "regenerate and rectify" strategy, leveraging generative models to disarm attack effects while preserving the integrity of the input image. Specifically, we utilize the in-distribution generative power of diffusion models to regenerate the entire image, aligning it with benign data. A rectification process is then employed to identify and replace adversarial regions with their regenerated benign counterparts. DISPATCH is attack-agnostic and requires no prior knowledge of the existing patches. Extensive experiments across multiple detectors and attacks demonstrate that DISPATCH consistently outperforms state-of-the-art defenses on both hiding attacks and creating attacks, achieving the best overall mAP.5 score of 89.3% on hiding attacks, and lowering the attack success rate to 24.8% on untargeted creating attacks. Moreover, it maintains strong robustness against adaptive attacks, making it a practical and reliable defense for object detection systems.

http://arxiv.org/abs/2412.12722
Defending LVLMs Against Vision Attacks through Partial-Perception Supervision. (92%)
Qi Zhou; Tianlin Li; Qing Guo; Dongxia Wang; Yun Lin; Yang Liu; Jin Song Dong
Recent studies have raised significant concerns regarding the vulnerability of Large Vision Language Models (LVLMs) to maliciously injected or perturbed input images, which can mislead their responses. Existing defense methods show that such vision attacks are sensitive to image modifications especially cropping, using majority voting across responses of modified images as corrected responses. However, these modifications often result in partial images and distort the semantics, which reduces response quality on clean images after voting. Instead of directly using responses from partial images for voting, we investigate using them to supervise the LVLM's responses to the original images. We propose a black-box, training-free method called DPS (Defense through Partial-Perception Supervision). In this approach, the model is prompted using the responses generated by a model that perceives only a partial image. With DPS, the model can adjust its response based on partial image understanding when under attack, while confidently maintaining its original response for clean input. Our findings show that the weak model can supervise the strong model: when faced with an attacked input, the strong model becomes less confident and adjusts its response based on the weak model's partial understanding, effectively defending against the attack. With clean input, it confidently maintains its original response. Empirical experiments show our method outperforms the baseline, cutting the average attack success rate by 76.3% across six datasets on three popular models.

http://arxiv.org/abs/2509.05372
Adversarial Bug Reports as a Security Risk in Language Model-Based Automated Program Repair. (87%)
Piotr Przymus; Andreas Happe; Jürgen Cito
Large Language Model (LLM) - based Automated Program Repair (APR) systems are increasingly integrated into modern software development workflows, offering automated patches in response to natural language bug reports. However, this reliance on untrusted user input introduces a novel and underexplored attack surface. In this paper, we investigate the security risks posed by adversarial bug reports -- realistic-looking issue submissions crafted to mislead APR systems into producing insecure or harmful code changes. We develop a comprehensive threat model and conduct an empirical study to evaluate the vulnerability of state-of-the-art APR systems to such attacks. Our demonstration comprises 51 adversarial bug reports generated across a spectrum of strategies, from manual curation to fully automated pipelines. We test these against leading APR model and assess both pre-repair defenses (e.g., LlamaGuard variants, PromptGuard variants, Granite-Guardian, and custom LLM filters) and post-repair detectors (GitHub Copilot, CodeQL). Our findings show that current defenses are insufficient: 90\% of crafted bug reports triggered attacker-aligned patches. The best pre-repair filter blocked only 47\%, while post-repair analysis-often requiring human oversight-was effective in just 58\% of cases. To support scalable security testing, we introduce a prototype framework for automating the generation of adversarial bug reports. Our analysis exposes a structural asymmetry: generating adversarial inputs is inexpensive, while detecting or mitigating them remains costly and error-prone. We conclude with practical recommendations for improving the robustness of APR systems against adversarial misuse and highlight directions for future work on trustworthy automated repair.

http://arxiv.org/abs/2401.09754
Towards Robust Graph Structural Learning Beyond Homophily via Preserving Neighbor Similarity. (83%)
Yulin Zhu; Yuni Lai; Xing Ai; Wai Lun LO; Gaolei Li; Jianhua Li; Di Tang; Xingxing Zhang; Mengpei Yang; Kai Zhou
Despite the tremendous success of graph-based learning systems in handling structural data, it has been widely investigated that they are fragile to adversarial attacks on homophilic graph data, where adversaries maliciously modify the semantic and topology information of the raw graph data to degrade the predictive performances. Motivated by this, a series of robust models are crafted to enhance the adversarial robustness of graph-based learning systems on homophilic graphs. However, the security of graph-based learning systems on heterophilic graphs remains a mystery to us. To bridge this gap, in this paper, we start to explore the vulnerability of graph-based learning systems regardless of the homophily degree, and theoretically prove that the update of the negative classification loss is negatively correlated with the pairwise similarities based on the powered aggregated neighbor features. The theoretical finding inspires us to craft a novel robust graph structural learning strategy that serves as a useful graph mining module in a robust model that incorporates a dual-kNN graph constructions pipeline to supervise the neighbor-similarity-preserved propagation, where the graph convolutional layer adaptively smooths or discriminates the features of node pairs according to their affluent local structures. In this way, the proposed methods can mine the ``better" topology of the raw graph data under diverse graph homophily and achieve more reliable data management on homophilic and heterophilic graphs.

http://arxiv.org/abs/2509.04214
An Automated, Scalable Machine Learning Model Inversion Assessment Pipeline. (22%)
Tyler Shumaker; Jessica Carpenter; David Saranchak; Nathaniel D. Bastian
Machine learning (ML) models have the potential to transform military battlefields, presenting a large external pressure to rapidly incorporate them into operational settings. However, it is well-established that these ML models are vulnerable to a number of adversarial attacks throughout the model deployment pipeline that threaten to negate battlefield advantage. One broad category is privacy attacks (such as model inversion) where an adversary can reverse engineer information from the model, such as the sensitive data used in its training. The ability to quantify the risk of model inversion attacks (MIAs) is not well studied, and there is a lack of automated developmental test and evaluation (DT&E) tools and metrics to quantify the effectiveness of privacy loss of the MIA. The current DT&E process is difficult because ML model inversions can be hard for a human to interpret, subjective when they are interpretable, and difficult to quantify in terms of inversion quality. Additionally, scaling the DT&E process is challenging due to many ML model architectures and data modalities that need to be assessed. In this work, we present a novel DT&E tool that quantifies the risk of data privacy loss from MIAs and introduces four adversarial risk dimensions to quantify privacy loss. Our DT&E pipeline combines inversion with vision language models (VLMs) to improve effectiveness while enabling scalable analysis. We demonstrate effectiveness using multiple MIA techniques and VLMs configured for zero-shot classification and image captioning. We benchmark the pipeline using several state-of-the-art MIAs in the computer vision domain with an image classification task that is typical in military applications. In general, our innovative pipeline extends the current model inversion DT&E capabilities by improving the effectiveness and scalability of the privacy loss analysis in an automated fashion.

http://arxiv.org/abs/2509.03985
NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Models. (5%)
Chuhan Zhang; Ye Zhang; Bowen Shi; Yuyou Gan; Tianyu Du; Shouling Ji; Dazhan Deng; Yingcai Wu
In deployment and application, large language models (LLMs) typically undergo safety alignment to prevent illegal and unethical outputs. However, the continuous advancement of jailbreak attack techniques, designed to bypass safety mechanisms with adversarial prompts, has placed increasing pressure on the security defenses of LLMs. Strengthening resistance to jailbreak attacks requires an in-depth understanding of the security mechanisms and vulnerabilities of LLMs. However, the vast number of parameters and complex structure of LLMs make analyzing security weaknesses from an internal perspective a challenging task. This paper presents NeuroBreak, a top-down jailbreak analysis system designed to analyze neuron-level safety mechanisms and mitigate vulnerabilities. We carefully design system requirements through collaboration with three experts in the field of AI security. The system provides a comprehensive analysis of various jailbreak attack methods. By incorporating layer-wise representation probing analysis, NeuroBreak offers a novel perspective on the model's decision-making process throughout its generation steps. Furthermore, the system supports the analysis of critical neurons from both semantic and functional perspectives, facilitating a deeper exploration of security mechanisms. We conduct quantitative evaluations and case studies to verify the effectiveness of our system, offering mechanistic insights for developing next-generation defense strategies against evolving jailbreak attacks.

http://arxiv.org/abs/2509.05377
Enhancing Gradient Variance and Differential Privacy in Quantum Federated Learning. (1%)
Duc-Thien Phan; Minh-Duong Nguyen; Quoc-Viet Pham; Huilong Pi
Upon integrating Quantum Neural Network (QNN) as the local model, Quantum Federated Learning (QFL) has recently confronted notable challenges. Firstly, exploration is hindered over sharp minima, decreasing learning performance. Secondly, the steady gradient descent results in more stable and predictable model transmissions over wireless channels, making the model more susceptible to attacks from adversarial entities. Additionally, the local QFL model is vulnerable to noise produced by the quantum device's intermediate noise states, since it requires the use of quantum gates and circuits for training. This local noise becomes intertwined with learning parameters during training, impairing model precision and convergence rate. To address these issues, we propose a new QFL technique that incorporates differential privacy and introduces a dedicated noise estimation strategy to quantify and mitigate the impact of intermediate quantum noise. Furthermore, we design an adaptive noise generation scheme to alleviate privacy threats associated with the vanishing gradient variance phenomenon of QNN and enhance robustness against device noise. Experimental results demonstrate that our algorithm effectively balances convergence, reduces communication costs, and mitigates the adverse effects of intermediate quantum noise while maintaining strong privacy protection. Using real-world datasets, we achieved test accuracy of up to 98.47\% for the MNIST dataset and 83.85\% for the CIFAR-10 dataset while maintaining fast execution times.

http://arxiv.org/abs/2508.18826
SWiFT: Soft-Mask Weight Fine-tuning for Bias Mitigation. (1%)
Junyu Yan; Feng Chen; Yuyang Xue; Yuning Du; Konstantinos Vilouras; Sotirios A. Tsaftaris; Steven McDonagh
Recent studies have shown that Machine Learning (ML) models can exhibit bias in real-world scenarios, posing significant challenges in ethically sensitive domains such as healthcare. Such bias can negatively affect model fairness, model generalization abilities and further risks amplifying social discrimination. There is a need to remove biases from trained models. Existing debiasing approaches often necessitate access to original training data and need extensive model retraining; they also typically exhibit trade-offs between model fairness and discriminative performance. To address these challenges, we propose Soft-Mask Weight Fine-Tuning (SWiFT), a debiasing framework that efficiently improves fairness while preserving discriminative performance with much less debiasing costs. Notably, SWiFT requires only a small external dataset and only a few epochs of model fine-tuning. The idea behind SWiFT is to first find the relative, and yet distinct, contributions of model parameters to both bias and predictive performance. Then, a two-step fine-tuning process updates each parameter with different gradient flows defined by its contribution. Extensive experiments with three bias sensitive attributes (gender, skin tone, and age) across four dermatological and two chest X-ray datasets demonstrate that SWiFT can consistently reduce model bias while achieving competitive or even superior diagnostic accuracy under common fairness and accuracy metrics, compared to the state-of-the-art. Specifically, we demonstrate improved model generalization ability as evidenced by superior performance on several out-of-distribution (OOD) datasets.

http://arxiv.org/abs/2508.20038
Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks. (1%)
Sheng Liu; Qiang Sheng; Danding Wang; Yang Li; Guang Yang; Juan Cao
Despite advances in improving large language model (LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

http://arxiv.org/abs/2509.08748
Prototype-Guided Robust Learning against Backdoor Attacks. (98%)
Wei Guo; Maura Pintor; Ambra Demontis; Battista Biggio
Backdoor attacks poison the training data to embed a backdoor in the model, causing it to behave normally on legitimate inputs but maliciously when specific trigger signals appear. Training a benign model from a dataset poisoned by backdoor attacks is challenging. Existing works rely on various assumptions and can only defend against backdoor attacks with specific trigger signals, high poisoning ratios, or when the defender possesses a large, untainted, validation dataset. In this paper, we propose a defense called Prototype-Guided Robust Learning (PGRL), which overcomes all the aforementioned limitations, being robust against diverse backdoor attacks. Leveraging a tiny set of benign samples, PGRL generates prototype vectors to guide the training process. We compare our PGRL with 8 existing defenses, showing that it achieves superior robustness. We also demonstrate that PGRL generalizes well across various architectures, datasets, and advanced attacks. Finally, to evaluate our PGRL in the worst-case scenario, we perform an adaptive attack, where the attackers fully know the details of the defense.

http://arxiv.org/abs/2509.06992
FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models. (96%)
Kun Zhai; Siheng Chen; Xingjun Ma; Yu-Gang Jiang
Federated Prompt Tuning (FPT) is an efficient method for cross-client collaborative fine-tuning of large Vision-Language Models (VLMs). However, models tuned using FPT are vulnerable to adversarial attacks, leading to misclassification in downstream tasks. In this work, we introduce Federated Adversarial Prompt Tuning (\textbf{FedAPT}), a novel method designed to enhance the adversarial robustness of FPT. We identify a key issue in FedAPT under non-independent and identically distributed (non-IID) settings: a \textit{class information gap} between clients and the global model. Clients rely solely on limited local label information to generate adversarial samples for training, while the global model must defend against adversarial attacks from global labels. To address this issue, we propose a \textbf{class-aware prompt generator} that generates visual prompts from text prompts. This generator is guided by a \emph{Global Label Embedding} (serving as a ``beacon") which encodes cross-client label information to create more globally-aligned visual prompts. Additionally, we propose a \textbf{cross-layer generator sharing} strategy to enhance prompt coupling across different layers of the model, further boosting adversarial robustness. Extensive experiments on multiple image classification datasets demonstrate the superiority of FedAPT in improving adversarial robustness, outperforming existing methods by a large margin. FedAPT also exhibits exceptional generalization in cross-domain and cross-dataset scenarios, indicating its effectiveness in real-world applications.

http://arxiv.org/abs/2509.03383
ANNIE: Be Careful of Your Robots. (92%)
Yiyang Huang; Zixuan Wang; Zishen Wan; Yapeng Tian; Haobo Xu; Yinhe Han; Yiming Gan
The integration of vision-language-action (VLA) models into embodied AI (EAI) robots is rapidly advancing their ability to perform complex, long-horizon tasks in humancentric environments. However, EAI systems introduce critical security risks: a compromised VLA model can directly translate adversarial perturbations on sensory input into unsafe physical actions. Traditional safety definitions and methodologies from the machine learning community are no longer sufficient. EAI systems raise new questions, such as what constitutes safety, how to measure it, and how to design effective attack and defense mechanisms in physically grounded, interactive settings. In this work, we present the first systematic study of adversarial safety attacks on embodied AI systems, grounded in ISO standards for human-robot interactions. We (1) formalize a principled taxonomy of safety violations (critical, dangerous, risky) based on physical constraints such as separation distance, velocity, and collision boundaries; (2) introduce ANNIEBench, a benchmark of nine safety-critical scenarios with 2,400 video-action sequences for evaluating embodied safety; and (3) ANNIE-Attack, a task-aware adversarial framework with an attack leader model that decomposes long-horizon goals into frame-level perturbations. Our evaluation across representative EAI models shows attack success rates exceeding 50% across all safety categories. We further demonstrate sparse and adaptive attack strategies and validate the real-world impact through physical robot experiments. These results expose a previously underexplored but highly consequential attack surface in embodied AI systems, highlighting the urgent need for security-driven defenses in the physical AI era. Code is available at https://github.com/RLCLab/Annie.

http://arxiv.org/abs/2509.08747
Silent Until Sparse: Backdoor Attacks on Semi-Structured Sparsity. (73%)
Wei Guo; Maura Pintor; Ambra Demontis; Battista Biggio
In the deployment phase, semi-structured sparsity accelerates the execution of deep neural networks on modern GPUs via sparse matrix multiplication. In this paper, targeting the semi-structured sparsity, we introduce a Silent Until Sparse (SUS) backdoor attack, where the released full model remains silent (benign), but becomes a backdoored model after sparsification. The attack operates in two phases: (i) in the backdoor training phase, the backdoor functionality is injected into specific weights that will be retained during the pruning process; (ii) in the backdoor hiding phase, the malicious behavior is concealed by fine-tuning elements that will be pruned away. This dual-phase approach ensures that the attack remains undetectable in the released model, but activates properly once the model is pruned with the semi-structured sparsity. Through extensive experiments, we show that our attack successfully threatens the semi-structured sparsity algorithms from both NVIDIA and PyTorch. Our empirical results show that, regardless of model architecture, the attack success rate of the released model remains below 10% prior to sparsification but exceeds 99% afterward. Moreover, we demonstrate that SUS attack is robust against state-of-the-art backdoor defenses and finetuning, highlighting a critical vulnerability in current model compression and deployment pipelines.

http://arxiv.org/abs/2509.03242
TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space. (54%)
Vita Gianmarco De; Nargiz Humbatova; Paolo Tonella
Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of inputs by features that make the DL model under test fail is largely unexplored. Existing approaches for DL testing introduce perturbations that may focus on specific failure-inducing features, while neglecting others that belong to different regions of the feature space. In this paper, we create an explicit topographical map of the input feature space. Our approach, named TopoMap, is both black-box and model-agnostic as it relies solely on features that characterise the input space. To discriminate the inputs according to the specific features they share, we first apply dimensionality reduction to obtain input embeddings, which are then subjected to clustering. Each DL model might require specific embedding computations and clustering algorithms to achieve a meaningful separation of inputs into discriminative groups. We propose a novel way to evaluate alternative configurations of embedding and clustering techniques. We used a deep neural network (DNN) as an approximation of a human evaluator who could tell whether a pair of clusters can be discriminated based on the features of the included elements. We use such a DNN to automatically select the optimal topographical map of the inputs among all those that are produced by different embedding/clustering configurations. The evaluation results show that the maps generated by TopoMap consist of distinguishable and meaningful regions. In addition, we evaluate the effectiveness of TopoMap using mutation analysis. In particular, we assess whether the clusters in our topographical map allow for an effective selection of mutation-killing inputs. Experimental results show that our approach outperforms random selection by 35% on average on killable mutants; by 61% on non-killable ones.

http://arxiv.org/abs/2509.03179
AutoDetect: Designing an Autoencoder-based Detection Method for Poisoning Attacks on Object Detection Applications in the Military Domain. (41%)
Alma M. Liezenga; Stefan Wijnja; Haan Puck de; Niels W. T. Brink; Stijn Jip J. van; Yori Kamphuis; Klamer Schutte
Poisoning attacks pose an increasing threat to the security and robustness of Artificial Intelligence systems in the military domain. The widespread use of open-source datasets and pretrained models exacerbates this risk. Despite the severity of this threat, there is limited research on the application and detection of poisoning attacks on object detection systems. This is especially problematic in the military domain, where attacks can have grave consequences. In this work, we both investigate the effect of poisoning attacks on military object detectors in practice, and the best approach to detect these attacks. To support this research, we create a small, custom dataset featuring military vehicles: MilCivVeh. We explore the vulnerability of military object detectors for poisoning attacks by implementing a modified version of the BadDet attack: a patch-based poisoning attack. We then assess its impact, finding that while a positive attack success rate is achievable, it requires a substantial portion of the data to be poisoned -- raising questions about its practical applicability. To address the detection challenge, we test both specialized poisoning detection methods and anomaly detection methods from the visual industrial inspection domain. Since our research shows that both classes of methods are lacking, we introduce our own patch detection method: AutoDetect, a simple, fast, and lightweight autoencoder-based method. Our method shows promising results in separating clean from poisoned samples using the reconstruction error of image slices, outperforming existing methods, while being less time- and memory-intensive. We urge that the availability of large, representative datasets in the military domain is a prerequisite to further evaluate risks of poisoning attacks and opportunities patch detection.

http://arxiv.org/abs/2509.02028
See No Evil: Adversarial Attacks Against Linguistic-Visual Association in Referring Multi-Object Tracking Systems. (31%)
Halima Bouzidi; Haoyu Liu; Mohammad Abdullah Al Faruque
Language-vision understanding has driven the development of advanced perception systems, most notably the emerging paradigm of Referring Multi-Object Tracking (RMOT). By leveraging natural-language queries, RMOT systems can selectively track objects that satisfy a given semantic description, guided through Transformer-based spatial-temporal reasoning modules. End-to-End (E2E) RMOT models further unify feature extraction, temporal memory, and spatial reasoning within a Transformer backbone, enabling long-range spatial-temporal modeling over fused textual-visual representations. Despite these advances, the reliability and robustness of RMOT remain underexplored. In this paper, we examine the security implications of RMOT systems from a design-logic perspective, identifying adversarial vulnerabilities that compromise both the linguistic-visual referring and track-object matching components. Additionally, we uncover a novel vulnerability in advanced RMOT models employing FIFO-based memory, whereby targeted and consistent attacks on their spatial-temporal reasoning introduce errors that persist within the history buffer over multiple subsequent frames. We present VEIL, a novel adversarial framework designed to disrupt the unified referring-matching mechanisms of RMOT models. We show that carefully crafted digital and physical perturbations can corrupt the tracking logic reliability, inducing track ID switches and terminations. We conduct comprehensive evaluations using the Refer-KITTI dataset to validate the effectiveness of VEIL and demonstrate the urgent need for security-aware RMOT designs for critical large-scale applications.

http://arxiv.org/abs/2509.08746
Stealth by Conformity: Evading Robust Aggregation through Adaptive Poisoning. (13%)
Ryan McGaughey; Rincon Jesus Martinez del; Ihsen Alouani
Federated Learning (FL) is a distributed learning paradigm designed to address privacy concerns. However, FL is vulnerable to poisoning attacks, where Byzantine clients compromise the integrity of the global model by submitting malicious updates. Robust aggregation methods have been widely adopted to mitigate such threats, relying on the core assumption that malicious updates are inherently out-of-distribution and can therefore be identified and excluded before aggregating client updates. In this paper, we challenge this underlying assumption by showing that a model can be poisoned while keeping malicious updates within the main distribution. We propose Chameleon Poisoning (CHAMP), an adaptive and evasive poisoning strategy that exploits side-channel feedback from the aggregation process to guide the attack. Specifically, the adversary continuously infers whether its malicious contribution has been incorporated into the global model and adapts accordingly. This enables a dynamic adjustment of the local loss function, balancing a malicious component with a camouflaging component, thereby increasing the effectiveness of the poisoning while evading robust aggregation defenses. CHAMP enables more effective and evasive poisoning, highlighting a fundamental limitation of existing robust aggregation defenses and underscoring the need for new strategies to secure federated learning against sophisticated adversaries. Our approach is evaluated in two datasets reaching an average increase of 47.07% in attack success rate against nine robust aggregation defenses.

http://arxiv.org/abs/2509.03108
Backdoor Poisoning Attack Against Face Spoofing Attack Detection Methods. (12%)
Shota Iwamatsu; Koichi Ito; Takafumi Aoki
Face recognition systems are robust against environmental changes and noise, and thus may be vulnerable to illegal authentication attempts using user face photos, such as spoofing attacks. To prevent such spoofing attacks, it is crucial to discriminate whether the input image is a live user image or a spoofed image prior to the face recognition process. Most existing spoofing attack detection methods utilize deep learning, which necessitates a substantial amount of training data. Consequently, if malicious data is injected into a portion of the training dataset, a specific spoofing attack may be erroneously classified as live, leading to false positives.In this paper, we propose a novel backdoor poisoning attack method to demonstrate the latent threat of backdoor poisoning within face anti-spoofing detection. The proposed method enables certain spoofing attacks to bypass detection by embedding features extracted from the spoofing attack's face image into a live face image without inducing any perceptible visual alterations.Through experiments conducted on public datasets, we demonstrate that the proposed method constitutes a realistic threat to existing spoofing attack detection systems.

http://arxiv.org/abs/2505.20357
Learning and Interpreting Gravitational-Wave Features from CNNs with a Random Forest Approach. (2%)
Jun Tian; He Wang; Jibo He; Yu Pan; Shuo Cao; Qingquan Jiang
Convolutional neural networks (CNNs) have become widely adopted in gravitational wave (GW) detection pipelines due to their ability to automatically learn hierarchical features from raw strain data. However, the physical meaning of these learned features remains underexplored, limiting the interpretability of such models. In this work, we propose a hybrid architecture that combines a CNN-based feature extractor with a random forest (RF) classifier to improve both detection performance and interpretability. Unlike prior approaches that directly connect classifiers to CNN outputs, our method introduces four physically interpretable metrics - variance, signal-to-noise ratio (SNR), waveform overlap, and peak amplitude - computed from the final convolutional layer. These are jointly used with the CNN output in the RF classifier to enable more informed decision boundaries. Tested on long-duration strain datasets, our hybrid model outperforms a baseline CNN model, achieving a relative improvement of 21\% in sensitivity at a fixed false alarm rate of 10 events per month. Notably, it also shows improved detection of low-SNR signals (SNR $\le$ 10), which are especially vulnerable to misclassification in noisy environments. Feature attribution via the RF model reveals that both CNN-extracted and handcrafted features contribute significantly to classification decisions, with learned variance and CNN outputs ranked among the most informative. These findings suggest that physically motivated post-processing of CNN feature maps can serve as a valuable tool for interpretable and efficient GW detection, bridging the gap between deep learning and domain knowledge.

http://arxiv.org/abs/2509.02042
Targeted Physical Evasion Attacks in the Near-Infrared Domain. (98%)
Pascal Zimmer; Simon Lachnit; Alexander Jan Zielinski; Ghassan Karame
A number of attacks rely on infrared light sources or heat-absorbing material to imperceptibly fool systems into misinterpreting visual input in various image recognition applications. However, almost all existing approaches can only mount untargeted attacks and require heavy optimizations due to the use-case-specific constraints, such as location and shape. In this paper, we propose a novel, stealthy, and cost-effective attack to generate both targeted and untargeted adversarial infrared perturbations. By projecting perturbations from a transparent film onto the target object with an off-the-shelf infrared flashlight, our approach is the first to reliably mount laser-free targeted attacks in the infrared domain. Extensive experiments on traffic signs in the digital and physical domains show that our approach is robust and yields higher attack success rates in various attack scenarios across bright lighting conditions, distances, and angles compared to prior work. Equally important, our attack is highly cost-effective, requiring less than US\$50 and a few tens of seconds for deployment. Finally, we propose a novel segmentation-based detection that thwarts our attack with an F1-score of up to 99%.

http://arxiv.org/abs/2509.02289
Passwords and FIDO2 Are Meant To Be Secret: A Practical Secure Authentication Channel for Web Browsers. (1%)
Anuj Gautam; Tarun Yadav; Garrett Smith; Kent Seamons; Scott Ruoti
Password managers provide significant security benefits to users. However, malicious client-side scripts and browser extensions can steal passwords after the manager has autofilled them into the web page. In this paper, we extend prior work by Stock and Johns, showing how password autofill can be hardened to prevent these local attacks. We implement our design in the Firefox browser and conduct experiments demonstrating that our defense successfully protects passwords from XSS attacks and malicious extensions. We also show that our implementation is compatible with 97% of the Alexa top 1000 websites. Next, we generalize our design, creating a second defense that prevents recently discovered local attacks against the FIDO2 protocols. We implement this second defense into Firefox, demonstrating that it protects the FIDO2 protocol against XSS attacks and malicious extensions. This defense is compatible with all websites, though it does require a small change (2-3 lines) to web servers implementing FIDO2.

http://arxiv.org/abs/2509.02863
Enhancing Machine Learning for Imbalanced Medical Data: A Quantum-Inspired Approach to Synthetic Oversampling (QI-SMOTE). (1%)
Vikas Kashtriya; Pardeep Singh
Class imbalance remains a critical challenge in machine learning (ML), particularly in the medical domain, where underrepresented minority classes lead to biased models and reduced predictive performance. This study introduces Quantum-Inspired SMOTE (QI-SMOTE), a novel data augmentation technique that enhances the performance of ML classifiers, including Random Forest (RF), Support Vector Machine (SVM), Logistic Regression (LR), k-Nearest Neighbors (KNN), Gradient Boosting (GB), and Neural Networks, by leveraging quantum principles such as quantum evolution and layered entanglement. Unlike conventional oversampling methods, QI-SMOTE generates synthetic instances that preserve complex data structures, improving model generalization and classification accuracy. We validate QI-SMOTE on the MIMIC-III and MIMIC-IV datasets, using mortality detection as a benchmark task due to their clinical significance and inherent class imbalance. We compare our method against traditional oversampling techniques, including Borderline-SMOTE, ADASYN, SMOTE-ENN, SMOTE-TOMEK, and SVM-SMOTE, using key performance metrics such as Accuracy, F1-score, G-Mean, and AUC-ROC. The results demonstrate that QI-SMOTE significantly improves the effectiveness of ensemble methods (RF, GB, ADA), kernel-based models (SVM), and deep learning approaches by producing more informative and balanced training data. By integrating quantum-inspired transformations into the ML pipeline, QI-SMOTE not only mitigates class imbalance but also enhances the robustness and reliability of predictive models in medical diagnostics and decision-making. This study highlights the potential of quantum-inspired resampling techniques in advancing state-of-the-art ML methodologies.

http://arxiv.org/abs/2501.12123
FL-CLEANER: byzantine and backdoor defense by CLustering Errors of Activation maps in Non-iid fedErated leaRning. (47%)
Mehdi Ben Ghali; Gouenou Coatrieux; Reda Bellafqira
Federated Learning (FL) enables clients to collaboratively train a global model using their local datasets while reinforcing data privacy, but it is prone to poisoning attacks. Existing defense mechanisms assume that clients' data are independent and identically distributed (IID), making them ineffective in real-world applications where data are non-IID. This paper presents FL-CLEANER, the first defense capable of filtering both byzantine and backdoor attackers' model updates in a non-IID FL environment. The originality of FL-CLEANER is twofold. First, it relies on a client confidence score derived from the reconstruction errors of each client's model activation maps for a given trigger set, with reconstruction errors obtained by means of a Conditional Variational Autoencoder trained according to a novel server-side strategy. Second, it uses an original ad-hoc trust propagation algorithm we propose. Based on previous client scores, it allows building a cluster of benign clients while flagging potential attackers. Experimental results on the datasets MNIST and FashionMNIST demonstrate the efficiency of FL-CLEANER against Byzantine attackers as well as to some state-of-the-art backdoors in non-IID scenarios; it achieves a close-to-zero (<1%) benign client misclassification rate, even in the absence of an attack, and achieves strong performance compared to state of the art defenses.

http://arxiv.org/abs/2509.01235
Geometric origin of adversarial vulnerability in deep learning. (33%)
Yixiong Ren; Wenkang Du; Jianhui Zhou; Haiping Huang
How to balance training accuracy and adversarial robustness has become a challenge since the birth of deep learning. Here, we introduce a geometry-aware deep learning framework that leverages layer-wise local training to sculpt the internal representations of deep neural networks. This framework promotes intra-class compactness and inter-class separation in feature space, leading to manifold smoothness and adversarial robustness against white or black box attacks. The performance can be explained by an energy model with Hebbian coupling between elements of the hidden representation. Our results thus shed light on the physics of learning in the direction of alignment between biological and artificial intelligence systems. Using the current framework, the deep network can assimilate new information into existing knowledge structures while reducing representation interference.

http://arxiv.org/abs/2509.01631
Unraveling LLM Jailbreaks Through Safety Knowledge Neurons. (13%)
Chongwen Zhao; Kaizhu Huang
Large Language Models (LLMs) are increasingly attracting attention in various applications. Nonetheless, there is a growing concern as some users attempt to exploit these models for malicious purposes, including the synthesis of controlled substances and the propagation of disinformation, a technique known as "Jailbreak." While some studies have achieved defenses against jailbreak attacks by modifying output distributions or detecting harmful content, the exact rationale still remains elusive. In this work, we present a novel neuron-level interpretability method that focuses on the role of safety-related knowledge neurons. Unlike existing approaches, our method projects the model's internal representation into a more consistent and interpretable vocabulary space. We then show that adjusting the activation of safety-related neurons can effectively control the model's behavior with a mean ASR higher than 97%. Building on this insight, we propose SafeTuning, a fine-tuning strategy that reinforces safety-critical neurons to improve model robustness against jailbreaks. SafeTuning consistently reduces attack success rates across multiple LLMs and outperforms all four baseline defenses. These findings offer a new perspective on understanding and defending against jailbreak attacks.

http://arxiv.org/abs/2508.17481
SoK: Cybersecurity Assessment of Humanoid Ecosystem. (12%)
Priyanka Prakash Surve; Asaf Shabtai; Yuval Elovici
Humanoids are progressing toward practical deployment across healthcare, industrial, defense, and service sectors. While typically considered cyber-physical systems (CPSs), their dependence on traditional networked software stacks (e.g., Linux operating systems), robot operating system (ROS) middleware, and over-the-air update channels, creates a distinct security profile that exposes them to vulnerabilities conventional CPS models do not fully address. Prior studies have mainly examined specific threats, such as LiDAR spoofing or adversarial machine learning (AML). This narrow focus overlooks how an attack targeting one component can cascade harm throughout the robot's interconnected systems. We address this gap through a systematization of knowledge (SoK) that takes a comprehensive approach, consolidating fragmented research from robotics, CPS, and network security domains. We introduce a seven-layer security model for humanoid robots, organizing 39 known attacks and 35 defenses across the humanoid ecosystem-from hardware to human-robot interaction. Building on this security model, we develop a quantitative 39x35 attack-defense matrix with risk-weighted scoring, validated through Monte Carlo analysis. We demonstrate our method by evaluating three real-world robots: Pepper, G1 EDU, and Digit. The scoring analysis revealed varying security maturity levels, with scores ranging from 39.9% to 79.5% across the platforms. This work introduces a structured, evidence-based assessment method that enables systematic security evaluation, supports cross-platform benchmarking, and guides prioritization of security investments in humanoid robotics.

http://arxiv.org/abs/2509.01444
Strata-Sword: A Hierarchical Safety Evaluation towards LLMs based on Reasoning Complexity of Jailbreak Instructions. (2%)
Shiji Zhao; Ranjie Duan; Jiexi Liu; Xiaojun Jia; Fengxiang Wang; Cheng Wei; Ruoxi Cheng; Yong Xie; Chang Liu; Qing Guo; Jialing Tao; Hui Xue; Xingxing Wei
Large language models (LLMs) have gained widespread recognition for their superior comprehension and have been deployed across numerous domains. Building on Chain-of-Thought (CoT) ideology, Large Reasoning models (LRMs) further exhibit strong reasoning skills, enabling them to infer user intent more accurately and respond appropriately. However, both LLMs and LRMs face the potential safety risks under jailbreak attacks, which raise concerns about their safety capabilities. Current safety evaluation methods often focus on the content dimensions, or simply aggregate different attack methods, lacking consideration of the complexity. In fact, instructions of different complexity can reflect the different safety capabilities of the model: simple instructions can reflect the basic values of the model, while complex instructions can reflect the model's ability to deal with deeper safety risks. Therefore, a comprehensive benchmark needs to be established to evaluate the safety performance of the model in the face of instructions of varying complexity, which can provide a better understanding of the safety boundaries of the LLMs. Thus, this paper first quantifies "Reasoning Complexity" as an evaluable safety dimension and categorizes 15 jailbreak attack methods into three different levels according to the reasoning complexity, establishing a hierarchical Chinese-English jailbreak safety benchmark for systematically evaluating the safety performance of LLMs. Meanwhile, to fully utilize unique language characteristics, we first propose some Chinese jailbreak attack methods, including the Chinese Character Disassembly attack, Lantern Riddle attack, and Acrostic Poem attack. A series of experiments indicate that current LLMs and LRMs show different safety boundaries under different reasoning complexity, which provides a new perspective to develop safer LLMs and LRMs.

http://arxiv.org/abs/2509.00826
Sequential Difference Maximization: Generating Adversarial Examples via Multi-Stage Optimization. (99%)
Xinlei Liu; Tao Hu; Peng Yi; Weitao Han; Jichao Xie; Baolin Li
Efficient adversarial attack methods are critical for assessing the robustness of computer vision models. In this paper, we reconstruct the optimization objective for generating adversarial examples as "maximizing the difference between the non-true labels' probability upper bound and the true label's probability," and propose a gradient-based attack method termed Sequential Difference Maximization (SDM). SDM establishes a three-layer optimization framework of "cycle-stage-step." The processes between cycles and between iterative steps are respectively identical, while optimization stages differ in terms of loss functions: in the initial stage, the negative probability of the true label is used as the loss function to compress the solution space; in subsequent stages, we introduce the Directional Probability Difference Ratio (DPDR) loss function to gradually increase the non-true labels' probability upper bound by compressing the irrelevant labels' probabilities. Experiments demonstrate that compared with previous SOTA methods, SDM not only exhibits stronger attack performance but also achieves higher attack cost-effectiveness. Additionally, SDM can be combined with adversarial training methods to enhance their defensive effects. The code is available at https://github.com/X-L-Liu/SDM.

http://arxiv.org/abs/2509.05332
Integrated Simulation Framework for Adversarial Attacks on Autonomous Vehicles. (45%)
Christos Anagnostopoulos; Ioulia Kapsali; Alexandros Gkillas; Nikos Piperigkos; Aris S. Lalos
Autonomous vehicles (AVs) rely on complex perception and communication systems, making them vulnerable to adversarial attacks that can compromise safety. While simulation offers a scalable and safe environment for robustness testing, existing frameworks typically lack comprehensive supportfor modeling multi-domain adversarial scenarios. This paper introduces a novel, open-source integrated simulation framework designed to generate adversarial attacks targeting both perception and communication layers of AVs. The framework provides high-fidelity modeling of physical environments, traffic dynamics, and V2X networking, orchestrating these components through a unified core that synchronizes multiple simulators based on a single configuration file. Our implementation supports diverse perception-level attacks on LiDAR sensor data, along with communication-level threats such as V2X message manipulation and GPS spoofing. Furthermore, ROS 2 integration ensures seamless compatibility with third-party AV software stacks. We demonstrate the framework's effectiveness by evaluating the impact of generated adversarial scenarios on a state-of-the-art 3D object detector, revealing significant performance degradation under realistic conditions.

http://arxiv.org/abs/2509.00973
Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation. (9%)
Kanchon Gharami; Hansaka Aluvihare; Shafika Showkat Moni; Berker Peköz
Large Language Models (LLMs) are increasingly deployed in mission-critical systems, facilitating tasks such as satellite operations, command-and-control, military decision support, and cyber defense. Many of these systems are accessed through application programming interfaces (APIs). When such APIs lack robust access controls, they can expose full or top-k logits, creating a significant and often overlooked attack surface. Prior art has mainly focused on reconstructing the output projection layer or distilling surface-level behaviors. However, regenerating a black-box model under tight query constraints remains underexplored. We address that gap by introducing a constrained replication pipeline that transforms partial logit leakage into a functional deployable substitute model clone. Our two-stage approach (i) reconstructs the output projection matrix by collecting top-k logits from under 10k black-box queries via singular value decomposition (SVD) over the logits, then (ii) distills the remaining architecture into compact student models with varying transformer depths, trained on an open source dataset. A 6-layer student recreates 97.6% of the 6-layer teacher model's hidden-state geometry, with only a 7.31% perplexity increase, and a 7.58 Negative Log-Likelihood (NLL). A 4-layer variant achieves 17.1% faster inference and 18.1% parameter reduction with comparable performance. The entire attack completes in under 24 graphics processing unit (GPU) hours and avoids triggering API rate-limit defenses. These results demonstrate how quickly a cost-limited adversary can clone an LLM, underscoring the urgent need for hardened inference APIs and secure on-premise defense deployments.

http://arxiv.org/abs/2508.20817
FusionCounting: Robust visible-infrared image fusion guided by crowd counting via multi-task learning. (3%)
He Li; Xinyu Liu; Weihang Kong; Xingchen Zhang
Visible and infrared image fusion (VIF) is an important multimedia task in computer vision. Most VIF methods focus primarily on optimizing fused image quality. Recent studies have begun incorporating downstream tasks, such as semantic segmentation and object detection, to provide semantic guidance for VIF. However, semantic segmentation requires extensive annotations, while object detection, despite reducing annotation efforts compared with segmentation, faces challenges in highly crowded scenes due to overlapping bounding boxes and occlusion. Moreover, although RGB-T crowd counting has gained increasing attention in recent years, no studies have integrated VIF and crowd counting into a unified framework. To address these challenges, we propose FusionCounting, a novel multi-task learning framework that integrates crowd counting into the VIF process. Crowd counting provides a direct quantitative measure of population density with minimal annotation, making it particularly suitable for dense scenes. Our framework leverages both input images and population density information in a mutually beneficial multi-task design. To accelerate convergence and balance tasks contributions, we introduce a dynamic loss function weighting strategy. Furthermore, we incorporate adversarial training to enhance the robustness of both VIF and crowd counting, improving the model's stability and resilience to adversarial attacks. Experimental results on public datasets demonstrate that FusionCounting not only enhances image fusion quality but also achieves superior crowd counting performance.

http://arxiv.org/abs/2509.00387
Unifying Adversarial Perturbation for Graph Neural Networks. (96%)
Jinluan Yang; Ruihao Zhang; Zhengyu Chen; Fei Wu; Kun Kuang
This paper studies the vulnerability of Graph Neural Networks (GNNs) to adversarial attacks on node features and graph structure. Various methods have implemented adversarial training to augment graph data, aiming to bolster the robustness and generalization of GNNs. These methods typically involve applying perturbations to the node feature, weights, or graph structure and subsequently minimizing the loss by learning more robust graph model parameters under the adversarial perturbations. Despite the effectiveness of adversarial training in enhancing GNNs' robustness and generalization abilities, its application has been largely confined to specific datasets and GNN types. In this paper, we propose a novel method, PerturbEmbedding, that integrates adversarial perturbation and training, enhancing GNNs' resilience to such attacks and improving their generalization ability. PerturbEmbedding performs perturbation operations directly on every hidden embedding of GNNs and provides a unified framework for most existing perturbation strategies/methods. We also offer a unified perspective on the forms of perturbations, namely random and adversarial perturbations. Through experiments on various datasets using different backbone models, we demonstrate that PerturbEmbedding significantly improves both the robustness and generalization abilities of GNNs, outperforming existing methods. The rejection of both random (non-targeted) and adversarial (targeted) perturbations further enhances the backbone model's performance.

http://arxiv.org/abs/2509.00373
Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models. (81%)
Sihao Wu; Gaojie Jin; Wei Huang; Jianhong Wang; Xiaowei Huang
Vision Language Models (VLMs) have demonstrated impressive capabilities in integrating visual and textual information for understanding and reasoning, but remain highly vulnerable to adversarial attacks. While activation steering has emerged as a promising defence, existing approaches often rely on task-specific contrastive prompts to extract harmful directions, which exhibit suboptimal performance and can degrade visual grounding performance. To address these limitations, we propose \textit{Sequence-Level Preference Optimization} for VLM (\textit{SPO-VLM}), a novel two-stage defense framework that combines activation-level intervention with policy-level optimization to enhance model robustness. In \textit{Stage I}, we compute adaptive layer-specific steering vectors from diverse data sources, enabling generalized suppression of harmful behaviors during inference. In \textit{Stage II}, we refine these steering vectors through a sequence-level preference optimization process. This stage integrates automated toxicity assessment, as well as visual-consistency rewards based on caption-image alignment, to achieve safe and semantically grounded text generation. The two-stage structure of SPO-VLM balances efficiency and effectiveness by combining a lightweight mitigation foundation in Stage I with deeper policy refinement in Stage II. Extensive experiments shown SPO-VLM enhances safety against attacks via activation steering and preference optimization, while maintaining strong performance on benign tasks without compromising visual understanding capabilities. We will release our code, model weights, and evaluation toolkit to support reproducibility and future research. \textcolor{red}{Warning: This paper may contain examples of offensive or harmful text and images.}

http://arxiv.org/abs/2509.00561
FreeTalk:A plug-and-play and black-box defense against speech synthesis attacks. (75%)
Yuwen Pu; Zhou Feng; Chunyi Zhou; Jiahao Chen; Chunqiang Hu; Haibo Hu; Shouling Ji
Recently, speech assistant and speech verification have been used in many fields, which brings much benefit and convenience for us. However, when we enjoy these speech applications, our speech may be collected by attackers for speech synthesis. For example, an attacker generates some inappropriate political opinions with the characteristic of the victim's voice by obtaining a piece of the victim's speech, which will greatly influence the victim's reputation. Specifically, with the appearance of some zero-shot voice conversion methods, the cost of speech synthesis attacks has been further reduced, which also brings greater challenges to user voice security and privacy. Some researchers have proposed the corresponding privacy-preserving methods. However, the existing approaches have some non-negligible drawbacks: low transferability and robustness, high computational overhead. These deficiencies seriously limit the existing method deployed in practical scenarios. Therefore, in this paper, we propose a lightweight, robust, plug-and-play privacy preservation method against speech synthesis attacks in a black-box setting. Our method generates and adds a frequency-domain perturbation to the original speech to achieve privacy protection and high speech quality. Then, we present a data augmentation strategy and noise smoothing mechanism to improve the robustness of the proposed method. Besides, to reduce the user's defense overhead, we also propose a novel identity-wise protection mechanism. It can generate a universal perturbation for one speaker and support privacy preservation for speech of any length. Finally, we conduct extensive experiments on 5 speech synthesis models, 5 speech verification models, 1 speech recognition model, and 2 datasets. The experimental results demonstrate that our method has satisfying privacy-preserving performance, high speech quality, and utility.

http://arxiv.org/abs/2509.05318
Backdoor Samples Detection Based on Perturbation Discrepancy Consistency in Pre-trained Language Models. (13%)
Zuquan Peng; Jianming Fu; Lixin Zou; Li Zheng; Yanzhen Ren; Guojun Peng
The use of unvetted third-party and internet data renders pre-trained models susceptible to backdoor attacks. Detecting backdoor samples is critical to prevent backdoor activation during inference or injection during training. However, existing detection methods often require the defender to have access to the poisoned models, extra clean samples, or significant computational resources to detect backdoor samples, limiting their practicality. To address this limitation, we propose a backdoor sample detection method based on perturbatio\textbf{N} discr\textbf{E}pancy consis\textbf{T}ency \textbf{E}valuation (\NETE). This is a novel detection method that can be used both pre-training and post-training phases. In the detection process, it only requires an off-the-shelf pre-trained model to compute the log probability of samples and an automated function based on a mask-filling strategy to generate perturbations. Our method is based on the interesting phenomenon that the change in perturbation discrepancy for backdoor samples is smaller than that for clean samples. Based on this phenomenon, we use curvature to measure the discrepancy in log probabilities between different perturbed samples and input samples, thereby evaluating the consistency of the perturbation discrepancy to determine whether the input sample is a backdoor sample. Experiments conducted on four typical backdoor attacks and five types of large language model backdoor attacks demonstrate that our detection strategy outperforms existing zero-shot black-box detection methods.

http://arxiv.org/abs/2509.00391
The Resurgence of GCG Adversarial Attacks on Large Language Models. (10%)
Yuting Tan; Xuying Li; Zhuo Li; Huizhen Shu; Peikang Hu
Gradient-based adversarial prompting, such as the Greedy Coordinate Gradient (GCG) algorithm, has emerged as a powerful method for jailbreaking large language models (LLMs). In this paper, we present a systematic appraisal of GCG and its annealing-augmented variant, T-GCG, across open-source LLMs of varying scales. Using Qwen2.5-0.5B, LLaMA-3.2-1B, and GPT-OSS-20B, we evaluate attack effectiveness on both safety-oriented prompts (AdvBench) and reasoning-intensive coding prompts. Our study reveals three key findings: (1) attack success rates (ASR) decrease with model size, reflecting the increasing complexity and non-convexity of larger models' loss landscapes; (2) prefix-based heuristics substantially overestimate attack effectiveness compared to GPT-4o semantic judgments, which provide a stricter and more realistic evaluation; and (3) coding-related prompts are significantly more vulnerable than adversarial safety prompts, suggesting that reasoning itself can be exploited as an attack vector. In addition, preliminary results with T-GCG show that simulated annealing can diversify adversarial search and achieve competitive ASR under prefix evaluation, though its benefits under semantic judgment remain limited. Together, these findings highlight the scalability limits of GCG, expose overlooked vulnerabilities in reasoning tasks, and motivate further development of annealing-inspired strategies for more robust adversarial evaluation.

http://arxiv.org/abs/2509.00633
On the Thermal Vulnerability of 3D-Stacked High-Bandwidth Memory Architectures. (1%)
Mehdi Elahi; Mohamed R. Elshamy; Abdel-Hameed A. Badawy; Ahmad Patooghy
3D-stacked High Bandwidth Memory (HBM) architectures provide high-performance memory interactions to address the well-known performance challenge, namely the memory wall. However, these architectures are susceptible to thermal vulnerabilities due to the inherent vertical adjacency that occurs during the manufacturing process of HBM architectures. We anticipate that adversaries may exploit the intense vertical and lateral adjacency to design and develop thermal performance degradation attacks on the memory banks that host data/instructions from victim applications. In such attacks, the adversary manages to inject short and intense heat pulses from vertically and/or laterally adjacent memory banks, creating a convergent thermal wave that maximizes impact and delays the victim application from accessing its data/instructions. As the attacking application does not access any out-of-range memory locations, it can bypass both design-time security tests and the operating system's memory management policies. In other words, since the attack mimics legitimate workloads, it will be challenging to detect.

http://arxiv.org/abs/2508.21472
Adversarial Patch Attack for Ship Detection via Localized Augmentation. (99%)
Chun Liu; Panpan Ding; Zheng Zheng; Hailong Wang; Bingqian Zhu; Tao Xu; Zhigang Han; Jiayao Wang
Current ship detection techniques based on remote sensing imagery primarily rely on the object detection capabilities of deep neural networks (DNNs). However, DNNs are vulnerable to adversarial patch attacks, which can lead to misclassification by the detection model or complete evasion of the targets. Numerous studies have demonstrated that data transformation-based methods can improve the transferability of adversarial examples. However, excessive augmentation of image backgrounds or irrelevant regions may introduce unnecessary interference, resulting in false detections of the object detection model. These errors are not caused by the adversarial patches themselves but rather by the over-augmentation of background and non-target areas. This paper proposes a localized augmentation method that applies augmentation only to the target regions, avoiding any influence on non-target areas. By reducing background interference, this approach enables the loss function to focus more directly on the impact of the adversarial patch on the detection model, thereby improving the attack success rate. Experiments conducted on the HRSC2016 dataset demonstrate that the proposed method effectively increases the success rate of adversarial patch attacks and enhances their transferability.

http://arxiv.org/abs/2504.08897
On the Adversarial Robustness of Spiking Neural Networks Trained by Local Learning. (99%)
Jiaqi Lin; Abhronil Sengupta
Recent research has shown the vulnerability of Spiking Neural Networks (SNNs) under adversarial examples that are nearly indistinguishable from clean data in the context of frame-based and event-based information. The majority of these studies are constrained in generating adversarial examples using Backpropagation Through Time (BPTT), a gradient-based method which lacks biological plausibility. In contrast, local learning methods, which relax many of BPTT's constraints, remain under-explored in the context of adversarial attacks. To address this problem, we examine adversarial robustness in SNNs through the framework of four types of training algorithms. We provide an in-depth analysis of the ineffectiveness of gradient-based adversarial attacks to generate adversarial instances in this scenario. To overcome these limitations, we introduce a hybrid adversarial attack paradigm that leverages the transferability of adversarial instances. The proposed hybrid approach demonstrates superior performance, outperforming existing adversarial attack methods. Furthermore, the generalizability of the method is assessed under multi-step adversarial attacks, adversarial attacks in black-box FGSM scenarios, and within the non-spiking domain.

http://arxiv.org/abs/2508.21715
Entropy-Based Non-Invasive Reliability Monitoring of Convolutional Neural Networks. (93%)
Amirhossein Nazeri; Wael Hafez
Convolutional Neural Networks (CNNs) have become the foundation of modern computer vision, achieving unprecedented accuracy across diverse image recognition tasks. While these networks excel on in-distribution data, they remain vulnerable to adversarial perturbations imperceptible input modifications that cause misclassification with high confidence. However, existing detection methods either require expensive retraining, modify network architecture, or degrade performance on clean inputs. Here we show that adversarial perturbations create immediate, detectable entropy signatures in CNN activations that can be monitored without any model modification. Using parallel entropy monitoring on VGG-16, we demonstrate that adversarial inputs consistently shift activation entropy by 7% in early convolutional layers, enabling 90% detection accuracy with false positives and false negative rates below 20%. The complete separation between clean and adversarial entropy distributions reveals that CNNs inherently encode distribution shifts in their activation patterns. This work establishes that CNN reliability can be assessed through activation entropy alone, enabling practical deployment of self-diagnostic vision systems that detect adversarial inputs in real-time without compromising original model performance.

http://arxiv.org/abs/2508.20863
Publish to Perish: Prompt Injection Attacks on LLM-Assisted Peer Review. (41%)
Matteo Gioele Collu; Umberto Salviati; Roberto Confalonieri; Mauro Conti; Giovanni Apruzzese
Large Language Models (LLMs) are increasingly being integrated into the scientific peer-review process, raising new questions about their reliability and resilience to manipulation. In this work, we investigate the potential for hidden prompt injection attacks, where authors embed adversarial text within a paper's PDF to influence the LLM-generated review. We begin by formalising three distinct threat models that envision attackers with different motivations -- not all of which implying malicious intent. For each threat model, we design adversarial prompts that remain invisible to human readers yet can steer an LLM's output toward the author's desired outcome. Using a user study with domain scholars, we derive four representative reviewing prompts used to elicit peer reviews from LLMs. We then evaluate the robustness of our adversarial prompts across (i) different reviewing prompts, (ii) different commercial LLM-based systems, and (iii) different peer-reviewed papers. Our results show that adversarial prompts can reliably mislead the LLM, sometimes in ways that adversely affect a "honest-but-lazy" reviewer. Finally, we propose and empirically assess methods to reduce detectability of adversarial prompts under automated content checks.

http://arxiv.org/abs/2508.21654
I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks. (38%)
Daryna Oliynyk; Rudolf Mayer; Kathrin Grosse; Andreas Rauber
Model stealing attacks endanger the confidentiality of machine learning models offered as a service. Although these models are kept secret, a malicious party can query a model to label data samples and train their own substitute model, violating intellectual property. While novel attacks in the field are continually being published, their design and evaluations are not standardised, making it challenging to compare prior works and assess progress in the field. This paper is the first to address this gap by providing recommendations for designing and evaluating model stealing attacks. To this end, we study the largest group of attacks that rely on training a substitute model -- those attacking image classification models. We propose the first comprehensive threat model and develop a framework for attack comparison. Further, we analyse attack setups from related works to understand which tasks and models have been studied the most. Based on our findings, we present best practices for attack development before, during, and beyond experiments and derive an extensive list of open research questions regarding the evaluation of model stealing attacks. Our findings and recommendations also transfer to other problem domains, hence establishing the first generic evaluation methodology for model stealing attacks.

http://arxiv.org/abs/2508.21636
Detecting Stealthy Data Poisoning Attacks in AI Code Generators. (13%)
Cristina Improta
Deep learning (DL) models for natural language-to-code generation have become integral to modern software development pipelines. However, their heavy reliance on large amounts of data, often collected from unsanitized online sources, exposes them to data poisoning attacks, where adversaries inject malicious samples to subtly bias model behavior. Recent targeted attacks silently replace secure code with semantically equivalent but vulnerable implementations without relying on explicit triggers to launch the attack, making it especially hard for detection methods to distinguish clean from poisoned samples. We present a systematic study on the effectiveness of existing poisoning detection methods under this stealthy threat model. Specifically, we perform targeted poisoning on three DL models (CodeBERT, CodeT5+, AST-T5), and evaluate spectral signatures analysis, activation clustering, and static analysis as defenses. Our results show that all methods struggle to detect triggerless poisoning, with representation-based approaches failing to isolate poisoned samples and static analysis suffering false positives and false negatives, highlighting the need for more robust, trigger-independent defenses for AI-assisted code generation.

http://arxiv.org/abs/2508.21816
The Demon is in Ambiguity: Revisiting Situation Recognition with Single Positive Multi-Label Learning. (1%)
Yiming Lin; Yuchen Niu; Shang Wang; Kaizhu Huang; Qiufeng Wang; Xiao-Bo Jin
Context recognition (SR) is a fundamental task in computer vision that aims to extract structured semantic summaries from images by identifying key events and their associated entities. Specifically, given an input image, the model must first classify the main visual events (verb classification), then identify the participating entities and their semantic roles (semantic role labeling), and finally localize these entities in the image (semantic role localization). Existing methods treat verb classification as a single-label problem, but we show through a comprehensive analysis that this formulation fails to address the inherent ambiguity in visual event recognition, as multiple verb categories may reasonably describe the same image. This paper makes three key contributions: First, we reveal through empirical analysis that verb classification is inherently a multi-label problem due to the ubiquitous semantic overlap between verb categories. Second, given the impracticality of fully annotating large-scale datasets with multiple labels, we propose to reformulate verb classification as a single positive multi-label learning (SPMLL) problem - a novel perspective in SR research. Third, we design a comprehensive multi-label evaluation benchmark for SR that is carefully designed to fairly evaluate model performance in a multi-label setting. To address the challenges of SPMLL, we futher develop the Graph Enhanced Verb Multilayer Perceptron (GE-VerbMLP), which combines graph neural networks to capture label correlations and adversarial training to optimize decision boundaries. Extensive experiments on real-world datasets show that our approach achieves more than 3\% MAP improvement while remaining competitive on traditional top-1 and top-5 accuracy metrics.

http://arxiv.org/abs/2508.21440
Time Tells All: Deanonymization of Blockchain RPC Users with Zero Transaction Fee (Extended Version). (1%)
Shan Wang; Ming Yang; Yu Liu; Yue Zhang; Shuaiqing Zhang; Zhen Ling; Jiannong Cao; Xinwen Fu
Remote Procedure Call (RPC) services have become a primary gateway for users to access public blockchains. While they offer significant convenience, RPC services also introduce critical privacy challenges that remain insufficiently examined. Existing deanonymization attacks either do not apply to blockchain RPC users or incur costs like transaction fees assuming an active network eavesdropper. In this paper, we propose a novel deanonymization attack that can link an IP address of a RPC user to this user's blockchain pseudonym. Our analysis reveals a temporal correlation between the timestamps of transaction confirmations recorded on the public ledger and those of TCP packets sent by the victim when querying transaction status. We assume a strong passive adversary with access to network infrastructure, capable of monitoring traffic at network border routers or Internet exchange points. By monitoring network traffic and analyzing public ledgers, the attacker can link the IP address of the TCP packet to the pseudonym of the transaction initiator by exploiting the temporal correlation. This deanonymization attack incurs zero transaction fee. We mathematically model and analyze the attack method, perform large-scale measurements of blockchain ledgers, and conduct real-world attacks to validate the attack. Our attack achieves a high success rate of over 95% against normal RPC users on various blockchain networks, including Ethereum, Bitcoin and Solana.

http://arxiv.org/abs/2508.21072
First-Place Solution to NeurIPS 2024 Invisible Watermark Removal Challenge. (74%)
Fahad Shamshad; Tameem Bakr; Yahia Shaaban; Noor Hussein; Karthik Nandakumar; Nils Lukas
Content watermarking is an important tool for the authentication and copyright protection of digital media. However, it is unclear whether existing watermarks are robust against adversarial attacks. We present the winning solution to the NeurIPS 2024 Erasing the Invisible challenge, which stress-tests watermark robustness under varying degrees of adversary knowledge. The challenge consisted of two tracks: a black-box and beige-box track, depending on whether the adversary knows which watermarking method was used by the provider. For the beige-box track, we leverage an adaptive VAE-based evasion attack, with a test-time optimization and color-contrast restoration in CIELAB space to preserve the image's quality. For the black-box track, we first cluster images based on their artifacts in the spatial or frequency-domain. Then, we apply image-to-image diffusion models with controlled noise injection and semantic priors from ChatGPT-generated captions to each cluster with optimized parameter settings. Empirical evaluations demonstrate that our method successfully achieves near-perfect watermark removal (95.7%) with negligible impact on the residual image's quality. We hope that our attacks inspire the development of more robust image watermarking methods.

http://arxiv.org/abs/2508.20595
Disruptive Attacks on Face Swapping via Low-Frequency Perceptual Perturbations. (54%)
Mengxiao Huang; Minglei Shu; Shuwang Zhou; Zhaoyang Liu
Deepfake technology, driven by Generative Adversarial Networks (GANs), poses significant risks to privacy and societal security. Existing detection methods are predominantly passive, focusing on post-event analysis without preventing attacks. To address this, we propose an active defense method based on low-frequency perceptual perturbations to disrupt face swapping manipulation, reducing the performance and naturalness of generated content. Unlike prior approaches that used low-frequency perturbations to impact classification accuracy,our method directly targets the generative process of deepfake techniques. We combine frequency and spatial domain features to strengthen defenses. By introducing artifacts through low-frequency perturbations while preserving high-frequency details, we ensure the output remains visually plausible. Additionally, we design a complete architecture featuring an encoder, a perturbation generator, and a decoder, leveraging discrete wavelet transform (DWT) to extract low-frequency components and generate perturbations that disrupt facial manipulation models. Experiments on CelebA-HQ and LFW demonstrate significant reductions in face-swapping effectiveness, improved defense success rates, and preservation of visual quality.

http://arxiv.org/abs/2508.21004
Lethe: Purifying Backdoored Large Language Models with Knowledge Dilution. (38%)
Chen Chen; Yuchen Sun; Jiaxin Gao; Xueluan Gong; Qian Wang; Ziyao Wang; Yongsen Zheng; Kwok-Yan Lam
Large language models (LLMs) have seen significant advancements, achieving superior performance in various Natural Language Processing (NLP) tasks. However, they remain vulnerable to backdoor attacks, where models behave normally for standard queries but generate harmful responses or unintended output when specific triggers are activated. Existing backdoor defenses either lack comprehensiveness, focusing on narrow trigger settings, detection-only mechanisms, and limited domains, or fail to withstand advanced scenarios like model-editing-based, multi-trigger, and triggerless attacks. In this paper, we present LETHE, a novel method to eliminate backdoor behaviors from LLMs through knowledge dilution using both internal and external mechanisms. Internally, LETHE leverages a lightweight dataset to train a clean model, which is then merged with the backdoored model to neutralize malicious behaviors by diluting the backdoor impact within the model's parametric memory. Externally, LETHE incorporates benign and semantically relevant evidence into the prompt to distract LLM's attention from backdoor features. Experimental results on classification and generation domains across 5 widely used LLMs demonstrate that LETHE outperforms 8 state-of-the-art defense baselines against 8 backdoor attacks. LETHE reduces the attack success rate of advanced backdoor attacks by up to 98% while maintaining model utility. Furthermore, LETHE has proven to be cost-efficient and robust against adaptive backdoor attacks.

http://arxiv.org/abs/2508.21099
Safe-Control: A Safety Patch for Mitigating Unsafe Content in Text-to-Image Generation Models. (26%)
Xiangtao Meng; Yingkai Dong; Ning Yu; Li Wang; Zheng Li; Shanqing Guo
Despite the advancements in Text-to-Image (T2I) generation models, their potential for misuse or even abuse raises serious safety concerns. Model developers have made tremendous efforts to introduce safety mechanisms that can address these concerns in T2I models. However, the existing safety mechanisms, whether external or internal, either remain susceptible to evasion under distribution shifts or require extensive model-specific adjustments. To address these limitations, we introduce Safe-Control, an innovative plug-and-play safety patch designed to mitigate unsafe content generation in T2I models. Using data-driven strategies and safety-aware conditions, Safe-Control injects safety control signals into the locked T2I model, acting as an update in a patch-like manner. Model developers can also construct various safety patches to meet the evolving safety requirements, which can be flexibly merged into a single, unified patch. Its plug-and-play design further ensures adaptability, making it compatible with other T2I models of similar denoising architecture. We conduct extensive evaluations on six diverse and public T2I models. Empirical results highlight that Safe-Control is effective in reducing unsafe content generation across six diverse T2I models with similar generative architectures, yet it successfully maintains the quality and text alignment of benign images. Compared to seven state-of-the-art safety mechanisms, including both external and internal defenses, Safe-Control significantly outperforms all baselines in reducing unsafe content generation. For example, it reduces the probability of unsafe content generation to 7%, compared to approximately 20% for most baseline methods, under both unsafe prompts and the latest adversarial attacks.

http://arxiv.org/abs/2508.21219
The WASM Cloak: Evaluating Browser Fingerprinting Defenses Under WebAssembly based Obfuscation. (11%)
A H M Nazmus Sakib; Mahsin Bin Akram; Joseph Spracklen; Sahan Kalutarage; Raveen Wijewickrama; Igor Bilogrevic; Murtuza Jadliwala
Browser fingerprinting defenses have historically focused on detecting JavaScript(JS)-based tracking techniques. However, the widespread adoption of WebAssembly (WASM) introduces a potential blind spot, as adversaries can convert JS to WASM's low-level binary format to obfuscate malicious logic. This paper presents the first systematic evaluation of how such WASM-based obfuscation impacts the robustness of modern fingerprinting defenses. We develop an automated pipeline that translates real-world JS fingerprinting scripts into functional WASM-obfuscated variants and test them against two classes of defenses: state-of-the-art detectors in research literature and commercial, in-browser tools. Our findings reveal a notable divergence: detectors proposed in the research literature that rely on feature-based analysis of source code show moderate vulnerability, stemming from outdated datasets or a lack of WASM compatibility. In contrast, defenses such as browser extensions and native browser features remained completely effective, as their API-level interception is agnostic to the script's underlying implementation. These results highlight a gap between academic and practical defense strategies and offer insights into strengthening detection approaches against WASM-based obfuscation, while also revealing opportunities for more evasive techniques in future attacks.

http://arxiv.org/abs/2508.20412
MindGuard: Tracking, Detecting, and Attributing MCP Tool Poisoning Attack via Decision Dependence Graph. (8%)
Zhiqiang Wang; Junyang Zhang; Guanquan Shi; HaoRan Cheng; Yunhao Yao; Kaiwen Guo; Haohua Du; Xiang-Yang Li
The Model Context Protocol (MCP) is increasingly adopted to standardize the interaction between LLM agents and external tools. However, this trend introduces a new threat: Tool Poisoning Attacks (TPA), where tool metadata is poisoned to induce the agent to perform unauthorized operations. Existing defenses that primarily focus on behavior-level analysis are fundamentally ineffective against TPA, as poisoned tools need not be executed, leaving no behavioral trace to monitor.
  Thus, we propose MindGuard, a decision-level guardrail for LLM agents, providing provenance tracking of call decisions, policy-agnostic detection, and poisoning source attribution against TPA. While fully explaining LLM decision remains challenging, our empirical findings uncover a strong correlation between LLM attention mechanisms and tool invocation decisions. Therefore, we choose attention as an empirical signal for decision tracking and formalize this as the Decision Dependence Graph (DDG), which models the LLM's reasoning process as a weighted, directed graph where vertices represent logical concepts and edges quantify the attention-based dependencies. We further design robust DDG construction and graph-based anomaly analysis mechanisms that efficiently detect and attribute TPA attacks. Extensive experiments on real-world datasets demonstrate that MindGuard achieves 94\%-99\% average precision in detecting poisoned invocations, 95\%-100\% attribution accuracy, with processing times under one second and no additional token cost. Moreover, DDG can be viewed as an adaptation of the classical Program Dependence Graph (PDG), providing a solid foundation for applying traditional security policies at the decision level.

http://arxiv.org/abs/2508.20570
Towards Mechanistic Defenses Against Typographic Attacks in CLIP. (8%)
Lorenz Hufe; Constantin Venhoff; Maximilian Dreyer; Sebastian Lapuschkin; Wojciech Samek
Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token. Building on these insights, we introduce a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, our method improves performance by up to 19.6% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.

http://arxiv.org/abs/2508.21197
GCAV: A Global Concept Activation Vector Framework for Cross-Layer Consistency in Interpretability. (3%)
Zhenghao He; Sanchit Sinha; Guangzhi Xiong; Aidong Zhang
Concept Activation Vectors (CAVs) provide a powerful approach for interpreting deep neural networks by quantifying their sensitivity to human-defined concepts. However, when computed independently at different layers, CAVs often exhibit inconsistencies, making cross-layer comparisons unreliable. To address this issue, we propose the Global Concept Activation Vector (GCAV), a novel framework that unifies CAVs into a single, semantically consistent representation. Our method leverages contrastive learning to align concept representations across layers and employs an attention-based fusion mechanism to construct a globally integrated CAV. By doing so, our method significantly reduces the variance in TCAV scores while preserving concept relevance, ensuring more stable and reliable concept attributions. To evaluate the effectiveness of GCAV, we introduce Testing with Global Concept Activation Vectors (TGCAV) as a method to apply TCAV to GCAV-based representations. We conduct extensive experiments on multiple deep neural networks, demonstrating that our method effectively mitigates concept inconsistency across layers, enhances concept localization, and improves robustness against adversarial perturbations. By integrating cross-layer information into a coherent framework, our method offers a more comprehensive and interpretable understanding of how deep learning models encode human-defined concepts. Code and models are available at https://github.com/Zhenghao-He/GCAV.

http://arxiv.org/abs/2508.21019
POSE: Phased One-Step Adversarial Equilibrium for Video Diffusion Models. (1%)
Jiaxiang Cheng; Bing Ma; Xuhua Ren; Hongyi Jin; Kai Yu; Peng Zhang; Wenyue Li; Yuan Zhou; Tianxiang Zheng; Qinglin Lu
The field of video diffusion generation faces critical bottlenecks in sampling efficiency, especially for large-scale models and long sequences. Existing video acceleration methods adopt image-based techniques but suffer from fundamental limitations: they neither model the temporal coherence of video frames nor provide single-step distillation for large-scale video models. To bridge this gap, we propose POSE (Phased One-Step Equilibrium), a distillation framework that reduces the sampling steps of large-scale video diffusion models, enabling the generation of high-quality videos in a single step. POSE employs a carefully designed two-phase process to distill video models:(i) stability priming: a warm-up mechanism to stabilize adversarial distillation that adapts the high-quality trajectory of the one-step generator from high to low signal-to-noise ratio regimes, optimizing the video quality of single-step mappings near the endpoints of flow trajectories. (ii) unified adversarial equilibrium: a flexible self-adversarial distillation mechanism that promotes stable single-step adversarial training towards a Nash equilibrium within the Gaussian noise space, generating realistic single-step videos close to real videos. For conditional video generation, we propose (iii) conditional adversarial consistency, a method to improve both semantic consistency and frame consistency between conditional frames and generated frames. Comprehensive experiments demonstrate that POSE outperforms other acceleration methods on VBench-I2V by average 7.15% in semantic alignment, temporal conference and frame quality, reducing the latency of the pre-trained model by 100$\times$, from 1000 seconds to 10 seconds, while maintaining competitive performance.

http://arxiv.org/abs/2508.20310
Differentially Private Federated Quantum Learning via Quantum Noise. (99%)
Atit Pokharel; Ratun Rahman; Shaba Shaon; Thomas Morris; Dinh C. Nguyen
Quantum federated learning (QFL) enables collaborative training of quantum machine learning (QML) models across distributed quantum devices without raw data exchange. However, QFL remains vulnerable to adversarial attacks, where shared QML model updates can be exploited to undermine information privacy. In the context of noisy intermediate-scale quantum (NISQ) devices, a key question arises: How can inherent quantum noise be leveraged to enforce differential privacy (DP) and protect model information during training and communication? This paper explores a novel DP mechanism that harnesses quantum noise to safeguard quantum models throughout the QFL process. By tuning noise variance through measurement shots and depolarizing channel strength, our approach achieves desired DP levels tailored to NISQ constraints. Simulations demonstrate the framework's effectiveness by examining the relationship between differential privacy budget and noise parameters, as well as the trade-off between security and training accuracy. Additionally, we demonstrate the framework's robustness against an adversarial attack designed to compromise model performance using adversarial examples, with evaluations based on critical metrics such as accuracy on adversarial examples, confidence scores for correct predictions, and attack success rates. The results reveal a tunable trade-off between privacy and robustness, providing an efficient solution for secure QFL on NISQ devices with significant potential for reliable quantum computing applications.

http://arxiv.org/abs/2508.20083
Disabling Self-Correction in Retrieval-Augmented Generation via Stealthy Retriever Poisoning. (50%)
Yanbo Dai; Zhenlan Ji; Zongjie Li; Kuan Li; Shuai Wang
Retrieval-Augmented Generation (RAG) has become a standard approach for improving the reliability of large language models (LLMs). Prior work demonstrates the vulnerability of RAG systems by misleading them into generating attacker-chosen outputs through poisoning the knowledge base. However, this paper uncovers that such attacks could be mitigated by the strong \textit{self-correction ability (SCA)} of modern LLMs, which can reject false context once properly configured. This SCA poses a significant challenge for attackers aiming to manipulate RAG systems.
  In contrast to previous poisoning methods, which primarily target the knowledge base, we introduce \textsc{DisarmRAG}, a new poisoning paradigm that compromises the retriever itself to suppress the SCA and enforce attacker-chosen outputs. This compromisation enables the attacker to straightforwardly embed anti-SCA instructions into the context provided to the generator, thereby bypassing the SCA. To this end, we present a contrastive-learning-based model editing technique that performs localized and stealthy edits, ensuring the retriever returns a malicious instruction only for specific victim queries while preserving benign retrieval behavior. To further strengthen the attack, we design an iterative co-optimization framework that automatically discovers robust instructions capable of bypassing prompt-based defenses. We extensively evaluate DisarmRAG across six LLMs and three QA benchmarks. Our results show near-perfect retrieval of malicious instructions, which successfully suppress SCA and achieve attack success rates exceeding 90\% under diverse defensive prompts. Also, the edited retriever remains stealthy under several detection methods, highlighting the urgent need for retriever-centric defenses.

http://arxiv.org/abs/2509.00088
AEGIS : Automated Co-Evolutionary Framework for Guarding Prompt Injections Schema. (47%)
Ting-Chun Liu; Ching-Yu Hsu; Kuan-Yi Lee; Chi-An Fu; Hung-yi Lee
Prompt injection attacks pose a significant challenge to the safe deployment of Large Language Models (LLMs) in real-world applications. While prompt-based detection offers a lightweight and interpretable defense strategy, its effectiveness has been hindered by the need for manual prompt engineering. To address this issue, we propose AEGIS , an Automated co-Evolutionary framework for Guarding prompt Injections Schema. Both attack and defense prompts are iteratively optimized against each other using a gradient-like natural language prompt optimization technique. This framework enables both attackers and defenders to autonomously evolve via a Textual Gradient Optimization (TGO) module, leveraging feedback from an LLM-guided evaluation loop. We evaluate our system on a real-world assignment grading dataset of prompt injection attacks and demonstrate that our method consistently outperforms existing baselines, achieving superior robustness in both attack success and detection. Specifically, the attack success rate (ASR) reaches 1.0, representing an improvement of 0.26 over the baseline. For detection, the true positive rate (TPR) improves by 0.23 compared to the previous best work, reaching 0.84, and the true negative rate (TNR) remains comparable at 0.89. Ablation studies confirm the importance of co-evolution, gradient buffering, and multi-objective optimization. We also confirm that this framework is effective in different LLMs. Our results highlight the promise of adversarial training as a scalable and effective approach for guarding prompt injections.

http://arxiv.org/abs/2509.00089
Learning from Peers: Collaborative Ensemble Adversarial Training. (13%)
Li Dengjin; Guo Yanming; Xie Yuxiang; Li Zheng; Chen Jiangming; Li Xiaolong; Lao Mingrui
Ensemble Adversarial Training (EAT) attempts to enhance the robustness of models against adversarial attacks by leveraging multiple models. However, current EAT strategies tend to train the sub-models independently, ignoring the cooperative benefits between sub-models. Through detailed inspections of the process of EAT, we find that that samples with classification disparities between sub-models are close to the decision boundary of ensemble, exerting greater influence on the robustness of ensemble. To this end, we propose a novel yet efficient Collaborative Ensemble Adversarial Training (CEAT), to highlight the cooperative learning among sub-models in the ensemble. To be specific, samples with larger predictive disparities between the sub-models will receive greater attention during the adversarial training of the other sub-models. CEAT leverages the probability disparities to adaptively assign weights to different samples, by incorporating a calibrating distance regularization. Extensive experiments on widely-adopted datasets show that our proposed method achieves the state-of-the-art performance over competitive EAT methods. It is noteworthy that CEAT is model-agnostic, which can be seamlessly adapted into various ensemble methods with flexible applicability.

http://arxiv.org/abs/2508.19488
PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense. (5%)
Xavier Cadet; Simona Boboila; Sie Hendrata Dharmawan; Alina Oprea; Peter Chin
Cyber defense requires automating defensive decision-making under stealthy, deceptive, and continuously evolving adversarial strategies. The FlipIt game provides a foundational framework for modeling interactions between a defender and an advanced adversary that compromises a system without being immediately detected. In FlipIt, the attacker and defender compete to control a shared resource by performing a Flip action and paying a cost. However, the existing FlipIt frameworks rely on a small number of heuristics or specialized learning techniques, which can lead to brittleness and the inability to adapt to new attacks. To address these limitations, we introduce PoolFlip, a multi-agent gym environment that extends the FlipIt game to allow efficient learning for attackers and defenders. Furthermore, we propose Flip-PSRO, a multi-agent reinforcement learning (MARL) approach that leverages population-based training to train defender agents equipped to generalize against a range of unknown, potentially adaptive opponents. Our empirical results suggest that Flip-PSRO defenders are $2\times$ more effective than baselines to generalize to a heuristic attack not exposed in training. In addition, our newly designed ownership-based utility functions ensure that Flip-PSRO defenders maintain a high level of control while optimizing performance.

http://arxiv.org/abs/2508.19819
From Research to Reality: Feasibility of Gradient Inversion Attacks in Federated Learning. (5%)
Viktor Valadi; Mattias Åkesson; Johan Östman; Salman Toor; Andreas Hellander
Gradient inversion attacks have garnered attention for their ability to compromise privacy in federated learning. However, many studies consider attacks with the model in inference mode, where training-time behaviors like dropout are disabled and batch normalization relies on fixed statistics. In this work, we systematically analyze how architecture and training behavior affect vulnerability, including the first in-depth study of inference-mode clients, which we show dramatically simplifies inversion. To assess attack feasibility under more realistic conditions, we turn to clients operating in standard training mode. In this setting, we find that successful attacks are only possible when several architectural conditions are met simultaneously: models must be shallow and wide, use skip connections, and, critically, employ pre-activation normalization. We introduce two novel attacks against models in training-mode with varying attacker knowledge, achieving state-of-the-art performance under realistic training conditions. We extend these efforts by presenting the first attack on a production-grade object-detection model. Here, to enable any visibly identifiable leakage, we revert to the lenient inference mode setting and make multiple architectural modifications to increase model vulnerability, with the extent of required changes highlighting the strong inherent robustness of such architectures. We conclude this work by offering the first comprehensive mapping of settings, clarifying which combinations of architectural choices and operational modes meaningfully impact privacy. Our analysis provides actionable insight into when models are likely vulnerable, when they appear robust, and where subtle leakage may persist. Together, these findings reframe how gradient inversion risk should be assessed in future research and deployment scenarios.

http://arxiv.org/abs/2508.20032
Pruning Strategies for Backdoor Defense in LLMs. (4%)
Santosh Chapagain; Shah Muhammad Hamdi; Soukaina Filali Boubrahimi
Backdoor attacks are a significant threat to the performance and integrity of pre-trained language models. Although such models are routinely fine-tuned for downstream NLP tasks, recent work shows they remain vulnerable to backdoor attacks that survive vanilla fine-tuning. These attacks are difficult to defend because end users typically lack knowledge of the attack triggers. Such attacks consist of stealthy malicious triggers introduced through subtle syntactic or stylistic manipulations, which can bypass traditional detection and remain in the model, making post-hoc purification essential. In this study, we explore whether attention-head pruning can mitigate these threats without any knowledge of the trigger or access to a clean reference model. To this end, we design and implement six pruning-based strategies: (i) gradient-based pruning, (ii) layer-wise variance pruning, (iii) gradient-based pruning with structured L1/L2 sparsification, (iv) randomized ensemble pruning, (v) reinforcement-learning-guided pruning, and (vi) Bayesian uncertainty pruning. Each method iteratively removes the least informative heads while monitoring validation accuracy to avoid over-pruning. Experimental evaluation shows that gradient-based pruning performs best while defending the syntactic triggers, whereas reinforcement learning and Bayesian pruning better withstand stylistic attacks.

http://arxiv.org/abs/2508.19641
Intellectual Property in Graph-Based Machine Learning as a Service: Attacks and Defenses. (3%)
Lincan Li; Bolin Shen; Chenxi Zhao; Yuxiang Sun; Kaixiang Zhao; Shirui Pan; Yushun Dong
Graph-structured data, which captures non-Euclidean relationships and interactions between entities, is growing in scale and complexity. As a result, training state-of-the-art graph machine learning (GML) models have become increasingly resource-intensive, turning these models and data into invaluable Intellectual Property (IP). To address the resource-intensive nature of model training, graph-based Machine-Learning-as-a-Service (GMLaaS) has emerged as an efficient solution by leveraging third-party cloud services for model development and management. However, deploying such models in GMLaaS also exposes them to potential threats from attackers. Specifically, while the APIs within a GMLaaS system provide interfaces for users to query the model and receive outputs, they also allow attackers to exploit and steal model functionalities or sensitive training data, posing severe threats to the safety of these GML models and the underlying graph data. To address these challenges, this survey systematically introduces the first taxonomy of threats and defenses at the level of both GML model and graph-structured data. Such a tailored taxonomy facilitates an in-depth understanding of GML IP protection. Furthermore, we present a systematic evaluation framework to assess the effectiveness of IP protection methods, introduce a curated set of benchmark datasets across various domains, and discuss their application scopes and future challenges. Finally, we establish an open-sourced versatile library named PyGIP, which evaluates various attack and defense techniques in GMLaaS scenarios and facilitates the implementation of existing benchmark methods. The library resource can be accessed at: https://labrai.github.io/PyGIP. We believe this survey will play a fundamental role in intellectual property protection for GML and provide practical recipes for the GML community.

http://arxiv.org/abs/2508.19180
MDD: a Mask Diffusion Detector to Protect Speaker Verification Systems from Adversarial Perturbations. (99%)
Yibo Bai; Sizhou Chen; Michele Panariello; Xiao-Lei Zhang; Massimiliano Todisco; Nicholas Evans
Speaker verification systems are increasingly deployed in security-sensitive applications but remain highly vulnerable to adversarial perturbations. In this work, we propose the Mask Diffusion Detector (MDD), a novel adversarial detection and purification framework based on a \textit{text-conditioned masked diffusion model}. During training, MDD applies partial masking to Mel-spectrograms and progressively adds noise through a forward diffusion process, simulating the degradation of clean speech features. A reverse process then reconstructs the clean representation conditioned on the input transcription. Unlike prior approaches, MDD does not require adversarial examples or large-scale pretraining. Experimental results show that MDD achieves strong adversarial detection performance and outperforms prior state-of-the-art methods, including both diffusion-based and neural codec-based approaches. Furthermore, MDD effectively purifies adversarially-manipulated speech, restoring speaker verification performance to levels close to those observed under clean conditions. These findings demonstrate the potential of diffusion-based masking strategies for secure and reliable speaker verification systems.

http://arxiv.org/abs/2508.18652
UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generation. (96%)
Runpeng Geng; Yanting Wang; Ying Chen; Jinyuan Jia
Retrieval-augmented generation (RAG) systems are widely deployed in real-world applications in diverse domains such as finance, healthcare, and cybersecurity. However, many studies showed that they are vulnerable to knowledge corruption attacks, where an attacker can inject adversarial texts into the knowledge database of a RAG system to induce the LLM to generate attacker-desired outputs. Existing studies mainly focus on attacking specific queries or queries with similar topics (or keywords). In this work, we propose UniC-RAG, a universal knowledge corruption attack against RAG systems. Unlike prior work, UniC-RAG jointly optimizes a small number of adversarial texts that can simultaneously attack a large number of user queries with diverse topics and domains, enabling an attacker to achieve various malicious objectives, such as directing users to malicious websites, triggering harmful command execution, or launching denial-of-service attacks. We formulate UniC-RAG as an optimization problem and further design an effective solution to solve it, including a balanced similarity-based clustering method to enhance the attack's effectiveness. Our extensive evaluations demonstrate that UniC-RAG is highly effective and significantly outperforms baselines. For instance, UniC-RAG could achieve over 90% attack success rate by injecting 100 adversarial texts into a knowledge database with millions of texts to simultaneously attack a large set of user queries (e.g., 2,000). Additionally, we evaluate existing defenses and show that they are insufficient to defend against UniC-RAG, highlighting the need for new defense mechanisms in RAG systems.

http://arxiv.org/abs/2508.18737
FLAegis: A Two-Layer Defense Framework for Federated Learning Against Poisoning Attacks. (92%)
Enrique Mármol Campos; Aurora González Vidal; José Luis Hernández Ramos; Antonio Skarmeta
Federated Learning (FL) has become a powerful technique for training Machine Learning (ML) models in a decentralized manner, preserving the privacy of the training datasets involved. However, the decentralized nature of FL limits the visibility of the training process, relying heavily on the honesty of participating clients. This assumption opens the door to malicious third parties, known as Byzantine clients, which can poison the training process by submitting false model updates. Such malicious clients may engage in poisoning attacks, manipulating either the dataset or the model parameters to induce misclassification. In response, this study introduces FLAegis, a two-stage defensive framework designed to identify Byzantine clients and improve the robustness of FL systems. Our approach leverages symbolic time series transformation (SAX) to amplify the differences between benign and malicious models, and spectral clustering, which enables accurate detection of adversarial behavior. Furthermore, we incorporate a robust FFT-based aggregation function as a final layer to mitigate the impact of those Byzantine clients that manage to evade prior defenses. We rigorously evaluate our method against five poisoning attacks, ranging from simple label flipping to adaptive optimization-based strategies. Notably, our approach outperforms state-of-the-art defenses in both detection precision and final model accuracy, maintaining consistently high performance even under strong adversarial conditions.

http://arxiv.org/abs/2508.18726
Flatness-aware Curriculum Learning via Adversarial Difficulty. (91%)
Hiroaki Aizawa; Yoshikazu Hayashi
Neural networks trained by empirical risk minimization often suffer from overfitting, especially to specific samples or domains, which leads to poor generalization. Curriculum Learning (CL) addresses this issue by selecting training samples based on the difficulty. From the optimization perspective, methods such as Sharpness-Aware Minimization (SAM) improve robustness and generalization by seeking flat minima. However, combining CL with SAM is not straightforward. In flat regions, both the loss values and the gradient norms tend to become uniformly small, which makes it difficult to evaluate sample difficulty and design an effective curriculum. To overcome this problem, we propose the Adversarial Difficulty Measure (ADM), which quantifies adversarial vulnerability by leveraging the robustness properties of models trained toward flat minima. Unlike loss- or gradient-based measures, which become ineffective as training progresses into flatter regions, ADM remains informative by measuring the normalized loss gap between original and adversarial examples. We incorporate ADM into CL-based training with SAM to dynamically assess sample difficulty. We evaluated our approach on image classification tasks, fine-grained recognition, and domain generalization. The results demonstrate that our method preserves the strengths of both CL and SAM while outperforming existing curriculum-based and flatness-aware training strategies.

http://arxiv.org/abs/2508.19456
ReLATE+: Unified Framework for Adversarial Attack Detection, Classification, and Resilient Model Selection in Time-Series Classification. (84%)
Cagla Ipek Kocal; Onat Gungor; Tajana Rosing; Baris Aksanli
Minimizing computational overhead in time-series classification, particularly in deep learning models, presents a significant challenge due to the high complexity of model architectures and the large volume of sequential data that must be processed in real time. This challenge is further compounded by adversarial attacks, emphasizing the need for resilient methods that ensure robust performance and efficient model selection. To address this challenge, we propose ReLATE+, a comprehensive framework that detects and classifies adversarial attacks, adaptively selects deep learning models based on dataset-level similarity, and thus substantially reduces retraining costs relative to conventional methods that do not leverage prior knowledge, while maintaining strong performance. ReLATE+ first checks whether the incoming data is adversarial and, if so, classifies the attack type, using this insight to identify a similar dataset from a repository and enable the reuse of the best-performing associated model. This approach ensures strong performance while reducing the need for retraining, and it generalizes well across different domains with varying data distributions and feature spaces. Experiments show that ReLATE+ reduces computational overhead by an average of 77.68%, enhancing adversarial resilience and streamlining robust model selection, all without sacrificing performance, within 2.02% of Oracle.

http://arxiv.org/abs/2508.19151
Saddle Hierarchy in Dense Associative Memory. (74%)
Robin Thériault; Daniele Tantari
Dense associative memory (DAM) models have been attracting renewed attention since they were shown to be robust to adversarial examples and closely related to state-of-the-art machine learning paradigms, such as the attention mechanisms in transformers and generative diffusion models. We study a DAM built upon a three-layer Boltzmann machine with Potts hidden units, which represent data clusters and classes. Through a statistical mechanics analysis, we derive saddle-point equations that characterize both the stationary points of DAMs trained on real data and the fixed points of DAMs trained on synthetic data within a teacher-student framework. Based on these results, we propose a novel regularization scheme that makes training significantly more stable. Moreover, we show empirically that our DAM learns interpretable solutions to both supervised and unsupervised classification problems. Pushing our theoretical analysis further, we find that the weights learned by relatively small DAMs correspond to unstable saddle points in larger DAMs. We implement a network-growing algorithm that leverages this saddle-point hierarchy to drastically reduce the computational cost of training dense associative memory.

http://arxiv.org/abs/2508.18805
Hidden Tail: Adversarial Image Causing Stealthy Resource Consumption in Vision-Language Models. (45%)
Rui Zhang; Zihan Wang; Tianli Yang; Hongwei Li; Wenbo Jiang; Qingchuan Zhao; Yang Liu; Guowen Xu
Vision-Language Models (VLMs) are increasingly deployed in real-world applications, but their high inference cost makes them vulnerable to resource consumption attacks. Prior attacks attempt to extend VLM output sequences by optimizing adversarial images, thereby increasing inference costs. However, these extended outputs often introduce irrelevant abnormal content, compromising attack stealthiness. This trade-off between effectiveness and stealthiness poses a major limitation for existing attacks. To address this challenge, we propose \textit{Hidden Tail}, a stealthy resource consumption attack that crafts prompt-agnostic adversarial images, inducing VLMs to generate maximum-length outputs by appending special tokens invisible to users. Our method employs a composite loss function that balances semantic preservation, repetitive special token induction, and suppression of the end-of-sequence (EOS) token, optimized via a dynamic weighting strategy. Extensive experiments show that \textit{Hidden Tail} outperforms existing attacks, increasing output length by up to 19.2$\times$ and reaching the maximum token limit, while preserving attack stealthiness. These results highlight the urgent need to improve the robustness of VLMs against efficiency-oriented adversarial threats. Our code is available at https://github.com/zhangrui4041/Hidden_Tail.

http://arxiv.org/abs/2508.18649
PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality. (15%)
Nanxi Li; Zhengyue Zhao; Chaowei Xiao
Safeguarding vision-language models (VLMs) is a critical challenge, as existing methods often suffer from over-defense, which harms utility, or rely on shallow alignment, failing to detect complex threats that require deep reasoning. To this end, we introduce PRISM (Principled Reasoning for Integrated Safety in Multimodality), a system2-like framework that aligns VLMs by embedding a structured, safety-aware reasoning process. Our framework consists of two key components: PRISM-CoT, a dataset that teaches safety-aware chain-of-thought reasoning, and PRISM-DPO, generated via Monte Carlo Tree Search (MCTS) to further refine this reasoning through Direct Preference Optimization to help obtain a delicate safety boundary. Comprehensive evaluations demonstrate PRISM's effectiveness, achieving remarkably low attack success rates including 0.15% on JailbreakV-28K for Qwen2-VL and 90% improvement over the previous best method on VLBreak for LLaVA-1.5. PRISM also exhibits strong robustness against adaptive attacks, significantly increasing computational costs for adversaries, and generalizes effectively to out-of-distribution challenges, reducing attack success rates to just 8.70% on the challenging multi-image MIS benchmark. Remarkably, this robust defense is achieved while preserving, and in some cases enhancing, model utility. To promote reproducibility, we have made our code, data, and model weights available at https://github.com/SaFoLab-WISC/PRISM.

http://arxiv.org/abs/2508.19183
Get Global Guarantees: On the Probabilistic Nature of Perturbation Robustness. (9%)
Wenchuan Mu; Kwan Hui Lim
In safety-critical deep learning applications, robustness measures the ability of neural models that handle imperceptible perturbations in input data, which may lead to potential safety hazards. Existing pre-deployment robustness assessment methods typically suffer from significant trade-offs between computational cost and measurement precision, limiting their practical utility. To address these limitations, this paper conducts a comprehensive comparative analysis of existing robustness definitions and associated assessment methodologies. We propose tower robustness to evaluate robustness, which is a novel, practical metric based on hypothesis testing to quantitatively evaluate probabilistic robustness, enabling more rigorous and efficient pre-deployment assessments. Our extensive comparative evaluation illustrates the advantages and applicability of our proposed approach, thereby advancing the systematic understanding and enhancement of model robustness in safety-critical deep learning applications.

http://arxiv.org/abs/2508.18019
Does simple trump complex? Comparing strategies for adversarial robustness in DNNs. (93%)
William Brooks; Marelie H. Davel; Coenraad Mouton
Deep Neural Networks (DNNs) have shown substantial success in various applications but remain vulnerable to adversarial attacks. This study aims to identify and isolate the components of two different adversarial training techniques that contribute most to increased adversarial robustness, particularly through the lens of margins in the input space -- the minimal distance between data points and decision boundaries. Specifically, we compare two methods that maximize margins: a simple approach which modifies the loss function to increase an approximation of the margin, and a more complex state-of-the-art method (Dynamics-Aware Robust Training) which builds upon this approach. Using a VGG-16 model as our base, we systematically isolate and evaluate individual components from these methods to determine their relative impact on adversarial robustness. We assess the effect of each component on the model's performance under various adversarial attacks, including AutoAttack and Projected Gradient Descent (PGD). Our analysis on the CIFAR-10 dataset reveals which elements most effectively enhance adversarial robustness, providing insights for designing more robust DNNs.

http://arxiv.org/abs/2508.19290
Efficient Model-Based Purification Against Adversarial Attacks for LiDAR Segmentation. (92%)
Alexandros Gkillas; Ioulia Kapsali; Nikos Piperigkos; Aris S. Lalos
LiDAR-based segmentation is essential for reliable perception in autonomous vehicles, yet modern segmentation networks are highly susceptible to adversarial attacks that can compromise safety. Most existing defenses are designed for networks operating directly on raw 3D point clouds and rely on large, computationally intensive generative models. However, many state-of-the-art LiDAR segmentation pipelines operate on more efficient 2D range view representations. Despite their widespread adoption, dedicated lightweight adversarial defenses for this domain remain largely unexplored. We introduce an efficient model-based purification framework tailored for adversarial defense in 2D range-view LiDAR segmentation. We propose a direct attack formulation in the range-view domain and develop an explainable purification network based on a mathematical justified optimization problem, achieving strong adversarial resilience with minimal computational overhead. Our method achieves competitive performance on open benchmarks, consistently outperforming generative and adversarial training baselines. More importantly, real-world deployment on a demo vehicle demonstrates the framework's ability to deliver accurate operation in practical autonomous driving scenarios.

http://arxiv.org/abs/2402.13532
Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggers. (83%)
Quanyu Long; Yue Deng; LeiLei Gan; Wenya Wang; Sinno Jialin Pan
Dense retrieval systems have been widely used in various NLP applications. However, their vulnerabilities to potential attacks have been underexplored. This paper investigates a novel attack scenario where the attackers aim to mislead the retrieval system into retrieving the attacker-specified contents. Those contents, injected into the retrieval corpus by attackers, can include harmful text like hate speech or spam. Unlike prior methods that rely on model weights and generate conspicuous, unnatural outputs, we propose a covert backdoor attack triggered by grammar errors. Our approach ensures that the attacked models can function normally for standard queries while covertly triggering the retrieval of the attacker's contents in response to minor linguistic mistakes. Specifically, dense retrievers are trained with contrastive loss and hard negative sampling. Surprisingly, our findings demonstrate that contrastive loss is notably sensitive to grammatical errors, and hard negative sampling can exacerbate susceptibility to backdoor attacks. Our proposed method achieves a high attack success rate with a minimal corpus poisoning rate of only 0.048\%, while preserving normal retrieval performance. This indicates that the method has negligible impact on user experience for error-free queries. Furthermore, evaluations across three real-world defense strategies reveal that the malicious passages embedded within the corpus remain highly resistant to detection and filtering, underscoring the robustness and subtlety of the proposed attack \footnote{Codes of this work are available at https://github.com/ruyue0001/Backdoor_DPR.}.

http://arxiv.org/abs/2508.17660
ClearMask: Noise-Free and Naturalness-Preserving Protection Against Voice Deepfake Attacks. (75%)
Yuanda Wang; Bocheng Chen; Hanqing Guo; Guangjing Wang; Weikang Ding; Qiben Yan
Voice deepfake attacks, which artificially impersonate human speech for malicious purposes, have emerged as a severe threat. Existing defenses typically inject noise into human speech to compromise voice encoders in speech synthesis models. However, these methods degrade audio quality and require prior knowledge of the attack approaches, limiting their effectiveness in diverse scenarios. Moreover, real-time audios, such as speech in virtual meetings and voice messages, are still exposed to voice deepfake threats. To overcome these limitations, we propose ClearMask, a noise-free defense mechanism against voice deepfake attacks. Unlike traditional approaches, ClearMask modifies the audio mel-spectrogram by selectively filtering certain frequencies, inducing a transferable voice feature loss without injecting noise. We then apply audio style transfer to further deceive voice decoders while preserving perceived sound quality. Finally, optimized reverberation is introduced to disrupt the output of voice generation models without affecting the naturalness of the speech. Additionally, we develop LiveMask to protect streaming speech in real-time through a universal frequency filter and reverberation generator. Our experimental results show that ClearMask and LiveMask effectively prevent voice deepfake attacks from deceiving speaker verification models and human listeners, even for unseen voice synthesis models and black-box API services. Furthermore, ClearMask demonstrates resilience against adaptive attackers who attempt to recover the original audio signal from the protected speech samples.

http://arxiv.org/abs/2508.18154
Assessing the Noise Robustness of Class Activation Maps: A Framework for Reliable Model Interpretability. (38%)
Syamantak Sarkar; Revoti P. Bora; Bhupender Kaushal; Sudhish N George; Kiran Raja
Class Activation Maps (CAMs) are one of the important methods for visualizing regions used by deep learning models. Yet their robustness to different noise remains underexplored. In this work, we evaluate and report the resilience of various CAM methods for different noise perturbations across multiple architectures and datasets. By analyzing the influence of different noise types on CAM explanations, we assess the susceptibility to noise and the extent to which dataset characteristics may impact explanation stability. The findings highlight considerable variability in noise sensitivity for various CAMs. We propose a robustness metric for CAMs that captures two key properties: consistency and responsiveness. Consistency reflects the ability of CAMs to remain stable under input perturbations that do not alter the predicted class, while responsiveness measures the sensitivity of CAMs to changes in the prediction caused by such perturbations. The metric is evaluated empirically across models, different perturbations, and datasets along with complementary statistical tests to exemplify the applicability of our proposed approach.

http://arxiv.org/abs/2508.17680
Robustness Feature Adapter for Efficient Adversarial Training. (16%)
Quanwei Wu; Jun Guo; Wei Wang; Yi Wang
Adversarial training (AT) with projected gradient descent is the most popular method to improve model robustness under adversarial attacks. However, computational overheads become prohibitively large when AT is applied to large backbone models. AT is also known to have the issue of robust overfitting. This paper contributes to solving both problems simultaneously towards building more trustworthy foundation models. In particular, we propose a new adapter-based approach for efficient AT directly in the feature space. We show that the proposed adapter-based approach can improve the inner-loop convergence quality by eliminating robust overfitting. As a result, it significantly increases computational efficiency and improves model accuracy by generalizing adversarial robustness to unseen attacks. We demonstrate the effectiveness of the new adapter-based approach in different backbone architectures and in AT at scale.

http://arxiv.org/abs/2503.00187
Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks. (12%)
Hanjiang Hu; Alexander Robey; Changliu Liu
Large language models (LLMs) are shown to be vulnerable to jailbreaking attacks where adversarial prompts are designed to elicit harmful responses. While existing defenses effectively mitigate single-turn attacks by detecting and filtering unsafe inputs, they fail against multi-turn jailbreaks that exploit contextual drift over multiple interactions, gradually leading LLMs away from safe behavior. To address this challenge, we propose a safety steering framework grounded in safe control theory, ensuring invariant safety in multi-turn dialogues. Our approach models the dialogue with LLMs using state-space representations and introduces a novel neural barrier function (NBF) to detect and filter harmful queries emerging from evolving contexts proactively. Our method achieves invariant safety at each turn of dialogue by learning a safety predictor that accounts for adversarial queries, preventing potential context drift toward jailbreaks. Extensive experiments under multiple LLMs show that our NBF-based safety steering outperforms safety alignment, prompt-based steering and lightweight LLM guardrails baselines, offering stronger defenses against multi-turn jailbreaks while maintaining a better trade-off among safety, helpfulness and over-refusal. Check out the website here https://sites.google.com/view/llm-nbf/home . Our code is available on https://github.com/HanjiangHu/NBF-LLM .

http://arxiv.org/abs/2508.18085
Quantum-Classical Hybrid Framework for Zero-Day Time-Push GNSS Spoofing Detection. (8%)
Abyad Enan; Mashrur Chowdhury; Sagar Dasgupta; Mizanur Rahman
Global Navigation Satellite Systems (GNSS) are critical for Positioning, Navigation, and Timing (PNT) applications. However, GNSS are highly vulnerable to spoofing attacks, where adversaries transmit counterfeit signals to mislead receivers. Such attacks can lead to severe consequences, including misdirected navigation, compromised data integrity, and operational disruptions. Most existing spoofing detection methods depend on supervised learning techniques and struggle to detect novel, evolved, and unseen attacks. To overcome this limitation, we develop a zero-day spoofing detection method using a Hybrid Quantum-Classical Autoencoder (HQC-AE), trained solely on authentic GNSS signals without exposure to spoofed data. By leveraging features extracted during the tracking stage, our method enables proactive detection before PNT solutions are computed. We focus on spoofing detection in static GNSS receivers, which are particularly susceptible to time-push spoofing attacks, where attackers manipulate timing information to induce incorrect time computations at the receiver. We evaluate our model against different unseen time-push spoofing attack scenarios: simplistic, intermediate, and sophisticated. Our analysis demonstrates that the HQC-AE consistently outperforms its classical counterpart, traditional supervised learning-based models, and existing unsupervised learning-based methods in detecting zero-day, unseen GNSS time-push spoofing attacks, achieving an average detection accuracy of 97.71% with an average false negative rate of 0.62% (when an attack occurs but is not detected). For sophisticated spoofing attacks, the HQC-AE attains an accuracy of 98.23% with a false negative rate of 1.85%. These findings highlight the effectiveness of our method in proactively detecting zero-day GNSS time-push spoofing attacks across various stationary GNSS receiver platforms.

http://arxiv.org/abs/2305.15276
Sparse Mean Estimation in Adversarial Settings via Incremental Learning. (1%)
Jianhao Ma; Rui Ray Chen; Yinghui He; Salar Fattahi; Wei Hu
In this paper, we study the problem of sparse mean estimation under adversarial corruptions, where the goal is to estimate the $k$-sparse mean of a heavy-tailed distribution from samples contaminated by adversarial noise. Existing methods face two key limitations: they require prior knowledge of the sparsity level $k$ and scale poorly to high-dimensional settings. We propose a simple and scalable estimator that addresses both challenges. Specifically, it learns the $k$-sparse mean without knowing $k$ in advance and operates in near-linear time and memory with respect to the ambient dimension. Under a moderate signal-to-noise ratio, our method achieves the optimal statistical rate, matching the information-theoretic lower bound. Extensive simulations corroborate our theoretical guarantees. At the heart of our approach is an incremental learning phenomenon: we show that a basic subgradient method applied to a nonconvex two-layer formulation with an $\ell_1$-loss can incrementally learn the $k$ nonzero components of the true mean while suppressing the rest. More broadly, our work is the first to reveal the incremental learning phenomenon of the subgradient method in the presence of heavy-tailed distributions and adversarial corruption.

http://arxiv.org/abs/2508.18060
FedGreed: A Byzantine-Robust Loss-Based Aggregation Method for Federated Learning. (1%)
Emmanouil Kritharakis; Antonios Makris; Dusan Jakovetic; Konstantinos Tserpes
Federated Learning (FL) enables collaborative model training across multiple clients while preserving data privacy by keeping local datasets on-device. In this work, we address FL settings where clients may behave adversarially, exhibiting Byzantine attacks, while the central server is trusted and equipped with a reference dataset. We propose FedGreed, a resilient aggregation strategy for federated learning that does not require any assumptions about the fraction of adversarial participants. FedGreed orders clients' local model updates based on their loss metrics evaluated against a trusted dataset on the server and greedily selects a subset of clients whose models exhibit the minimal evaluation loss. Unlike many existing approaches, our method is designed to operate reliably under heterogeneous (non-IID) data distributions, which are prevalent in real-world deployments. FedGreed exhibits convergence guarantees and bounded optimality gaps under strong adversarial behavior. Experimental evaluations on MNIST, FMNIST, and CIFAR-10 demonstrate that our method significantly outperforms standard and robust federated learning baselines, such as Mean, Trimmed Mean, Median, Krum, and Multi-Krum, in the majority of adversarial scenarios considered, including label flipping and Gaussian noise injection attacks. All experiments were conducted using the Flower federated learning framework.

http://arxiv.org/abs/2508.17174
Sharpness-Aware Geometric Defense for Robust Out-Of-Distribution Detection. (96%)
Jeng-Lin Li; Ming-Ching Chang; Wei-Chao Chen
Out-of-distribution (OOD) detection ensures safe and reliable model deployment. Contemporary OOD algorithms using geometry projection can detect OOD or adversarial samples from clean in-distribution (ID) samples. However, this setting regards adversarial ID samples as OOD, leading to incorrect OOD predictions. Existing efforts on OOD detection with ID and OOD data under attacks are minimal. In this paper, we develop a robust OOD detection method that distinguishes adversarial ID samples from OOD ones. The sharp loss landscape created by adversarial training hinders model convergence, impacting the latent embedding quality for OOD score calculation. Therefore, we introduce a {\bf Sharpness-aware Geometric Defense (SaGD)} framework to smooth out the rugged adversarial loss landscape in the projected latent geometry. Enhanced geometric embedding convergence enables accurate ID data characterization, benefiting OOD detection against adversarial attacks. We use Jitter-based perturbation in adversarial training to extend the defense ability against unseen attacks. Our SaGD framework significantly improves FPR and AUC over the state-of-the-art defense approaches in differentiating CIFAR-100 from six other OOD datasets under various attacks. We further examine the effects of perturbations at various adversarial training levels, revealing the relationship between the sharp loss landscape and adversarial OOD detection.

http://arxiv.org/abs/2508.17186
Advancing Weakly-Supervised Change Detection in Satellite Images via Adversarial Class Prompting. (56%)
Zhenghui Zhao; Chen Wu; Di Wang; Hongruixuan Chen; Cuiqun Chen; Zhuo Zheng; Bo Du; Liangpei Zhang
Weakly-Supervised Change Detection (WSCD) aims to distinguish specific object changes (e.g., objects appearing or disappearing) from background variations (e.g., environmental changes due to light, weather, or seasonal shifts) in paired satellite images, relying only on paired image (i.e., image-level) classification labels. This technique significantly reduces the need for dense annotations required in fully-supervised change detection. However, as image-level supervision only indicates whether objects have changed in a scene, WSCD methods often misclassify background variations as object changes, especially in complex remote-sensing scenarios. In this work, we propose an Adversarial Class Prompting (AdvCP) method to address this co-occurring noise problem, including two phases: a) Adversarial Prompt Mining: After each training iteration, we introduce adversarial prompting perturbations, using incorrect one-hot image-level labels to activate erroneous feature mappings. This process reveals co-occurring adversarial samples under weak supervision, namely background variation features that are likely to be misclassified as object changes. b) Adversarial Sample Rectification: We integrate these adversarially prompt-activated pixel samples into training by constructing an online global prototype. This prototype is built from an exponentially weighted moving average of the current batch and all historical training data. Our AdvCP can be seamlessly integrated into current WSCD methods without adding additional inference cost. Experiments on ConvNet, Transformer, and Segment Anything Model (SAM)-based baselines demonstrate significant performance enhancements. Furthermore, we demonstrate the generalizability of AdvCP to other multi-class weakly-supervised dense prediction scenarios. Code is available at https://github.com/zhenghuizhao/AdvCP

http://arxiv.org/abs/2508.17265
AdaGAT: Adaptive Guidance Adversarial Training for the Robustness of Deep Neural Networks. (54%)
Zhenyu Liu; Huizhi Liang; Xinrun Li; Vaclav Snasel; Varun Ojha
Adversarial distillation (AD) is a knowledge distillation technique that facilitates the transfer of robustness from teacher deep neural network (DNN) models to lightweight target (student) DNN models, enabling the target models to perform better than only training the student model independently. Some previous works focus on using a small, learnable teacher (guide) model to improve the robustness of a student model. Since a learnable guide model starts learning from scratch, maintaining its optimal state for effective knowledge transfer during co-training is challenging. Therefore, we propose a novel Adaptive Guidance Adversarial Training (AdaGAT) method. Our method, AdaGAT, dynamically adjusts the training state of the guide model to install robustness to the target model. Specifically, we develop two separate loss functions as part of the AdaGAT method, allowing the guide model to participate more actively in backpropagation to achieve its optimal state. We evaluated our approach via extensive experiments on three datasets: CIFAR-10, CIFAR-100, and TinyImageNet, using the WideResNet-34-10 model as the target model. Our observations reveal that appropriately adjusting the guide model within a certain accuracy range enhances the target model's robustness across various adversarial attacks compared to a variety of baseline models.

http://arxiv.org/abs/2508.17247
Uncovering and Mitigating Destructive Multi-Embedding Attacks in Deepfake Proactive Forensics. (10%)
Lixin Jia; Haiyang Sun; Zhiqing Guo; Yunfeng Diao; Dan Ma; Gaobo Yang
With the rapid evolution of deepfake technologies and the wide dissemination of digital media, personal privacy is facing increasingly serious security threats. Deepfake proactive forensics, which involves embedding imperceptible watermarks to enable reliable source tracking, serves as a crucial defense against these threats. Although existing methods show strong forensic ability, they rely on an idealized assumption of single watermark embedding, which proves impractical in real-world scenarios. In this paper, we formally define and demonstrate the existence of Multi-Embedding Attacks (MEA) for the first time. When a previously protected image undergoes additional rounds of watermark embedding, the original forensic watermark can be destroyed or removed, rendering the entire proactive forensic mechanism ineffective. To address this vulnerability, we propose a general training paradigm named Adversarial Interference Simulation (AIS). Rather than modifying the network architecture, AIS explicitly simulates MEA scenarios during fine-tuning and introduces a resilience-driven loss function to enforce the learning of sparse and stable watermark representations. Our method enables the model to maintain the ability to extract the original watermark correctly even after a second embedding. Extensive experiments demonstrate that our plug-and-play AIS training paradigm significantly enhances the robustness of various existing methods against MEA.

http://arxiv.org/abs/2508.17405
FRAME : Comprehensive Risk Assessment Framework for Adversarial Machine Learning Threats. (2%)
Avishag Shapira; Simon Shigol; Asaf Shabtai
The widespread adoption of machine learning (ML) systems increased attention to their security and emergence of adversarial machine learning (AML) techniques that exploit fundamental vulnerabilities in ML systems, creating an urgent need for comprehensive risk assessment for ML-based systems. While traditional risk assessment frameworks evaluate conventional cybersecurity risks, they lack ability to address unique challenges posed by AML threats. Existing AML threat evaluation approaches focus primarily on technical attack robustness, overlooking crucial real-world factors like deployment environments, system dependencies, and attack feasibility. Attempts at comprehensive AML risk assessment have been limited to domain-specific solutions, preventing application across diverse systems. Addressing these limitations, we present FRAME, the first comprehensive and automated framework for assessing AML risks across diverse ML-based systems. FRAME includes a novel risk assessment method that quantifies AML risks by systematically evaluating three key dimensions: target system's deployment environment, characteristics of diverse AML techniques, and empirical insights from prior research. FRAME incorporates a feasibility scoring mechanism and LLM-based customization for system-specific assessments. Additionally, we developed a comprehensive structured dataset of AML attacks enabling context-aware risk assessment. From an engineering application perspective, FRAME delivers actionable results designed for direct use by system owners with only technical knowledge of their systems, without expertise in AML. We validated it across six diverse real-world applications. Our evaluation demonstrated exceptional accuracy and strong alignment with analysis by AML experts. FRAME enables organizations to prioritize AML risks, supporting secure AI deployment in real-world environments.

http://arxiv.org/abs/2508.17215
How to make Medical AI Systems safer? Simulating Vulnerabilities, and Threats in Multimodal Medical RAG System. (1%)
Kaiwen Zuo; Zelin Liu; Raman Dutt; Ziyang Wang; Zhongtian Sun; Yeming Wang; Fan Mo; Pietro Liò
Large Vision-Language Models (LVLMs) augmented with Retrieval-Augmented Generation (RAG) are increasingly employed in medical AI to enhance factual grounding through external clinical image-text retrieval. However, this reliance creates a significant attack surface. We propose MedThreatRAG, a novel multimodal poisoning framework that systematically probes vulnerabilities in medical RAG systems by injecting adversarial image-text pairs. A key innovation of our approach is the construction of a simulated semi-open attack environment, mimicking real-world medical systems that permit periodic knowledge base updates via user or pipeline contributions. Within this setting, we introduce and emphasize Cross-Modal Conflict Injection (CMCI), which embeds subtle semantic contradictions between medical images and their paired reports. These mismatches degrade retrieval and generation by disrupting cross-modal alignment while remaining sufficiently plausible to evade conventional filters. While basic textual and visual attacks are included for completeness, CMCI demonstrates the most severe degradation. Evaluations on IU-Xray and MIMIC-CXR QA tasks show that MedThreatRAG reduces answer F1 scores by up to 27.66% and lowers LLaVA-Med-1.5 F1 rates to as low as 51.36%. Our findings expose fundamental security gaps in clinical RAG systems and highlight the urgent need for threat-aware design and robust multimodal consistency checks. Finally, we conclude with a concise set of guidelines to inform the safe development of future multimodal medical RAG systems.

http://arxiv.org/abs/2508.17244
L-XAIDS: A LIME-based eXplainable AI framework for Intrusion Detection Systems. (1%)
Aoun E Muhammad; Kin-Choong Yow; Nebojsa Bacanin-Dzakula; Muhammad Attique Khan
Recent developments in Artificial Intelligence (AI) and their applications in critical industries such as healthcare, fin-tech and cybersecurity have led to a surge in research in explainability in AI. Innovative research methods are being explored to extract meaningful insight from blackbox AI systems to make the decision-making technology transparent and interpretable. Explainability becomes all the more critical when AI is used in decision making in domains like fintech, healthcare and safety critical systems such as cybersecurity and autonomous vehicles. However, there is still ambiguity lingering on the reliable evaluations for the users and nature of transparency in the explanations provided for the decisions made by black-boxed AI. To solve the blackbox nature of Machine Learning based Intrusion Detection Systems, a framework is proposed in this paper to give an explanation for IDSs decision making. This framework uses Local Interpretable Model-Agnostic Explanations (LIME) coupled with Explain Like I'm five (ELI5) and Decision Tree algorithms to provide local and global explanations and improve the interpretation of IDSs. The local explanations provide the justification for the decision made on a specific input. Whereas, the global explanations provides the list of significant features and their relationship with attack traffic. In addition, this framework brings transparency in the field of ML driven IDS that might be highly significant for wide scale adoption of eXplainable AI in cyber-critical systems. Our framework is able to achieve 85 percent accuracy in classifying attack behaviour on UNSW-NB15 dataset, while at the same time displaying the feature significance ranking of the top 10 features used in the classification.

http://arxiv.org/abs/2508.16937
NAT: Learning to Attack Neurons for Enhanced Adversarial Transferability. (82%)
Krishna Kanth Nakka; Alexandre Alahi
The generation of transferable adversarial perturbations typically involves training a generator to maximize embedding separation between clean and adversarial images at a single mid-layer of a source model. In this work, we build on this approach and introduce Neuron Attack for Transferability (NAT), a method designed to target specific neuron within the embedding. Our approach is motivated by the observation that previous layer-level optimizations often disproportionately focus on a few neurons representing similar concepts, leaving other neurons within the attacked layer minimally affected. NAT shifts the focus from embedding-level separation to a more fundamental, neuron-specific approach. We find that targeting individual neurons effectively disrupts the core units of the neural network, providing a common basis for transferability across different models. Through extensive experiments on 41 diverse ImageNet models and 9 fine-grained models, NAT achieves fooling rates that surpass existing baselines by over 14\% in cross-model and 4\% in cross-domain settings. Furthermore, by leveraging the complementary attacking capabilities of the trained generators, we achieve impressive fooling rates within just 10 queries. Our code is available at: https://krishnakanthnakka.github.io/NAT/

http://arxiv.org/abs/2508.19277
POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization. (2%)
Xinyu Li; Tianjin Huang; Ronghui Mu; Xiaowei Huang; Gaojie Jin
Recent advances in Chain-of-Thought (CoT) prompting have substantially enhanced the reasoning capabilities of large language models (LLMs), enabling sophisticated problem-solving through explicit multi-step reasoning traces. However, these enhanced reasoning processes introduce novel attack surfaces, particularly vulnerabilities to computational inefficiency through unnecessarily verbose reasoning chains that consume excessive resources without corresponding performance gains. Prior overthinking attacks typically require restrictive conditions including access to external knowledge sources for data poisoning, reliance on retrievable poisoned content, and structurally obvious templates that limit practical applicability in real-world scenarios. To address these limitations, we propose POT (Prompt-Only OverThinking), a novel black-box attack framework that employs LLM-based iterative optimization to generate covert and semantically natural adversarial prompts, eliminating dependence on external data access and model retrieval. Extensive experiments across diverse model architectures and datasets demonstrate that POT achieves superior performance compared to other methods.

http://arxiv.org/abs/2508.18306
SALMAN: Stability Analysis of Language Models Through the Maps Between Graph-based Manifolds. (2%)
Wuxinlin Cheng; Yupeng Cao; Jinwen Wu; Koduvayur Subbalakshmi; Tian Han; Zhuo Feng
Recent strides in pretrained transformer-based language models have propelled state-of-the-art performance in numerous NLP tasks. Yet, as these models grow in size and deployment, their robustness under input perturbations becomes an increasingly urgent question. Existing robustness methods often diverge between small-parameter and large-scale models (LLMs), and they typically rely on labor-intensive, sample-specific adversarial designs. In this paper, we propose a unified, local (sample-level) robustness framework (SALMAN) that evaluates model stability without modifying internal parameters or resorting to complex perturbation heuristics. Central to our approach is a novel Distance Mapping Distortion (DMD) measure, which ranks each sample's susceptibility by comparing input-to-output distance mappings in a near-linear complexity manner. By demonstrating significant gains in attack efficiency and robust training, we position our framework as a practical, model-agnostic tool for advancing the reliability of transformer-based NLP systems.

http://arxiv.org/abs/2508.17043
ZAPS: A Zero-Knowledge Proof Protocol for Secure UAV Authentication with Flight Path Privacy. (1%)
Shayesta Naziri; Xu Wang; Guangsheng Yu; Christy Jie Liang; Wei Ni
The increasing deployment of Unmanned Aerial Vehicles (UAVs) for military, commercial, and logistics applications has raised significant concerns regarding flight path privacy. Conventional UAV communication systems often expose flight path data to third parties, making them vulnerable to tracking, surveillance, and location inference attacks. Existing encryption techniques provide security but fail to ensure complete privacy, as adversaries can still infer movement patterns through metadata analysis. To address these challenges, we propose a zk-SNARK(Zero-Knowledge Succinct Non-Interactive Argument of Knowledge)-based privacy-preserving flight path authentication and verification framework. Our approach ensures that a UAV can prove its authorisation, validate its flight path with a control centre, and comply with regulatory constraints without revealing any sensitive trajectory information. By leveraging zk-SNARKs, the UAV can generate cryptographic proofs that verify compliance with predefined flight policies while keeping the exact path and location undisclosed. This method mitigates risks associated with real-time tracking, identity exposure, and unauthorised interception, thereby enhancing UAV operational security in adversarial environments. Our proposed solution balances privacy, security, and computational efficiency, making it suitable for resource-constrained UAVs in both civilian and military applications.

http://arxiv.org/abs/2508.17158
Towards Safeguarding LLM Fine-tuning APIs against Cipher Attacks. (1%)
Jack Youstra; Mohammed Mahfoud; Yang Yan; Henry Sleight; Ethan Perez; Mrinank Sharma
Large language model fine-tuning APIs enable widespread model customization, yet pose significant safety risks. Recent work shows that adversaries can exploit access to these APIs to bypass model safety mechanisms by encoding harmful content in seemingly harmless fine-tuning data, evading both human monitoring and standard content filters. We formalize the fine-tuning API defense problem, and introduce the Cipher Fine-tuning Robustness benchmark (CIFR), a benchmark for evaluating defense strategies' ability to retain model safety in the face of cipher-enabled attackers while achieving the desired level of fine-tuning functionality. We include diverse cipher encodings and families, with some kept exclusively in the test set to evaluate for generalization across unseen ciphers and cipher families. We then evaluate different defenses on the benchmark and train probe monitors on model internal activations from multiple fine-tunes. We show that probe monitors achieve over 99% detection accuracy, generalize to unseen cipher variants and families, and compare favorably to state-of-the-art monitoring approaches. We open-source CIFR and the code to reproduce our experiments to facilitate further research in this critical area. Code and data are available online https://github.com/JackYoustra/safe-finetuning-api

http://arxiv.org/abs/2508.06827
Who's the Evil Twin? Differential Auditing for Undesired Behavior. (93%)
Ishwar Balappanawar; Venkata Hasith Vattikuti; Greta Kintzley; Ronan Azimi-Mancel; Satvik Golechha
Detecting hidden behaviors in neural networks poses a significant challenge due to minimal prior knowledge and potential adversarial obfuscation. We explore this problem by framing detection as an adversarial game between two teams: the red team trains two similar models, one trained solely on benign data and the other trained on data containing hidden harmful behavior, with the performance of both being nearly indistinguishable on the benign dataset. The blue team, with limited to no information about the harmful behaviour, tries to identify the compromised model. We experiment using CNNs and try various blue team strategies, including Gaussian noise analysis, model diffing, integrated gradients, and adversarial attacks under different levels of hints provided by the red team. Results show high accuracy for adversarial-attack-based methods (100\% correct prediction, using hints), which is very promising, whilst the other techniques yield more varied performance. During our LLM-focused rounds, we find that there are not many parallel methods that we could apply from our study with CNNs. Instead, we find that effective LLM auditing methods require some hints about the undesired distribution, which can then used in standard black-box and open-weight methods to probe the models further and reveal their misalignment. We open-source our auditing games (with the model and data) and hope that our findings contribute to designing better audits.

http://arxiv.org/abs/2508.16225
An Investigation of Visual Foundation Models Robustness. (62%)
Sandeep Gupta; Roberto Passerone
Visual Foundation Models (VFMs) are becoming ubiquitous in computer vision, powering systems for diverse tasks such as object detection, image classification, segmentation, pose estimation, and motion tracking. VFMs are capitalizing on seminal innovations in deep learning models, such as LeNet-5, AlexNet, ResNet, VGGNet, InceptionNet, DenseNet, YOLO, and ViT, to deliver superior performance across a range of critical computer vision applications. These include security-sensitive domains like biometric verification, autonomous vehicle perception, and medical image analysis, where robustness is essential to fostering trust between technology and the end-users. This article investigates network robustness requirements crucial in computer vision systems to adapt effectively to dynamic environments influenced by factors such as lighting, weather conditions, and sensor characteristics. We examine the prevalent empirical defenses and robust training employed to enhance vision network robustness against real-world challenges such as distributional shifts, noisy and spatially distorted inputs, and adversarial attacks. Subsequently, we provide a comprehensive analysis of the challenges associated with these defense mechanisms, including network properties and components to guide ablation studies and benchmarking metrics to evaluate network robustness.

http://arxiv.org/abs/2508.14527
Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehicles. (56%)
Jiangfan Liu; Yongkang Guo; Fangzhi Zhong; Tianyuan Zhang; Zonglei Jing; Siyuan Liang; Jiakai Wang; Mingchuan Zhang; Aishan Liu; Xianglong Liu
The generation of safety-critical scenarios in simulation has become increasingly crucial for safety evaluation in autonomous vehicles prior to road deployment in society. However, current approaches largely rely on predefined threat patterns or rule-based strategies, which limit their ability to expose diverse and unforeseen failure modes. To overcome these, we propose ScenGE, a framework that can generate plentiful safety-critical scenarios by reasoning novel adversarial cases and then amplifying them with complex traffic flows. Given a simple prompt of a benign scene, it first performs Meta-Scenario Generation, where a large language model, grounded in structured driving knowledge, infers an adversarial agent whose behavior poses a threat that is both plausible and deliberately challenging. This meta-scenario is then specified in executable code for precise in-simulator control. Subsequently, Complex Scenario Evolution uses background vehicles to amplify the core threat introduced by Meta-Scenario. It builds an adversarial collaborator graph to identify key agent trajectories for optimization. These perturbations are designed to simultaneously reduce the ego vehicle's maneuvering space and create critical occlusions. Extensive experiments conducted on multiple reinforcement learning based AV models show that ScenGE uncovers more severe collision cases (+31.96%) on average than SoTA baselines. Additionally, our ScenGE can be applied to large model based AV systems and deployed on different simulators; we further observe that adversarial training on our scenarios improves the model robustness. Finally, we validate our framework through real-world vehicle tests and human evaluation, confirming that the generated scenarios are both plausible and critical. We hope our paper can build up a critical step towards building public trust and ensuring their safe deployment.

http://arxiv.org/abs/2307.12555
Robust Graph Contrastive Learning with Information Restoration. (38%)
Yulin Zhu; Xing Ai; Yevgeniy Vorobeychik; Kai Zhou
The graph contrastive learning (GCL) framework has gained remarkable achievements in graph representation learning. However, similar to graph neural networks (GNNs), GCL models are susceptible to graph structural attacks. As an unsupervised method, GCL faces greater challenges in defending against adversarial attacks. Furthermore, there has been limited research on enhancing the robustness of GCL. To thoroughly explore the failure of GCL on the poisoned graphs, we investigate the detrimental effects of graph structural attacks against the GCL framework. We discover that, in addition to the conventional observation that graph structural attacks tend to connect dissimilar node pairs, these attacks also diminish the mutual information between the graph and its representations from an information-theoretical perspective, which is the cornerstone of the high-quality node embeddings for GCL. Motivated by this theoretical insight, we propose a robust graph contrastive learning framework with a learnable sanitation view that endeavors to sanitize the augmented graphs by restoring the diminished mutual information caused by the structural attacks. Additionally, we design a fully unsupervised tuning strategy to tune the hyperparameters without accessing the label information, which strictly coincides with the defender's knowledge. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method compared to competitive baselines.

http://arxiv.org/abs/2508.16481
Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harms. (38%)
Jonathan Nöther; Adish Singla; Goran Radanovic
Ensuring the safe use of agentic systems requires a thorough understanding of the range of malicious behaviors these systems may exhibit when under attack. In this paper, we evaluate the robustness of LLM-based agentic systems against attacks that aim to elicit harmful actions from agents. To this end, we propose a novel taxonomy of harms for agentic systems and a novel benchmark, BAD-ACTS, for studying the security of agentic systems with respect to a wide range of harmful actions. BAD-ACTS consists of 4 implementations of agentic systems in distinct application environments, as well as a dataset of 188 high-quality examples of harmful actions. This enables a comprehensive study of the robustness of agentic systems across a wide range of categories of harmful behaviors, available tools, and inter-agent communication structures. Using this benchmark, we analyze the robustness of agentic systems against an attacker that controls one of the agents in the system and aims to manipulate other agents to execute a harmful target action. Our results show that the attack has a high success rate, demonstrating that even a single adversarial agent within the system can have a significant impact on the security. This attack remains effective even when agents use a simple prompting-based defense strategy. However, we additionally propose a more effective defense based on message monitoring. We believe that this benchmark provides a diverse testbed for the security research of agentic systems. The benchmark can be found at github.com/JNoether/BAD-ACTS

http://arxiv.org/abs/2508.16217
PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting. (10%)
Hohyun Na; Seunghoo Hong; Simon S. Woo
The success of diffusion models has enabled effortless, high-quality image modifications that precisely align with users' intentions, thereby raising concerns about their potential misuse by malicious actors. Previous studies have attempted to mitigate such misuse through adversarial attacks. However, these approaches heavily rely on image-level inconsistencies, which pose fundamental limitations in addressing the influence of textual prompts. In this paper, we propose PromptFlare, a novel adversarial protection method designed to protect images from malicious modifications facilitated by diffusion-based inpainting models. Our approach leverages the cross-attention mechanism to exploit the intrinsic properties of prompt embeddings. Specifically, we identify and target shared token of prompts that is invariant and semantically uninformative, injecting adversarial noise to suppress the sampling process. The injected noise acts as a cross-attention decoy, diverting the model's focus away from meaningful prompt-image alignments and thereby neutralizing the effect of prompt. Extensive experiments on the EditBench dataset demonstrate that our method achieves state-of-the-art performance across various metrics while significantly reducing computational overhead and GPU memory usage. These findings highlight PromptFlare as a robust and efficient protection against unauthorized image manipulations. The code is available at https://github.com/NAHOHYUN-SKKU/PromptFlare.

http://arxiv.org/abs/2508.16124
Domain Adaptation via Feature Refinement. (1%)
Savvas Karatsiolis; Andreas Kamilaris
We propose Domain Adaptation via Feature Refinement (DAFR2), a simple yet effective framework for unsupervised domain adaptation under distribution shift. The proposed method synergistically combines three key components: adaptation of Batch Normalization statistics using unlabeled target data, feature distillation from a source-trained model and hypothesis transfer. By aligning feature distributions at the statistical and representational levels, DAFR2 produces robust and domain-invariant feature spaces that generalize across similar domains without requiring target labels, complex architectures or sophisticated training objectives. Extensive experiments on benchmark datasets, including CIFAR10-C, CIFAR100-C, MNIST-C and PatchCamelyon-C, demonstrate that the proposed algorithm outperforms prior methods in robustness to corruption. Theoretical and empirical analyses further reveal that our method achieves improved feature alignment, increased mutual information between the domains and reduced sensitivity to input perturbations.

http://arxiv.org/abs/2508.16406
Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models. (1%)
Guangyu Yang; Jinghong Chen; Jingbiao Mei; Weizhe Lin; Bill Byrne
Large Language Models (LLMs) remain vulnerable to jailbreak attacks, which attempt to elicit harmful responses from LLMs. The evolving nature and diversity of these attacks pose many challenges for defense systems, including (1) adaptation to counter emerging attack strategies without costly retraining, and (2) control of the trade-off between safety and utility. To address these challenges, we propose Retrieval-Augmented Defense (RAD), a novel framework for jailbreak detection that incorporates a database of known attack examples into Retrieval-Augmented Generation, which is used to infer the underlying, malicious user query and jailbreak strategy used to attack the system. RAD enables training-free updates for newly discovered jailbreak strategies and provides a mechanism to balance safety and utility. Experiments on StrongREJECT show that RAD substantially reduces the effectiveness of strong jailbreak attacks such as PAP and PAIR while maintaining low rejection rates for benign queries. We propose a novel evaluation scheme and show that RAD achieves a robust safety-utility trade-off across a range of operating points in a controllable manner.

http://arxiv.org/abs/2508.15481
On Evaluating the Adversarial Robustness of Foundation Models for Multimodal Entity Linking. (99%)
Fang Wang; Yongjie Wang; Zonghao Yang; Minghao Hu; Xiaoying Bai
The explosive growth of multimodal data has driven the rapid development of multimodal entity linking (MEL) models. However, existing studies have not systematically investigated the impact of visual adversarial attacks on MEL models. We conduct the first comprehensive evaluation of the robustness of mainstream MEL models under different adversarial attack scenarios, covering two core tasks: Image-to-Text (I2T) and Image+Text-to-Text (IT2T). Experimental results show that current MEL models generally lack sufficient robustness against visual perturbations. Interestingly, contextual semantic information in input can partially mitigate the impact of adversarial perturbations. Based on this insight, we propose an LLM and Retrieval-Augmented Entity Linking (LLM-RetLink), which significantly improves the model's anti-interference ability through a two-stage process: first, extracting initial entity descriptions using large vision models (LVMs), and then dynamically generating candidate descriptive sentences via web-based retrieval. Experiments on five datasets demonstrate that LLM-RetLink improves the accuracy of MEL by 0.4%-35.7%, especially showing significant advantages under adversarial conditions. This research highlights a previously unexplored facet of MEL robustness, constructs and releases the first MEL adversarial example dataset, and sets the stage for future work aimed at strengthening the resilience of multimodal systems in adversarial environments.

http://arxiv.org/abs/2508.15650
Towards a 3D Transfer-based Black-box Attack via Critical Feature Guidance. (99%)
Shuchao Pang; Zhenghan Chen; Shen Zhang; Liming Lu; Siyuan Liang; Anan Du; Yongbin Zhou
Deep neural networks for 3D point clouds have been demonstrated to be vulnerable to adversarial examples. Previous 3D adversarial attack methods often exploit certain information about the target models, such as model parameters or outputs, to generate adversarial point clouds. However, in realistic scenarios, it is challenging to obtain any information about the target models under conditions of absolute security. Therefore, we focus on transfer-based attacks, where generating adversarial point clouds does not require any information about the target models. Based on our observation that the critical features used for point cloud classification are consistent across different DNN architectures, we propose CFG, a novel transfer-based black-box attack method that improves the transferability of adversarial point clouds via the proposed Critical Feature Guidance. Specifically, our method regularizes the search of adversarial point clouds by computing the importance of the extracted features, prioritizing the corruption of critical features that are likely to be adopted by diverse architectures. Further, we explicitly constrain the maximum deviation extent of the generated adversarial point clouds in the loss function to ensure their imperceptibility. Extensive experiments conducted on the ModelNet40 and ScanObjectNN benchmark datasets demonstrate that the proposed CFG outperforms the state-of-the-art attack methods by a large margin.

http://arxiv.org/abs/2312.12556
Tensor Train Decomposition for Adversarial Attacks on Computer Vision Models. (96%)
Andrei Chertkov; Ivan Oseledets
Deep neural networks (DNNs) are widely used today, but they are vulnerable to adversarial attacks. To develop effective methods of defense, it is important to understand the potential weak spots of DNNs. Often attacks are organized taking into account the architecture of models (white-box approach) and based on gradient methods, but for real-world DNNs this approach in most cases is impossible. At the same time, several gradient-free optimization algorithms are used to attack black-box models. However, classical methods are often ineffective in the multidimensional case. To organize black-box attacks for computer vision models, in this work, we propose the use of an optimizer based on the low-rank tensor train (TT) format, which has gained popularity in various practical multidimensional applications in recent years. Combined with the attribution of the target image, which is built by the auxiliary (white-box) model, the TT-based optimization method makes it possible to organize an effective black-box attack by small perturbation of pixels in the target image. The superiority of the proposed approach over three popular baselines is demonstrated for seven modern DNNs on the ImageNet dataset.

http://arxiv.org/abs/2508.15252
Retrieval-Augmented Review Generation for Poisoning Recommender Systems. (92%)
Shiyi Yang; Xinshu Li; Guanglin Zhou; Chen Wang; Xiwei Xu; Liming Zhu; Lina Yao
Recent studies have shown that recommender systems (RSs) are highly vulnerable to data poisoning attacks, where malicious actors inject fake user profiles, including a group of well-designed fake ratings, to manipulate recommendations. Due to security and privacy constraints in practice, attackers typically possess limited knowledge of the victim system and thus need to craft profiles that have transferability across black-box RSs. To maximize the attack impact, the profiles often remains imperceptible. However, generating such high-quality profiles with the restricted resources is challenging. Some works suggest incorporating fake textual reviews to strengthen the profiles; yet, the poor quality of the reviews largely undermines the attack effectiveness and imperceptibility under the practical setting.
  To tackle the above challenges, in this paper, we propose to enhance the quality of the review text by harnessing in-context learning (ICL) capabilities of multimodal foundation models. To this end, we introduce a demonstration retrieval algorithm and a text style transfer strategy to augment the navie ICL. Specifically, we propose a novel practical attack framework named RAGAN to generate high-quality fake user profiles, which can gain insights into the robustness of RSs. The profiles are generated by a jailbreaker and collaboratively optimized on an instructional agent and a guardian to improve the attack transferability and imperceptibility. Comprehensive experiments on various real-world datasets demonstrate that RAGAN achieves the state-of-the-art poisoning attack performance.

http://arxiv.org/abs/2508.15283
Adversarial Attacks against Neural Ranking Models via In-Context Learning. (81%)
Amin Bigdeli; Negar Arabzadeh; Ebrahim Bagheri; Charles L. A. Clarke
While neural ranking models (NRMs) have shown high effectiveness, they remain susceptible to adversarial manipulation. In this work, we introduce Few-Shot Adversarial Prompting (FSAP), a novel black-box attack framework that leverages the in-context learning capabilities of Large Language Models (LLMs) to generate high-ranking adversarial documents. Unlike previous approaches that rely on token-level perturbations or manual rewriting of existing documents, FSAP formulates adversarial attacks entirely through few-shot prompting, requiring no gradient access or internal model instrumentation. By conditioning the LLM on a small support set of previously observed harmful examples, FSAP synthesizes grammatically fluent and topically coherent documents that subtly embed false or misleading information and rank competitively against authentic content. We instantiate FSAP in two modes: FSAP-IntraQ, which leverages harmful examples from the same query to enhance topic fidelity, and FSAP-InterQ, which enables broader generalization by transferring adversarial patterns across unrelated queries. Our experiments on the TREC 2020 and 2021 Health Misinformation Tracks, using four diverse neural ranking models, reveal that FSAP-generated documents consistently outrank credible, factually accurate documents. Furthermore, our analysis demonstrates that these adversarial outputs exhibit strong stance alignment and low detectability, posing a realistic and scalable threat to neural retrieval systems. FSAP also effectively generalizes across both proprietary and open-source LLMs.

http://arxiv.org/abs/2508.15565
Any-to-any Speaker Attribute Perturbation for Asynchronous Voice Anonymization. (75%)
Liping Chen; Chenyang Guo; Rui Wang; Kong Aik Lee; Zhenhua Ling
Speaker attribute perturbation offers a feasible approach to asynchronous voice anonymization by employing adversarially perturbed speech as anonymized output. In order to enhance the identity unlinkability among anonymized utterances from the same original speaker, the targeted attack training strategy is usually applied to anonymize the utterances to a common designated speaker. However, this strategy may violate the privacy of the designated speaker who is an actual speaker. To mitigate this risk, this paper proposes an any-to-any training strategy. It is accomplished by defining a batch mean loss to anonymize the utterances from various speakers within a training mini-batch to a common pseudo-speaker, which is approximated as the average speaker in the mini-batch. Based on this, a speaker-adversarial speech generation model is proposed, incorporating the supervision from both the untargeted attack and the any-to-any strategies. The speaker attribute perturbations are generated and incorporated into the original speech to produce its anonymized version. The effectiveness of the proposed model was justified in asynchronous voice anonymization through experiments conducted on the VoxCeleb datasets. Additional experiments were carried out to explore the potential limitations of speaker-adversarial speech in voice privacy protection. With them, we aim to provide insights for future research on its protective efficacy against black-box speaker extractors \textcolor{black}{and adaptive attacks, as well as} generalization to out-of-domain datasets \textcolor{black}{and stability}. Audio samples and open-source code are published in https://github.com/VoicePrivacy/any-to-any-speaker-attribute-perturbation.

http://arxiv.org/abs/2508.15182
SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks. (75%)
Xiangman Li; Xiaodong Wu; Qi Li; Jianbing Ni; Rongxing Lu
Jailbreak attacks pose a serious threat to the safety of Large Language Models (LLMs) by crafting adversarial prompts that bypass alignment mechanisms, causing the models to produce harmful, restricted, or biased content. In this paper, we propose SafeLLM, a novel unlearning-based defense framework that unlearn the harmful knowledge from LLMs while preserving linguistic fluency and general capabilities. SafeLLM employs a three-stage pipeline: (1) dynamic unsafe output detection using a hybrid approach that integrates external classifiers with model-internal evaluations; (2) token-level harmful content tracing through feedforward network (FFN) activations to localize harmful knowledge; and (3) constrained optimization to suppress unsafe behavior without degrading overall model quality. SafeLLM achieves targeted and irreversible forgetting by identifying and neutralizing FFN substructures responsible for harmful generation pathways. Extensive experiments on prominent LLMs (Vicuna, LLaMA, and GPT-J) across multiple jailbreak benchmarks show that SafeLLM substantially reduces attack success rates while maintaining high general-purpose performance. Compared to standard defense methods such as supervised fine-tuning and direct preference optimization, SafeLLM offers stronger safety guarantees, more precise control over harmful behavior, and greater robustness to unseen attacks. Moreover, SafeLLM maintains the general performance after the harmful knowledge unlearned. These results highlight unlearning as a promising direction for scalable and effective LLM safety.

http://arxiv.org/abs/2412.01528
CopyrightShield: Enhancing Diffusion Model Security against Copyright Infringement Attacks. (75%)
Zhixiang Guo; Siyuan Liang; Aishan Liu; Dacheng Tao
Diffusion models have attracted significant attention due to its exceptional data generation capabilities in fields such as image synthesis. However, recent studies have shown that diffusion models are vulnerable to copyright infringement attacks, where attackers inject strategically modified non-infringing images into the training set, inducing the model to generate infringing content under the prompt of specific poisoned captions. To address this issue, we first propose a defense framework, CopyrightShield, to defend against the above attack. Specifically, we analyze the memorization mechanism of diffusion models and find that attacks exploit the model's overfitting to specific spatial positions and prompts, causing it to reproduce poisoned samples under backdoor triggers. Based on this, we propose a poisoned sample detection method using spatial masking and data attribution to quantify poisoning risk and accurately identify hidden backdoor samples. To further mitigate memorization of poisoned features, we introduce an adaptive optimization strategy that integrates a dynamic penalty term into the training loss, reducing reliance on infringing features while preserving generative performance. Experimental results demonstrate that CopyrightShield significantly improves poisoned sample detection performance across two attack scenarios, achieving average F1-scores of 0.665, retarding the First-Attack Epoch (FAE) of 115.2% and decreasing the Copyright Infringement Rate (CIR) by 56.7%. Compared to the SoTA backdoor defense in diffusion models, the defense effect is improved by about 25%, showcasing its superiority and practicality in enhancing the security of diffusion models.

http://arxiv.org/abs/2508.15454
Mini-Batch Robustness Verification of Deep Neural Networks. (68%)
Saar Tzour-Shaday; Dana Drachsler Cohen
Neural network image classifiers are ubiquitous in many safety-critical applications. However, they are susceptible to adversarial attacks. To understand their robustness to attacks, many local robustness verifiers have been proposed to analyze $ε$-balls of inputs. Yet, existing verifiers introduce a long analysis time or lose too much precision, making them less effective for a large set of inputs. In this work, we propose a new approach to local robustness: group local robustness verification. The key idea is to leverage the similarity of the network computations of certain $ε$-balls to reduce the overall analysis time. We propose BaVerLy, a sound and complete verifier that boosts the local robustness verification of a set of $ε$-balls by dynamically constructing and verifying mini-batches. BaVerLy adaptively identifies successful mini-batch sizes, accordingly constructs mini-batches of $ε$-balls that have similar network computations, and verifies them jointly. If a mini-batch is verified, all $ε$-balls are proven robust. Otherwise, one $ε$-ball is suspected as not being robust, guiding the refinement. In the latter case, BaVerLy leverages the analysis results to expedite the analysis of that $ε$-ball as well as the other $ε$-balls in the batch. We evaluate BaVerLy on fully connected and convolutional networks for MNIST and CIFAR-10. Results show that BaVerLy scales the common one by one verification by 2.3x on average and up to 4.1x, in which case it reduces the total analysis time from 24 hours to 6 hours.

http://arxiv.org/abs/2508.15310
IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents. (26%)
Hengyu An; Jinghuai Zhang; Tianyu Du; Chunyi Zhou; Qingming Li; Tao Lin; Shouling Ji
Large language model (LLM) agents are widely deployed in real-world applications, where they leverage tools to retrieve and manipulate external data for complex tasks. However, when interacting with untrusted data sources (e.g., fetching information from public websites), tool responses may contain injected instructions that covertly influence agent behaviors and lead to malicious outcomes, a threat referred to as Indirect Prompt Injection (IPI). Existing defenses typically rely on advanced prompting strategies or auxiliary detection models. While these methods have demonstrated some effectiveness, they fundamentally rely on assumptions about the model's inherent security, which lacks structural constraints on agent behaviors. As a result, agents still retain unrestricted access to tool invocations, leaving them vulnerable to stronger attack vectors that can bypass the security guardrails of the model. To prevent malicious tool invocations at the source, we propose a novel defensive task execution paradigm, called IPIGuard, which models the agents' task execution process as a traversal over a planned Tool Dependency Graph (TDG). By explicitly decoupling action planning from interaction with external data, IPIGuard significantly reduces unintended tool invocations triggered by injected instructions, thereby enhancing robustness against IPI attacks. Experiments on the AgentDojo benchmark show that IPIGuard achieves a superior balance between effectiveness and robustness, paving the way for the development of safer agentic systems in dynamic environments.

http://arxiv.org/abs/2508.15764
Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Space. (13%)
Kiarash Kazari; Ezzeldin Shereen; György Dán
We address the problem of detecting adversarial attacks against cooperative multi-agent reinforcement learning with continuous action space. We propose a decentralized detector that relies solely on the local observations of the agents and makes use of a statistical characterization of the normal behavior of observable agents. The proposed detector utilizes deep neural networks to approximate the normal behavior of agents as parametric multivariate Gaussian distributions. Based on the predicted density functions, we define a normality score and provide a characterization of its mean and variance. This characterization allows us to employ a two-sided CUSUM procedure for detecting deviations of the normality score from its mean, serving as a detector of anomalous behavior in real-time. We evaluate our scheme on various multi-agent PettingZoo benchmarks against different state-of-the-art attack methods, and our results demonstrate the effectiveness of our method in detecting impactful adversarial attacks. Particularly, it outperforms the discrete counterpart by achieving AUC-ROC scores of over 0.95 against the most impactful attacks in all evaluated environments.

http://arxiv.org/abs/2508.15934
Strategic Sample Selection for Improved Clean-Label Backdoor Attacks in Text Classification. (8%)
Onur Alp Kirci; M. Emre Gursoy
Backdoor attacks pose a significant threat to the integrity of text classification models used in natural language processing. While several dirty-label attacks that achieve high attack success rates (ASR) have been proposed, clean-label attacks are inherently more difficult. In this paper, we propose three sample selection strategies to improve attack effectiveness in clean-label scenarios: Minimum, Above50, and Below50. Our strategies identify those samples which the model predicts incorrectly or with low confidence, and by injecting backdoor triggers into such samples, we aim to induce a stronger association between the trigger patterns and the attacker-desired target label. We apply our methods to clean-label variants of four canonical backdoor attacks (InsertSent, WordInj, StyleBkd, SynBkd) and evaluate them on three datasets (IMDB, SST2, HateSpeech) and four model types (LSTM, BERT, DistilBERT, RoBERTa). Results show that the proposed strategies, particularly the Minimum strategy, significantly improve the ASR over random sample selection with little or no degradation in the model's clean accuracy. Furthermore, clean-label attacks enhanced by our strategies outperform BITE, a state of the art clean-label attack method, in many configurations.

http://arxiv.org/abs/2508.15669
Exploiting Policy Idling for Dexterous Manipulation. (1%)
Annie S. Chen; Philemon Brakel; Antonia Bronars; Annie Xie; Sandy Huang; Oliver Groth; Maria Bauza; Markus Wulfmeier; Nicolas Heess; Dushyant Rao
Learning-based methods for dexterous manipulation have made notable progress in recent years. However, learned policies often still lack reliability and exhibit limited robustness to important factors of variation. One failure pattern that can be observed across many settings is that policies idle, i.e. they cease to move beyond a small region of states when they reach certain states. This policy idling is often a reflection of the training data. For instance, it can occur when the data contains small actions in areas where the robot needs to perform high-precision motions, e.g., when preparing to grasp an object or object insertion. Prior works have tried to mitigate this phenomenon e.g. by filtering the training data or modifying the control frequency. However, these approaches can negatively impact policy performance in other ways. As an alternative, we investigate how to leverage the detectability of idling behavior to inform exploration and policy improvement. Our approach, Pause-Induced Perturbations (PIP), applies perturbations at detected idling states, thus helping it to escape problematic basins of attraction. On a range of challenging simulated dual-arm tasks, we find that this simple approach can already noticeably improve test-time performance, with no additional supervision or training. Furthermore, since the robot tends to idle at critical points in a movement, we also find that learning from the resulting episodes leads to better iterative policy improvement compared to prior approaches. Our perturbation strategy also leads to a 15-35% improvement in absolute success rate on a real-world insertion task that requires complex multi-finger manipulation.

http://arxiv.org/abs/2508.14699
Foe for Fraud: Transferable Adversarial Attacks in Credit Card Fraud Detection. (99%)
Jan Lum Fok; Qingwen Zeng; Shiping Chen; Oscar Fawkes; Huaming Chen
Credit card fraud detection (CCFD) is a critical application of Machine Learning (ML) in the financial sector, where accurately identifying fraudulent transactions is essential for mitigating financial losses. ML models have demonstrated their effectiveness in fraud detection task, in particular with the tabular dataset. While adversarial attacks have been extensively studied in computer vision and deep learning, their impacts on the ML models, particularly those trained on CCFD tabular datasets, remains largely unexplored. These latent vulnerabilities pose significant threats to the security and stability of the financial industry, especially in high-value transactions where losses could be substantial. To address this gap, in this paper, we present a holistic framework that investigate the robustness of CCFD ML model against adversarial perturbations under different circumstances. Specifically, the gradient-based attack methods are incorporated into the tabular credit card transaction data in both black- and white-box adversarial attacks settings. Our findings confirm that tabular data is also susceptible to subtle perturbations, highlighting the need for heightened awareness among financial technology practitioners regarding ML model security and trustworthiness. Furthermore, the experiments by transferring adversarial samples from gradient-based attack method to non-gradient-based models also verify our findings. Our results demonstrate that such attacks remain effective, emphasizing the necessity of developing robust defenses for CCFD algorithms.

http://arxiv.org/abs/2508.15020
TAIGen: Training-Free Adversarial Image Generation via Diffusion Models. (99%)
Susim Roy; Anubhooti Jain; Mayank Vatsa; Richa Singh
Adversarial attacks from generative models often produce low-quality images and require substantial computational resources. Diffusion models, though capable of high-quality generation, typically need hundreds of sampling steps for adversarial generation. This paper introduces TAIGen, a training-free black-box method for efficient adversarial image generation. TAIGen produces adversarial examples using only 3-20 sampling steps from unconditional diffusion models. Our key finding is that perturbations injected during the mixing step interval achieve comparable attack effectiveness without processing all timesteps. We develop a selective RGB channel strategy that applies attention maps to the red channel while using GradCAM-guided perturbations on green and blue channels. This design preserves image structure while maximizing misclassification in target models. TAIGen maintains visual quality with PSNR above 30 dB across all tested datasets. On ImageNet with VGGNet as source, TAIGen achieves 70.6% success against ResNet, 80.8% against MNASNet, and 97.8% against ShuffleNet. The method generates adversarial examples 10x faster than existing diffusion-based attacks. Our method achieves the lowest robust accuracy, indicating it is the most impactful attack as the defense mechanism is least successful in purifying the images generated by TAIGen.

http://arxiv.org/abs/2508.14853
Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent. (98%)
Sajib Biswas; Mao Nishino; Samuel Jacob Chacko; Xiuwen Liu
As large language models (LLMs) are increasingly deployed in critical applications, ensuring their robustness and safety alignment remains a major challenge. Despite the overall success of alignment techniques such as reinforcement learning from human feedback (RLHF) on typical prompts, LLMs remain vulnerable to jailbreak attacks enabled by crafted adversarial triggers appended to user prompts. Most existing jailbreak methods either rely on inefficient searches over discrete token spaces or direct optimization of continuous embeddings. While continuous embeddings can be given directly to selected open-source models as input, doing so is not feasible for proprietary models. On the other hand, projecting these embeddings back into valid discrete tokens introduces additional complexity and often reduces attack effectiveness. We propose an intrinsic optimization method which directly optimizes relaxed one-hot encodings of the adversarial suffix tokens using exponentiated gradient descent coupled with Bregman projection, ensuring that the optimized one-hot encoding of each token always remains within the probability simplex. We provide theoretical proof of convergence for our proposed method and implement an efficient algorithm that effectively jailbreaks several widely used LLMs. Our method achieves higher success rates and faster convergence compared to three state-of-the-art baselines, evaluated on five open-source LLMs and four adversarial behavior datasets curated for evaluating jailbreak methods. In addition to individual prompt attacks, we also generate universal adversarial suffixes effective across multiple prompts and demonstrate transferability of optimized suffixes to different LLMs.

http://arxiv.org/abs/2508.18235
Sealing The Backdoor: Unlearning Adversarial Text Triggers In Diffusion Models Using Knowledge Distillation. (84%)
Ashwath Vaithinathan Aravindan; Abha Jha; Matthew Salaway; Atharva Sandeep Bhide; Duygu Nur Yaldiz
Text-to-image diffusion models have revolutionized generative AI, but their vulnerability to backdoor attacks poses significant security risks. Adversaries can inject imperceptible textual triggers into training data, causing models to generate manipulated outputs. Although text-based backdoor defenses in classification models are well-explored, generative models lack effective mitigation techniques against. We address this by selectively erasing the model's learned associations between adversarial text triggers and poisoned outputs, while preserving overall generation quality. Our approach, Self-Knowledge Distillation with Cross-Attention Guidance (SKD-CAG), uses knowledge distillation to guide the model in correcting responses to poisoned prompts while maintaining image quality by exploiting the fact that the backdoored model still produces clean outputs in the absence of triggers. Using the cross-attention mechanism, SKD-CAG neutralizes backdoor influences at the attention level, ensuring the targeted removal of adversarial effects. Extensive experiments show that our method outperforms existing approaches, achieving removal accuracy 100\% for pixel backdoors and 93\% for style-based attacks, without sacrificing robustness or image fidelity. Our findings highlight targeted unlearning as a promising defense to secure generative models. Code and model weights can be found at https://github.com/Mystic-Slice/Sealing-The-Backdoor .

http://arxiv.org/abs/2508.14757
Distributional Adversarial Attacks and Training in Deep Hedging. (78%)
Guangyi He; Tobias Sutter; Lukas Gonon
In this paper, we study the robustness of classical deep hedging strategies under distributional shifts by leveraging the concept of adversarial attacks. We first demonstrate that standard deep hedging models are highly vulnerable to small perturbations in the input distribution, resulting in significant performance degradation. Motivated by this, we propose an adversarial training framework tailored to increase the robustness of deep hedging strategies. Our approach extends pointwise adversarial attacks to the distributional setting and introduces a computationally tractable reformulation of the adversarial optimization problem over a Wasserstein ball. This enables the efficient training of hedging strategies that are resilient to distributional perturbations. Through extensive numerical experiments, we show that adversarially trained deep hedging strategies consistently outperform their classical counterparts in terms of out-of-sample performance and resilience to model misspecification. Our findings establish a practical and effective framework for robust deep hedging under realistic market uncertainties.

http://arxiv.org/abs/2508.14530
DOPA: Stealthy and Generalizable Backdoor Attacks from a Single Client under Challenging Federated Constraints. (74%)
Xuezheng Qin; Ruwei Huang; Xiaolong Tang; Feng Li
Federated Learning (FL) is increasingly adopted for privacy-preserving collaborative training, but its decentralized nature makes it particularly susceptible to backdoor attacks. Existing attack methods, however, often rely on idealized assumptions and fail to remain effective under real-world constraints, such as limited attacker control, non-IID data distributions, and the presence of diverse defense mechanisms. To address this gap, we propose DOPA (Divergent Optimization Path Attack), a novel framework that simulates heterogeneous local training dynamics and seeks consensus across divergent optimization trajectories to craft universally effective and stealthy backdoor triggers. By leveraging consistency signals across simulated paths to guide optimization, DOPA overcomes the challenge of heterogeneity-induced instability and achieves practical attack viability under stringent federated constraints. We validate DOPA on a comprehensive suite of 12 defense strategies, two model architectures (ResNet18/VGG16), two datasets (CIFAR-10/TinyImageNet), and both mild and extreme non-IID settings. Despite operating under a single-client, black-box, and sparsely participating threat model, DOPA consistently achieves high attack success, minimal accuracy degradation, low runtime, and long-term persistence. These results demonstrate a more practical attack paradigm, offering new perspectives for designing robust defense strategies in federated learning systems

http://arxiv.org/abs/2508.15031
A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives. (54%)
Kaixiang Zhao; Lincan Li; Kaize Ding; Neil Zhenqiang Gong; Yue Zhao; Yushun Dong
Machine learning (ML) models have significantly grown in complexity and utility, driving advances across multiple domains. However, substantial computational resources and specialized expertise have historically restricted their wide adoption. Machine-Learning-as-a-Service (MLaaS) platforms have addressed these barriers by providing scalable, convenient, and affordable access to sophisticated ML models through user-friendly APIs. While this accessibility promotes widespread use of advanced ML capabilities, it also introduces vulnerabilities exploited through Model Extraction Attacks (MEAs). Recent studies have demonstrated that adversaries can systematically replicate a target model's functionality by interacting with publicly exposed interfaces, posing threats to intellectual property, privacy, and system security. In this paper, we offer a comprehensive survey of MEAs and corresponding defense strategies. We propose a novel taxonomy that classifies MEAs according to attack mechanisms, defense approaches, and computing environments. Our analysis covers various attack techniques, evaluates their effectiveness, and highlights challenges faced by existing defenses, particularly the critical trade-off between preserving model utility and ensuring security. We further assess MEAs within different computing paradigms and discuss their technical, ethical, legal, and societal implications, along with promising directions for future research. This systematic survey aims to serve as a valuable reference for researchers, practitioners, and policymakers engaged in AI security and privacy. Additionally, we maintain an online repository continuously updated with related literature at https://github.com/kzhao5/ModelExtractionPapers.

http://arxiv.org/abs/2509.00027
Mitigating Data Exfiltration Attacks through Layer-Wise Learning Rate Decay Fine-Tuning. (2%)
Elie Thellier; Huiyu Li; Nicholas Ayache; Hervé Delingette
Data lakes enable the training of powerful machine learning models on sensitive, high-value medical datasets, but also introduce serious privacy risks due to potential leakage of protected health information. Recent studies show adversaries can exfiltrate training data by embedding latent representations into model parameters or inducing memorization via multi-task learning. These attacks disguise themselves as benign utility models while enabling reconstruction of high-fidelity medical images, posing severe privacy threats with legal and ethical implications. In this work, we propose a simple yet effective mitigation strategy that perturbs model parameters at export time through fine-tuning with a decaying layer-wise learning rate to corrupt embedded data without degrading task performance. Evaluations on DermaMNIST, ChestMNIST, and MIMIC-CXR show that our approach maintains utility task performance, effectively disrupts state-of-the-art exfiltration attacks, outperforms prior defenses, and renders exfiltrated data unusable for training. Ablations and discussions on adaptive attacks highlight challenges and future directions. Our findings offer a practical defense against data leakage in data lake-trained models and centralized federated learning.

http://arxiv.org/abs/2508.14526
CoFacS -- Simulating a Complete Factory to Study the Security of Interconnected Production. (1%)
Stefan Lenz; David Schachtschneider; Simon Jonas; Liam Tirpitz; Sandra Geisler; Martin Henze
While the digitization of industrial factories provides tremendous improvements for the production of goods, it also renders such systems vulnerable to serious cyber-attacks. To research, test, and validate security measures protecting industrial networks against such cyber-attacks, the security community relies on testbeds to simulate industrial systems, as utilizing live systems endangers costly components or even human life. However, existing testbeds focus on individual parts of typically complex production lines in industrial factories. Consequently, the impact of cyber-attacks on industrial networks as well as the effectiveness of countermeasures cannot be evaluated in an end-to-end manner. To address this issue and facilitate research on novel security mechanisms, we present CoFacS, the first COmplete FACtory Simulation that replicates an entire production line and affords the integration of real-life industrial applications. To showcase that CoFacS accurately captures real-world behavior, we validate it against a physical model factory widely used in security research. We show that CoFacS has a maximum deviation of 0.11% to the physical reference, which enables us to study the impact of physical attacks or network-based cyber-attacks. Moreover, we highlight how CoFacS enables security research through two cases studies surrounding attack detection and the resilience of 5G-based industrial communication against jamming.

http://arxiv.org/abs/2508.15036
MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs. (1%)
Ruyi Ding; Tianhong Xu; Xinyi Shen; Aidong Adam Ding; Yunsi Fei
The transformer architecture has become a cornerstone of modern AI, fueling remarkable progress across applications in natural language processing, computer vision, and multimodal learning. As these models continue to scale explosively for performance, implementation efficiency remains a critical challenge. Mixture of Experts (MoE) architectures, selectively activating specialized subnetworks (experts), offer a unique balance between model accuracy and computational cost. However, the adaptive routing in MoE architectures, where input tokens are dynamically directed to specialized experts based on their semantic meaning inadvertently opens up a new attack surface for privacy breaches. These input-dependent activation patterns leave distinctive temporal and spatial traces in hardware execution, which adversaries could exploit to deduce sensitive user data. In this work, we propose MoEcho, discovering a side channel analysis based attack surface that compromises user privacy on MoE based systems. Specifically, in MoEcho, we introduce four novel architectural side channels on different computing platforms, including Cache Occupancy Channels and Pageout+Reload on CPUs, and Performance Counter and TLB Evict+Reload on GPUs, respectively. Exploiting these vulnerabilities, we propose four attacks that effectively breach user privacy in large language models (LLMs) and vision language models (VLMs) based on MoE architectures: Prompt Inference Attack, Response Reconstruction Attack, Visual Inference Attack, and Visual Reconstruction Attack. MoEcho is the first runtime architecture level security analysis of the popular MoE structure common in modern transformers, highlighting a serious security and privacy threat and calling for effective and timely safeguards when harnessing MoE based models for developing efficient large scale AI services.

http://arxiv.org/abs/2506.10459
Boosting Adversarial Transferability for Hyperspectral Image Classification Using 3D Structure-invariant Transformation and Weighted Intermediate Feature Divergence. (99%)
Chun Liu; Bingqian Zhu; Tao Xu; Zheng Zheng; Zheng Li; Wei Yang; Zhigang Han; Jiayao Wang
Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, which pose security challenges to hyperspectral image (HSI) classification based on DNNs. Numerous adversarial attack methods have been designed in the domain of natural images. However, different from natural images, HSIs contains high-dimensional rich spectral information, which presents new challenges for generating adversarial examples. Based on the specific characteristics of HSIs, this paper proposes a novel method to enhance the transferability of the adversarial examples for HSI classification using 3D structure-invariant transformation and weighted intermediate feature divergence. While keeping the HSIs structure invariant, the proposed method divides the image into blocks in both spatial and spectral dimensions. Then, various transformations are applied on each block to increase input diversity and mitigate the overfitting to substitute models. Moreover, a weighted intermediate feature divergence loss is also designed by leveraging the differences between the intermediate features of original and adversarial examples. It constrains the perturbation direction by enlarging the feature maps of the original examples, and assigns different weights to different feature channels to destroy the features that have a greater impact on HSI classification. Extensive experiments demonstrate that the adversarial examples generated by the proposed method achieve more effective adversarial transferability on three public HSI datasets. Furthermore, the method maintains robust attack performance even under defense strategies.

http://arxiv.org/abs/2508.13739
Enhancing Targeted Adversarial Attacks on Large Vision-Language Models through Intermediate Projector Guidance. (93%)
Yiming Cao; Yanjie Li; Kaisheng Liang; Yuni Lai; Bin Xiao
Targeted adversarial attacks are essential for proactively identifying security flaws in Vision-Language Models before real-world deployment. However, current methods perturb images to maximize global similarity with the target text or reference image at the encoder level, collapsing rich visual semantics into a single global vector. This limits attack granularity, hindering fine-grained manipulations such as modifying a car while preserving its background. Furthermore, these methods largely overlook the projector module, a critical semantic bridge between the visual encoder and the language model in VLMs, thereby failing to disrupt the full vision-language alignment pipeline within VLMs and limiting attack effectiveness. To address these issues, we propose the Intermediate Projector Guided Attack (IPGA), the first method to attack using the intermediate stage of the projector module, specifically the widely adopted Q-Former, which transforms global image embeddings into fine-grained visual features. This enables more precise control over adversarial perturbations by operating on semantically meaningful visual tokens rather than a single global representation. Specifically, IPGA leverages the Q-Former pretrained solely on the first vision-language alignment stage, without LLM fine-tuning, which improves both attack effectiveness and transferability across diverse VLMs. Furthermore, we propose Residual Query Alignment (RQA) to preserve unrelated visual content, thereby yielding more controlled and precise adversarial manipulations. Extensive experiments show that our attack method consistently outperforms existing methods in both standard global image captioning tasks and fine-grained visual question-answering tasks in black-box environment. Additionally, IPGA successfully transfers to multiple commercial VLMs, including Google Gemini and OpenAI GPT.

http://arxiv.org/abs/2508.13812
Timestep-Compressed Attack on Spiking Neural Networks through Timestep-Level Backpropagation. (93%)
Donghwa Kang; Doohyun Kim; Sang-Ki Ko; Jinkyu Lee; Hyeongboo Baek; Brent ByungHoon Kang
State-of-the-art (SOTA) gradient-based adversarial attacks on spiking neural networks (SNNs), which largely rely on extending FGSM and PGD frameworks, face a critical limitation: substantial attack latency from multi-timestep processing, rendering them infeasible for practical real-time applications. This inefficiency stems from their design as direct extensions of ANN paradigms, which fail to exploit key SNN properties. In this paper, we propose the timestep-compressed attack (TCA), a novel framework that significantly reduces attack latency. TCA introduces two components founded on key insights into SNN behavior. First, timestep-level backpropagation (TLBP) is based on our finding that global temporal information in backpropagation to generate perturbations is not critical for an attack's success, enabling per-timestep evaluation for early stopping. Second, adversarial membrane potential reuse (A-MPR) is motivated by the observation that initial timesteps are inefficiently spent accumulating membrane potential, a warm-up phase that can be pre-calculated and reused. Our experiments on VGG-11 and ResNet-17 with the CIFAR-10/100 and CIFAR10-DVS datasets show that TCA significantly reduces the required attack latency by up to 56.6% and 57.1% compared to SOTA methods in white-box and black-box settings, respectively, while maintaining a comparable attack success rate.

http://arxiv.org/abs/2508.13853
FedUP: Efficient Pruning-based Federated Unlearning for Model Poisoning Attacks. (83%)
Nicolò Romandini; Cristian Borcea; Rebecca Montanari; Luca Foschini
Federated Learning (FL) can be vulnerable to attacks, such as model poisoning, where adversaries send malicious local weights to compromise the global model. Federated Unlearning (FU) is emerging as a solution to address such vulnerabilities by selectively removing the influence of detected malicious contributors on the global model without complete retraining. However, unlike typical FU scenarios where clients are trusted and cooperative, applying FU with malicious and possibly colluding clients is challenging because their collaboration in unlearning their data cannot be assumed. This work presents FedUP, a lightweight FU algorithm designed to efficiently mitigate malicious clients' influence by pruning specific connections within the attacked model. Our approach achieves efficiency by relying only on clients' weights from the last training round before unlearning to identify which connections to inhibit. Isolating malicious influence is non-trivial due to overlapping updates from benign and malicious clients. FedUP addresses this by carefully selecting and zeroing the highest magnitude weights that diverge the most between the latest updates from benign and malicious clients while preserving benign information. FedUP is evaluated under a strong adversarial threat model, where up to 50%-1 of the clients could be malicious and have full knowledge of the aggregation process. We demonstrate the effectiveness, robustness, and efficiency of our solution through experiments across IID and Non-IID data, under label-flipping and backdoor attacks, and by comparing it with state-of-the-art (SOTA) FU solutions. In all scenarios, FedUP reduces malicious influence, lowering accuracy on malicious data to match that of a model retrained from scratch while preserving performance on benign data. FedUP achieves effective unlearning while consistently being faster and saving storage compared to the SOTA.

http://arxiv.org/abs/2508.14128
CCFC: Core & Core-Full-Core Dual-Track Defense for LLM Jailbreak Protection. (68%)
Jiaming Hu; Haoyu Wang; Debarghya Mukherjee; Ioannis Ch. Paschalidis
Jailbreak attacks pose a serious challenge to the safe deployment of large language models (LLMs). We introduce CCFC (Core & Core-Full-Core), a dual-track, prompt-level defense framework designed to mitigate LLMs' vulnerabilities from prompt injection and structure-aware jailbreak attacks. CCFC operates by first isolating the semantic core of a user query via few-shot prompting, and then evaluating the query using two complementary tracks: a core-only track to ignore adversarial distractions (e.g., toxic suffixes or prefix injections), and a core-full-core (CFC) track to disrupt the structural patterns exploited by gradient-based or edit-based attacks. The final response is selected based on a safety consistency check across both tracks, ensuring robustness without compromising on response quality. We demonstrate that CCFC cuts attack success rates by 50-75% versus state-of-the-art defenses against strong adversaries (e.g., DeepInception, GCG), without sacrificing fidelity on benign queries. Our method consistently outperforms state-of-the-art prompt-level defenses, offering a practical and effective solution for safer LLM deployment.

http://arxiv.org/abs/2508.14925
MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers. (38%)
Zhiqiang Wang; Yichao Gao; Yanting Wang; Suyuan Liu; Haifeng Sun; Haoran Cheng; Guanquan Shi; Haohua Du; Xiangyang Li
By providing a standardized interface for LLM agents to interact with external tools, the Model Context Protocol (MCP) is quickly becoming a cornerstone of the modern autonomous agent ecosystem. However, it creates novel attack surfaces due to untrusted external tools. While prior work has focused on attacks injected through external tool outputs, we investigate a more fundamental vulnerability: Tool Poisoning, where malicious instructions are embedded within a tool's metadata without execution. To date, this threat has been primarily demonstrated through isolated cases, lacking a systematic, large-scale evaluation.
  We introduce MCPTox, the first benchmark to systematically evaluate agent robustness against Tool Poisoning in realistic MCP settings. MCPTox is constructed upon 45 live, real-world MCP servers and 353 authentic tools. To achieve this, we design three distinct attack templates to generate a comprehensive suite of 1312 malicious test cases by few-shot learning, covering 10 categories of potential risks. Our evaluation on 20 prominent LLM agents setting reveals a widespread vulnerability to Tool Poisoning, with o1-mini, achieving an attack success rate of 72.8\%. We find that more capable models are often more susceptible, as the attack exploits their superior instruction-following abilities. Finally, the failure case analysis reveals that agents rarely refuse these attacks, with the highest refused rate (Claude-3.7-Sonnet) less than 3\%, demonstrating that existing safety alignment is ineffective against malicious actions that use legitimate tools for unauthorized operation. Our findings create a crucial empirical baseline for understanding and mitigating this widespread threat, and we release MCPTox for the development of verifiably safer AI agents. Our dataset is available at an anonymized repository: \textit{https://anonymous.4open.science/r/AAAI26-7C02}.

http://arxiv.org/abs/2508.13730
On the Security and Privacy of Federated Learning: A Survey with Attacks, Defenses, Frameworks, Applications, and Future Directions. (11%)
Daniel M. Jimenez-Gutierrez; Yelizaveta Falkouskaya; Jose L. Hernandez-Ramos; Aris Anagnostopoulos; Ioannis Chatzigiannakis; Andrea Vitaletti
Federated Learning (FL) is an emerging distributed machine learning paradigm enabling multiple clients to train a global model collaboratively without sharing their raw data. While FL enhances data privacy by design, it remains vulnerable to various security and privacy threats. This survey provides a comprehensive overview of more than 200 papers regarding the state-of-the-art attacks and defense mechanisms developed to address these challenges, categorizing them into security-enhancing and privacy-preserving techniques. Security-enhancing methods aim to improve FL robustness against malicious behaviors such as byzantine attacks, poisoning, and Sybil attacks. At the same time, privacy-preserving techniques focus on protecting sensitive data through cryptographic approaches, differential privacy, and secure aggregation. We critically analyze the strengths and limitations of existing methods, highlight the trade-offs between privacy, security, and model performance, and discuss the implications of non-IID data distributions on the effectiveness of these defenses. Furthermore, we identify open research challenges and future directions, including the need for scalable, adaptive, and energy-efficient solutions operating in dynamic and heterogeneous FL environments. Our survey aims to guide researchers and practitioners in developing robust and privacy-preserving FL systems, fostering advancements safeguarding collaborative learning frameworks' integrity and confidentiality.

http://arxiv.org/abs/2505.12332
VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloning. (11%)
Qianyue Hu; Junyan Wu; Wei Lu; Xiangyang Luo
Diffusion Models (DMs) have achieved remarkable success in realistic voice cloning (VC), while they also increase the risk of malicious misuse. Existing proactive defenses designed for traditional VC models aim to disrupt the forgery process, but they have been proven incompatible with DMs due to the intricate generative mechanisms of diffusion. To bridge this gap, we introduce VoiceCloak, a multi-dimensional proactive defense framework with the goal of obfuscating speaker identity and degrading perceptual quality in potential unauthorized VC. To achieve these goals, we conduct a focused analysis to identify specific vulnerabilities within DMs, allowing VoiceCloak to disrupt the cloning process by introducing adversarial perturbations into the reference audio. Specifically, to obfuscate speaker identity, VoiceCloak first targets speaker identity by distorting representation learning embeddings to maximize identity variation, which is guided by auditory perception principles. Additionally, VoiceCloak disrupts crucial conditional guidance processes, particularly attention context, thereby preventing the alignment of vocal characteristics that are essential for achieving convincing cloning. Then, to address the second objective, VoiceCloak introduces score magnitude amplification to actively steer the reverse trajectory away from the generation of high-quality speech. Noise-guided semantic corruption is further employed to disrupt structural speech semantics captured by DMs, degrading output quality. Extensive experiments highlight VoiceCloak's outstanding defense success rate against unauthorized diffusion-based voice cloning. Audio samples of VoiceCloak are available at https://voice-cloak.github.io/VoiceCloak/.

http://arxiv.org/abs/2508.13672
ITL-LIME: Instance-Based Transfer Learning for Enhancing Local Explanations in Low-Resource Data Settings. (4%)
Rehan Raza; Guanjin Wang; Kevin Wong; Hamid Laga; Marco Fisichella
Explainable Artificial Intelligence (XAI) methods, such as Local Interpretable Model-Agnostic Explanations (LIME), have advanced the interpretability of black-box machine learning models by approximating their behavior locally using interpretable surrogate models. However, LIME's inherent randomness in perturbation and sampling can lead to locality and instability issues, especially in scenarios with limited training data. In such cases, data scarcity can result in the generation of unrealistic variations and samples that deviate from the true data manifold. Consequently, the surrogate model may fail to accurately approximate the complex decision boundary of the original model. To address these challenges, we propose a novel Instance-based Transfer Learning LIME framework (ITL-LIME) that enhances explanation fidelity and stability in data-constrained environments. ITL-LIME introduces instance transfer learning into the LIME framework by leveraging relevant real instances from a related source domain to aid the explanation process in the target domain. Specifically, we employ clustering to partition the source domain into clusters with representative prototypes. Instead of generating random perturbations, our method retrieves pertinent real source instances from the source cluster whose prototype is most similar to the target instance. These are then combined with the target instance's neighboring real instances. To define a compact locality, we further construct a contrastive learning-based encoder as a weighting mechanism to assign weights to the instances from the combined set based on their proximity to the target instance. Finally, these weighted source and target instances are used to train the surrogate model for explanation purposes.

http://arxiv.org/abs/2508.13481
Enhancing Robustness of Implicit Neural Representations Against Weight Perturbations. (2%)
Wenyong Zhou; Yuxin Cheng; Zhengwu Liu; Taiqiang Wu; Chen Zhang; Ngai Wong
Implicit Neural Representations (INRs) encode discrete signals in a continuous manner using neural networks, demonstrating significant value across various multimedia applications. However, the vulnerability of INRs presents a critical challenge for their real-world deployments, as the network weights might be subjected to unavoidable perturbations. In this work, we investigate the robustness of INRs for the first time and find that even minor perturbations can lead to substantial performance degradation in the quality of signal reconstruction. To mitigate this issue, we formulate the robustness problem in INRs by minimizing the difference between loss with and without weight perturbations. Furthermore, we derive a novel robust loss function to regulate the gradient of the reconstruction loss with respect to weights, thereby enhancing the robustness. Extensive experiments on reconstruction tasks across multiple modalities demonstrate that our method achieves up to a 7.5~dB improvement in peak signal-to-noise ratio (PSNR) values compared to original INRs under noisy conditions.

http://arxiv.org/abs/2508.13579
Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance. (1%)
Yue Fang; Yuxin Guo; Jiaran Gao; Hongxin Ding; Xinke Jiang; Weibin Liao; Yongxin Xu; Yinghao Zhu; Zhibang Yang; Liantao Ma; Junfeng Zhao; Yasha Wang
Improving large language models (LLMs) for electronic health record (EHR) reasoning is essential for enabling accurate and generalizable clinical predictions. While LLMs excel at medical text understanding, they underperform on EHR-based prediction tasks due to challenges in modeling temporally structured, high-dimensional data. Existing approaches often rely on hybrid paradigms, where LLMs serve merely as frozen prior retrievers while downstream deep learning (DL) models handle prediction, failing to improve the LLM's intrinsic reasoning capacity and inheriting the generalization limitations of DL models. To this end, we propose EAG-RL, a novel two-stage training framework designed to intrinsically enhance LLMs' EHR reasoning ability through expert attention guidance, where expert EHR models refer to task-specific DL models trained on EHR data. Concretely, EAG-RL first constructs high-quality, stepwise reasoning trajectories using expert-guided Monte Carlo Tree Search to effectively initialize the LLM's policy. Then, EAG-RL further optimizes the policy via reinforcement learning by aligning the LLM's attention with clinically salient features identified by expert EHR models. Extensive experiments on two real-world EHR datasets show that EAG-RL improves the intrinsic EHR reasoning ability of LLMs by an average of 14.62%, while also enhancing robustness to feature perturbations and generalization to unseen clinical domains. These results demonstrate the practical potential of EAG-RL for real-world deployment in clinical prediction tasks. Our code have been available at https://github.com/devilran6/EAG-RL.

http://arxiv.org/abs/2508.13309
DAASH: A Meta-Attack Framework for Synthesizing Effective and Stealthy Adversarial Examples. (99%)
Abdullah Al Nomaan Nafi; Habibur Rahaman; Zafaryab Haider; Tanzim Mahfuz; Fnu Suya; Swarup Bhunia; Prabuddha Chakraborty
Numerous techniques have been proposed for generating adversarial examples in white-box settings under strict Lp-norm constraints. However, such norm-bounded examples often fail to align well with human perception, and only recently have a few methods begun specifically exploring perceptually aligned adversarial examples. Moreover, it remains unclear whether insights from Lp-constrained attacks can be effectively leveraged to improve perceptual efficacy. In this paper, we introduce DAASH, a fully differentiable meta-attack framework that generates effective and perceptually aligned adversarial examples by strategically composing existing Lp-based attack methods. DAASH operates in a multi-stage fashion: at each stage, it aggregates candidate adversarial examples from multiple base attacks using learned, adaptive weights and propagates the result to the next stage. A novel meta-loss function guides this process by jointly minimizing misclassification loss and perceptual distortion, enabling the framework to dynamically modulate the contribution of each base attack throughout the stages. We evaluate DAASH on adversarially trained models across CIFAR-10, CIFAR-100, and ImageNet. Despite relying solely on Lp-constrained based methods, DAASH significantly outperforms state-of-the-art perceptual attacks such as AdvAD -- achieving higher attack success rates (e.g., 20.63\% improvement) and superior visual quality, as measured by SSIM, LPIPS, and FID (improvements $\approx$ of 11, 0.015, and 5.7, respectively). Furthermore, DAASH generalizes well to unseen defenses, making it a practical and strong baseline for evaluating robustness without requiring handcrafted adaptive attacks for each new defense.

http://arxiv.org/abs/2504.03782
Deep Positive-Negative Prototypes for Adversarially Robust Discriminative Prototypical Learning. (99%)
Ramin Zarei Sabzevar; Hamed Mohammadzadeh; Tahmineh Tavakoli; Ahad Harati
Despite the advantages of discriminative prototype-based methods, their role in adversarial robustness remains underexplored. Meanwhile, current adversarial training methods predominantly focus on robustness against adversarial attacks without explicitly leveraging geometric structures in the latent space, usually resulting in reduced accuracy on the original clean data. We propose a novel framework named Adversarially trained Deep Positive-Negative Prototypes (Adv-DPNP), which integrates discriminative prototype-based learning with adversarial training. Adv-DPNP uses unified class prototypes that serve as both classifier weights and robust anchors in the latent space. Moreover, a novel dual-branch training mechanism maintains stable prototypes by updating them exclusively with clean data, while the feature extractor is trained on both clean and adversarial inputs to increase invariance to adversarial perturbations. In addition, we use a composite loss that combines positive-prototype alignment, negative-prototype repulsion, and consistency regularization to further enhance discrimination, adversarial robustness, and clean accuracy. Extensive experiments on standard benchmarks (CIFAR-10/100 and SVHN) confirm that Adv-DPNP improves clean accuracy over state-of-the-art defenses and baseline methods, while maintaining competitive or superior robustness under a suite of widely used attacks, including FGSM, PGD, C\&W, and AutoAttack. We also evaluate robustness to common corruptions on CIFAR-10-C, where Adv-DPNP achieves the highest average accuracy across severities and corruption types. Additionally, we provide an in-depth analysis of the discriminative quality of the learned feature representations, highlighting the effectiveness of Adv-DPNP in maintaining compactness and clear separation in the latent space.

http://arxiv.org/abs/2508.07795
Boosting Active Defense Persistence: A Two-Stage Defense Framework Combining Interruption and Poisoning Against Deepfake. (92%)
Hongrui Zheng; Yuezun Li; Liejun Wang; Yunfeng Diao; Zhiqing Guo
Active defense strategies have been developed to counter the threat of deepfake technology. However, a primary challenge is their lack of persistence, as their effectiveness is often short-lived. Attackers can bypass these defenses by simply collecting protected samples and retraining their models. This means that static defenses inevitably fail when attackers retrain their models, which severely limits practical use. We argue that an effective defense not only distorts forged content but also blocks the model's ability to adapt, which occurs when attackers retrain their models on protected images. To achieve this, we propose an innovative Two-Stage Defense Framework (TSDF). Benefiting from the intensity separation mechanism designed in this paper, the framework uses dual-function adversarial perturbations to perform two roles. First, it can directly distort the forged results. Second, it acts as a poisoning vehicle that disrupts the data preparation process essential for an attacker's retraining pipeline. By poisoning the data source, TSDF aims to prevent the attacker's model from adapting to the defensive perturbations, thus ensuring the defense remains effective long-term. Comprehensive experiments show that the performance of traditional interruption methods degrades sharply when it is subjected to adversarial retraining. However, our framework shows a strong dual defense capability, which can improve the persistence of active defense. Our code will be available at https://github.com/vpsg-research/TSDF.

http://arxiv.org/abs/2503.12339
Augmented Adversarial Trigger Learning. (87%)
Zhe Wang; Yanjun Qi
Gradient optimization-based adversarial attack methods automate the learning of adversarial triggers to generate jailbreak prompts or leak system prompts. In this work, we take a closer look at the optimization objective of adversarial trigger learning and propose ATLA: Adversarial Trigger Learning with Augmented objectives. ATLA improves the negative log-likelihood loss used by previous studies into a weighted loss formulation that encourages the learned adversarial triggers to optimize more towards response format tokens. This enables ATLA to learn an adversarial trigger from just one query-response pair and the learned trigger generalizes well to other similar queries. We further design a variation to augment trigger optimization with an auxiliary loss that suppresses evasive responses. We showcase how to use ATLA to learn adversarial suffixes jailbreaking LLMs and to extract hidden system prompts. Empirically we demonstrate that ATLA consistently outperforms current state-of-the-art techniques, achieving nearly 100% success in attacking while requiring 80% fewer queries. ATLA learned jailbreak suffixes demonstrate high generalization to unseen queries and transfer well to new LLMs. We released our code https://github.com/QData/ALTA_Augmented_Adversarial_Trigger_Learning

http://arxiv.org/abs/2508.13316
Efficient Constraint-Aware Flow Matching via Randomized Exploration. (83%)
Zhengyan Huan; Jacob Boerma; Li-Ping Liu; Shuchin Aeron
We consider the problem of generating samples via Flow Matching (FM) with an additional requirement that the generated samples must satisfy given constraints. We consider two scenarios, viz.: (a) when a differentiable distance function to the constraint set is given, and (b) when the constraint set is only available via queries to a membership oracle. For case (a), we propose a simple adaptation of the FM objective with an additional term that penalizes the distance between the constraint set and the generated samples. For case (b), we propose to employ randomization and learn a mean flow that is numerically shown to have a high likelihood of satisfying the constraints. This approach deviates significantly from existing works that require simple convex constraints, knowledge of a barrier function, or a reflection mechanism to constrain the probability flow. Furthermore, in the proposed setting we show that a two-stage approach, where both stages approximate the same original flow but with only the second stage probing the constraints via randomization, is more computationally efficient. Through several synthetic cases of constrained generation, we numerically show that the proposed approaches achieve significant gains in terms of constraint satisfaction while matching the target distributions. As a showcase for a practical oracle-based constraint, we show how our approach can be used for training an adversarial example generator, using queries to a hard-label black-box classifier. We conclude with several future research directions. Our code is available at https://github.com/ZhengyanHuan/FM-RE.

http://arxiv.org/abs/2508.13048
MAJIC: Markovian Adaptive Jailbreaking via Iterative Composition of Diverse Innovative Strategies. (68%)
Weiwei Qi; Shuo Shao; Wei Gu; Tianhang Zheng; Puning Zhao; Zhan Qin; Kui Ren
Large Language Models (LLMs) have exhibited remarkable capabilities but remain vulnerable to jailbreaking attacks, which can elicit harmful content from the models by manipulating the input prompts. Existing black-box jailbreaking techniques primarily rely on static prompts crafted with a single, non-adaptive strategy, or employ rigid combinations of several underperforming attack methods, which limits their adaptability and generalization. To address these limitations, we propose MAJIC, a Markovian adaptive jailbreaking framework that attacks black-box LLMs by iteratively combining diverse innovative disguise strategies. MAJIC first establishes a ``Disguise Strategy Pool'' by refining existing strategies and introducing several innovative approaches. To further improve the attack performance and efficiency, MAJIC formulate the sequential selection and fusion of strategies in the pool as a Markov chain. Under this formulation, MAJIC initializes and employs a Markov matrix to guide the strategy composition, where transition probabilities between strategies are dynamically adapted based on attack outcomes, thereby enabling MAJIC to learn and discover effective attack pathways tailored to the target model. Our empirical results demonstrate that MAJIC significantly outperforms existing jailbreak methods on prominent models such as GPT-4o and Gemini-2.0-flash, achieving over 90\% attack success rate with fewer than 15 queries per attempt on average.

http://arxiv.org/abs/2508.12852
RoTO: Robust Topology Obfuscation Against Tomography Inference Attacks. (56%)
Chengze Du; Heng Xu; Zhiwei Yu; Ying Zhou; Zili Meng; Jialong Li
Tomography inference attacks aim to reconstruct network topology by analyzing end-to-end probe delays. Existing defenses mitigate these attacks by manipulating probe delays to mislead inference, but rely on two strong assumptions: (i) probe packets can be perfectly detected and altered, and (ii) attackers use known, fixed inference algorithms. These assumptions often break in practice, leading to degraded defense performance under detection errors or adaptive adversaries. We present RoTO, a robust topology obfuscation scheme that eliminates both assumptions by modeling uncertainty in attacker-observed delays through a distributional formulation. RoTO casts the defense objective as a min-max optimization problem that maximizes expected topological distortion across this uncertainty set, without relying on perfect probe control or specific attacker models. To approximate attacker behavior, RoTO leverages graph neural networks for inference simulation and adversarial training. We also derive an upper bound on attacker success probability, and demonstrate that our approach enhances topology obfuscation performance through the optimization of this upper bound. Experimental results show that RoTO outperforms existing defense methods, achieving average improvements of 34% in structural similarity and 42.6% in link distance while maintaining strong robustness and concealment capabilities.

http://arxiv.org/abs/2505.20841
Concealment of Intent: A Game-Theoretic Analysis. (50%)
Xinbo Wu; Abhishek Umrawal; Lav R. Varshney
As large language models (LLMs) grow more capable, concerns about their safe deployment have also grown. Although alignment mechanisms have been introduced to deter misuse, they remain vulnerable to carefully designed adversarial prompts. In this work, we present a scalable attack strategy: intent-hiding adversarial prompting, which conceals malicious intent through the composition of skills. We develop a game-theoretic framework to model the interaction between such attacks and defense systems that apply both prompt and response filtering. Our analysis identifies equilibrium points and reveals structural advantages for the attacker. To counter these threats, we propose and analyze a defense mechanism tailored to intent-hiding attacks. Empirically, we validate the attack's effectiveness on multiple real-world LLMs across a range of malicious behaviors, demonstrating clear advantages over existing adversarial prompting techniques.

http://arxiv.org/abs/2502.15320
Adversarially-Robust Gossip Algorithms for Approximate Quantile and Mean Computations. (13%)
Bernhard Haeupler; Marc Kaufmann; Raghu Raman Ravi; Ulysse Schaller
This paper presents gossip algorithms for aggregation tasks that demonstrate both robustness to adversarial corruptions of any order of magnitude and optimality across a substantial range of these corruption levels. Gossip algorithms distribute information in a scalable and efficient way by having random pairs of nodes exchange small messages. Value aggregation problems are of particular interest in this setting, as they occur frequently in practice, and many elegant algorithms have been proposed for computing aggregates and statistics such as averages and quantiles. An important and well-studied advantage of gossip algorithms is their robustness to message delays, network churn, and unreliable message transmissions. However, these crucial robustness guarantees only hold if all nodes follow the protocol and no messages are corrupted. In this paper, we remedy this by providing a framework to model both adversarial participants and message corruptions in gossip-style communications by allowing an adversary to control a small fraction of the nodes or corrupt messages arbitrarily. Despite this very powerful and general corruption model, we show that robust gossip algorithms can be designed for many important aggregation problems. Our algorithms guarantee that almost all nodes converge to an approximately correct answer with optimal efficiency and essentially as fast as without corruptions. The design of adversarially-robust gossip algorithms poses completely new challenges. Despite this, our algorithms remain very simple variations of known non-robust algorithms with often only subtle changes to avoid non-compliant nodes gaining too much influence over outcomes. While our algorithms remain simple, their analysis is much more complex and often requires a completely different approach than the non-adversarial setting.

http://arxiv.org/abs/2508.08789
Never Compromise to Vulnerabilities: A Comprehensive Survey on AI Governance. (8%)
Yuchu Jiang; Jian Zhao; Yuchen Yuan; Tianle Zhang; Yao Huang; Yanghao Zhang; Yan Wang; Yanshu Li; Xizhong Guo; Yusheng Zhao; Jun Zhang; Zhi Zhang; Xiaojian Lin; Yixiu Zou; Haoxuan Ma; Yuhu Shang; Yuzhi Hu; Keshu Cai; Ruochen Zhang; Boyuan Chen; Yilan Gao; Ziheng Jiao; Yi Qin; Shuangjun Du; Xiao Tong; Zhekun Liu; Yu Chen; Xuankun Rong; Rui Wang; Yejie Zheng; Zhaoxin Fan; Murat Sensoy; Hongyuan Zhang; Pan Zhou; Lei Jin; Hao Zhao; Xu Yang; Jiaojiao Zhao; Jianshu Li; Joey Tianyi Zhou; Zhi-Qi Cheng; Longtao Huang; Zhiyi Liu; Zheng Zhu; Jianan Li; Gang Wang; Qi Li; Xu-Yao Zhang; Yaodong Yang; Mang Ye; Wenqi Ren; Zhaofeng He; Hang Su; Rongrong Ni; Liping Jing; Xingxing Wei; Junliang Xing; Massimo Alioto; Shengmei Shen; Petia Radeva; Dacheng Tao; Ya-Qin Zhang; Shuicheng Yan; Chi Zhang; Zhongjiang He; Xuelong Li
The rapid advancement of AI has expanded its capabilities across domains, yet introduced critical technical vulnerabilities, such as algorithmic bias and adversarial sensitivity, that pose significant societal risks, including misinformation, inequity, security breaches, physical harm, and eroded public trust. These challenges highlight the urgent need for robust AI governance. We propose a comprehensive framework integrating technical and societal dimensions, structured around three interconnected pillars: Intrinsic Security (system reliability), Derivative Security (real-world harm mitigation), and Social Ethics (value alignment and accountability). Uniquely, our approach unifies technical methods, emerging evaluation benchmarks, and policy insights to promote transparency, accountability, and trust in AI systems. Through a systematic review of over 300 studies, we identify three core challenges: (1) the generalization gap, where defenses fail against evolving threats; (2) inadequate evaluation protocols that overlook real-world risks; and (3) fragmented regulations leading to inconsistent oversight. These shortcomings stem from treating governance as an afterthought, rather than a foundational design principle, resulting in reactive, siloed efforts that fail to address the interdependence of technical integrity and societal trust. To overcome this, we present an integrated research agenda that bridges technical rigor with social responsibility. Our framework offers actionable guidance for researchers, engineers, and policymakers to develop AI systems that are not only robust and secure but also ethically aligned and publicly trustworthy. The accompanying repository is available at https://github.com/ZTianle/Awesome-AI-SG.

http://arxiv.org/abs/2412.05934
Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models. (8%)
Ma Teng; Jia Xiaojun; Duan Ranjie; Li Xinfeng; Huang Yihao; Jia Xiaoshuang; Chu Zhixuan; Ren Wenqi
With the rapid advancement of multimodal large language models (MLLMs), concerns regarding their security have increasingly captured the attention of both academia and industry. Although MLLMs are vulnerable to jailbreak attacks, designing effective jailbreak attacks poses unique challenges, especially given the highly constrained adversarial capabilities in real-world deployment scenarios. Previous works concentrate risks into a single modality, resulting in limited jailbreak performance. In this paper, we propose a heuristic-induced multimodal risk distribution jailbreak attack method, called HIMRD, which is black-box and consists of two elements: multimodal risk distribution strategy and heuristic-induced search strategy. The multimodal risk distribution strategy is used to distribute harmful semantics into multiple modalities to effectively circumvent the single-modality protection mechanisms of MLLMs. The heuristic-induced search strategy identifies two types of prompts: the understanding-enhancing prompt, which helps MLLMs reconstruct the malicious prompt, and the inducing prompt, which increases the likelihood of affirmative outputs over refusals, enabling a successful jailbreak attack. HIMRD achieves an average attack success rate (ASR) of 90% across seven open-source MLLMs and an average ASR of around 68% in three closed-source MLLMs. HIMRD reveals cross-modal security vulnerabilities in current MLLMs and underscores the imperative for developing defensive strategies to mitigate such emerging risks. Code is available at https://github.com/MaTengSYSU/HIMRD-jailbreak.

http://arxiv.org/abs/2508.12538
Systematic Analysis of MCP Security. (4%)
Yongjian Guo; Puzhuo Liu; Wanlun Ma; Zehang Deng; Xiaogang Zhu; Peng Di; Xi Xiao; Sheng Wen
The Model Context Protocol (MCP) has emerged as a universal standard that enables AI agents to seamlessly connect with external tools, significantly enhancing their functionality. However, while MCP brings notable benefits, it also introduces significant vulnerabilities, such as Tool Poisoning Attacks (TPA), where hidden malicious instructions exploit the sycophancy of large language models (LLMs) to manipulate agent behavior. Despite these risks, current academic research on MCP security remains limited, with most studies focusing on narrow or qualitative analyses that fail to capture the diversity of real-world threats. To address this gap, we present the MCP Attack Library (MCPLIB), which categorizes and implements 31 distinct attack methods under four key classifications: direct tool injection, indirect tool injection, malicious user attacks, and LLM inherent attack. We further conduct a quantitative analysis of the efficacy of each attack. Our experiments reveal key insights into MCP vulnerabilities, including agents' blind reliance on tool descriptions, sensitivity to file-based attacks, chain attacks exploiting shared context, and difficulty distinguishing external data from executable commands. These insights, validated through attack experiments, underscore the urgency for robust defense strategies and informed MCP design. Our contributions include 1) constructing a comprehensive MCP attack taxonomy, 2) introducing a unified attack framework MCPLIB, and 3) conducting empirical vulnerability analysis to enhance MCP security mechanisms. This work provides a foundational framework, supporting the secure evolution of MCP ecosystems.

http://arxiv.org/abs/2508.09288
Can AI Keep a Secret? Contextual Integrity Verification: A Provable Security Architecture for LLMs. (1%)
Aayush Gupta
Large language models (LLMs) remain acutely vulnerable to prompt injection and related jailbreak attacks; heuristic guardrails (rules, filters, LLM judges) are routinely bypassed. We present Contextual Integrity Verification (CIV), an inference-time security architecture that attaches cryptographically signed provenance labels to every token and enforces a source-trust lattice inside the transformer via a pre-softmax hard attention mask (with optional FFN/residual gating). CIV provides deterministic, per-token non-interference guarantees on frozen models: lower-trust tokens cannot influence higher-trust representations. On benchmarks derived from recent taxonomies of prompt-injection vectors (Elite-Attack + SoK-246), CIV attains 0% attack success rate under the stated threat model while preserving 93.1% token-level similarity and showing no degradation in model perplexity on benign tasks; we note a latency overhead attributable to a non-optimized data path. Because CIV is a lightweight patch -- no fine-tuning required -- we demonstrate drop-in protection for Llama-3-8B and Mistral-7B. We release a reference implementation, an automated certification harness, and the Elite-Attack corpus to support reproducible research.

http://arxiv.org/abs/2508.12672
Robust Federated Learning under Adversarial Attacks via Loss-Based Client Clustering. (1%)
Emmanouil Kritharakis; Dusan Jakovetic; Antonios Makris; Konstantinos Tserpes
Federated Learning (FL) enables collaborative model training across multiple clients without sharing private data. We consider FL scenarios wherein FL clients are subject to adversarial (Byzantine) attacks, while the FL server is trusted (honest) and has a trustworthy side dataset. This may correspond to, e.g., cases where the server possesses trusted data prior to federation, or to the presence of a trusted client that temporarily assumes the server role. Our approach requires only two honest participants, i.e., the server and one client, to function effectively, without prior knowledge of the number of malicious clients. Theoretical analysis demonstrates bounded optimality gaps even under strong Byzantine attacks. Experimental results show that our algorithm significantly outperforms standard and robust FL baselines such as Mean, Trimmed Mean, Median, Krum, and Multi-Krum under various attack strategies including label flipping, sign flipping, and Gaussian noise addition across MNIST, FMNIST, and CIFAR-10 benchmarks using the Flower framework.

http://arxiv.org/abs/2508.07402
ForensicsSAM: Toward Robust and Unified Image Forgery Detection and Localization Resisting to Adversarial Attack. (99%)
Rongxuan Peng; Shunquan Tan; Chenqi Kong; Anwei Luo; Alex C. Kot; Jiwu Huang
Parameter-efficient fine-tuning (PEFT) has emerged as a popular strategy for adapting large vision foundation models, such as the Segment Anything Model (SAM) and LLaVA, to downstream tasks like image forgery detection and localization (IFDL). However, existing PEFT-based approaches overlook their vulnerability to adversarial attacks. In this paper, we show that highly transferable adversarial images can be crafted solely via the upstream model, without accessing the downstream model or training data, significantly degrading the IFDL performance. To address this, we propose ForensicsSAM, a unified IFDL framework with built-in adversarial robustness. Our design is guided by three key ideas: (1) To compensate for the lack of forgery-relevant knowledge in the frozen image encoder, we inject forgery experts into each transformer block to enhance its ability to capture forgery artifacts. These forgery experts are always activated and shared across any input images. (2) To detect adversarial images, we design an light-weight adversary detector that learns to capture structured, task-specific artifact in RGB domain, enabling reliable discrimination across various attack methods. (3) To resist adversarial attacks, we inject adversary experts into the global attention layers and MLP modules to progressively correct feature shifts induced by adversarial noise. These adversary experts are adaptively activated by the adversary detector, thereby avoiding unnecessary interference with clean images. Extensive experiments across multiple benchmarks demonstrate that ForensicsSAM achieves superior resistance to various adversarial attack methods, while also delivering state-of-the-art performance in image-level forgery detection and pixel-level forgery localization. The resource is available at https://github.com/siriusPRX/ForensicsSAM.

http://arxiv.org/abs/2508.12384
ViT-EnsembleAttack: Augmenting Ensemble Models for Stronger Adversarial Transferability in Vision Transformers. (98%)
Hanwen Cao; Haobo Lu; Xiaosen Wang; Kun He
Ensemble-based attacks have been proven to be effective in enhancing adversarial transferability by aggregating the outputs of models with various architectures. However, existing research primarily focuses on refining ensemble weights or optimizing the ensemble path, overlooking the exploration of ensemble models to enhance the transferability of adversarial attacks. To address this gap, we propose applying adversarial augmentation to the surrogate models, aiming to boost overall generalization of ensemble models and reduce the risk of adversarial overfitting. Meanwhile, observing that ensemble Vision Transformers (ViTs) gain less attention, we propose ViT-EnsembleAttack based on the idea of model adversarial augmentation, the first ensemble-based attack method tailored for ViTs to the best of our knowledge. Our approach generates augmented models for each surrogate ViT using three strategies: Multi-head dropping, Attention score scaling, and MLP feature mixing, with the associated parameters optimized by Bayesian optimization. These adversarially augmented models are ensembled to generate adversarial examples. Furthermore, we introduce Automatic Reweighting and Step Size Enlargement modules to boost transferability. Extensive experiments demonstrate that ViT-EnsembleAttack significantly enhances the adversarial transferability of ensemble-based attacks on ViTs, outperforming existing methods by a substantial margin. Code is available at https://github.com/Trustworthy-AI-Group/TransferAttack.

http://arxiv.org/abs/2508.12430
Adversarial Attacks on VQA-NLE: Exposing and Alleviating Inconsistencies in Visual Question Answering Explanations. (33%)
Yahsin Yeh; Yilun Wu; Bokai Ruan; Honghan Shuai
Natural language explanations in visual question answering (VQA-NLE) aim to make black-box models more transparent by elucidating their decision-making processes. However, we find that existing VQA-NLE systems can produce inconsistent explanations and reach conclusions without genuinely understanding the underlying context, exposing weaknesses in either their inference pipeline or explanation-generation mechanism. To highlight these vulnerabilities, we not only leverage an existing adversarial strategy to perturb questions but also propose a novel strategy that minimally alters images to induce contradictory or spurious outputs. We further introduce a mitigation method that leverages external knowledge to alleviate these inconsistencies, thereby bolstering model robustness. Extensive evaluations on two standard benchmarks and two widely used VQA-NLE models underscore the effectiveness of our attacks and the potential of knowledge-based defenses, ultimately revealing pressing security and reliability concerns in current VQA-NLE systems.

http://arxiv.org/abs/2508.12132
TriQDef: Disrupting Semantic and Gradient Alignment to Prevent Adversarial Patch Transferability in Quantized Neural Networks. (98%)
Amira Guesmi; Bassem Ouni; Muhammad Shafique
Quantized Neural Networks (QNNs) are increasingly deployed in edge and resource-constrained environments due to their efficiency in computation and memory usage. While shown to distort the gradient landscape and weaken conventional pixel-level attacks, it provides limited robustness against patch-based adversarial attacks-localized, high-saliency perturbations that remain surprisingly transferable across bit-widths. Existing defenses either overfit to fixed quantization settings or fail to address this cross-bit generalization vulnerability. We introduce \textbf{TriQDef}, a tri-level quantization-aware defense framework designed to disrupt the transferability of patch-based adversarial attacks across QNNs. TriQDef consists of: (1) a Feature Disalignment Penalty (FDP) that enforces semantic inconsistency by penalizing perceptual similarity in intermediate representations; (2) a Gradient Perceptual Dissonance Penalty (GPDP) that explicitly misaligns input gradients across bit-widths by minimizing structural and directional agreement via Edge IoU and HOG Cosine metrics; and (3) a Joint Quantization-Aware Training Protocol that unifies these penalties within a shared-weight training scheme across multiple quantization levels. Extensive experiments on CIFAR-10 and ImageNet demonstrate that TriQDef reduces Attack Success Rates (ASR) by over 40\% on unseen patch and quantization combinations, while preserving high clean accuracy. Our findings underscore the importance of disrupting both semantic and perceptual gradient alignment to mitigate patch transferability in QNNs.

http://arxiv.org/abs/2508.12072
Mitigating Jailbreaks with Intent-Aware LLMs. (83%)
Wei Jie Yeo; Ranjan Satapathy; Erik Cambria
Despite extensive safety-tuning, large language models (LLMs) remain vulnerable to jailbreak attacks via adversarially crafted instructions, reflecting a persistent trade-off between safety and task performance. In this work, we propose Intent-FT, a simple and lightweight fine-tuning approach that explicitly trains LLMs to infer the underlying intent of an instruction before responding. By fine-tuning on a targeted set of adversarial instructions, Intent-FT enables LLMs to generalize intent deduction to unseen attacks, thereby substantially improving their robustness. We comprehensively evaluate both parametric and non-parametric attacks across open-source and proprietary models, considering harmfulness from attacks, utility, over-refusal, and impact against white-box threats. Empirically, Intent-FT consistently mitigates all evaluated attack categories, with no single attack exceeding a 50\% success rate -- whereas existing defenses remain only partially effective. Importantly, our method preserves the model's general capabilities and reduces excessive refusals on benign instructions containing superficially harmful keywords. Furthermore, models trained with Intent-FT accurately identify hidden harmful intent in adversarial attacks, and these learned intentions can be effectively transferred to enhance vanilla model defenses.

http://arxiv.org/abs/2508.11854
ComplicitSplat: Downstream Models are Vulnerable to Blackbox Attacks by 3D Gaussian Splat Camouflages. (68%)
Matthew Hull; Haoyang Yang; Pratham Mehta; Mansi Phute; Aeree Cho; Haorang Wang; Matthew Lau; Wenke Lee; Wilian Lunardi; Martin Andreoni; Polo Chau
As 3D Gaussian Splatting (3DGS) gains rapid adoption in safety-critical tasks for efficient novel-view synthesis from static images, how might an adversary tamper images to cause harm? We introduce ComplicitSplat, the first attack that exploits standard 3DGS shading methods to create viewpoint-specific camouflage - colors and textures that change with viewing angle - to embed adversarial content in scene objects that are visible only from specific viewpoints and without requiring access to model architecture or weights. Our extensive experiments show that ComplicitSplat generalizes to successfully attack a variety of popular detector - both single-stage, multi-stage, and transformer-based models on both real-world capture of physical objects and synthetic scenes. To our knowledge, this is the first black-box attack on downstream object detectors using 3DGS, exposing a novel safety risk for applications like autonomous navigation and other mission-critical robotic systems.

http://arxiv.org/abs/2411.01077
Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection. (22%)
Zhipeng Wei; Yuqi Liu; N. Benjamin Erichson
Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another LLM as a Judge to evaluate the harmfulness of generated text. However, we reveal that these Judge LLMs are vulnerable to token segmentation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accuracy and allowing harmful content to be misclassified as safe. In this paper, we introduce Emoji Attack, a novel strategy that amplifies existing jailbreak prompts by exploiting token segmentation bias. Our method leverages in-context learning to systematically insert emojis into text before it is evaluated by a Judge LLM, inducing embedding distortions that significantly lower the likelihood of detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambiguity, making them particularly effective in this attack. Through experiments on state-of-the-art Judge LLMs, we demonstrate that Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.

http://arxiv.org/abs/2508.11959
Rigorous Feature Importance Scores based on Shapley Value and Banzhaf Index. (22%)
Xuanxiang Huang; Olivier Létoffé; Joao Marques-Silva
Feature attribution methods based on game theory are ubiquitous in the field of eXplainable Artificial Intelligence (XAI). Recent works proposed rigorous feature attribution using logic-based explanations, specifically targeting high-stakes uses of machine learning (ML) models. Typically, such works exploit weak abductive explanation (WAXp) as the characteristic function to assign importance to features. However, one possible downside is that the contribution of non-WAXp sets is neglected. In fact, non-WAXp sets can also convey important information, because of the relationship between formal explanations (XPs) and adversarial examples (AExs). Accordingly, this paper leverages Shapley value and Banzhaf index to devise two novel feature importance scores. We take into account non-WAXp sets when computing feature contribution, and the novel scores quantify how effective each feature is at excluding AExs. Furthermore, the paper identifies properties and studies the computational complexity of the proposed scores.

http://arxiv.org/abs/2508.13214
Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions. (12%)
Xuyang Guo; Zekai Huang; Zhao Song; Jiahao Zhang
Large Language Models (LLMs) have recently demonstrated strong emergent abilities in complex reasoning and zero-shot generalization, showing unprecedented potential for LLM-as-a-judge applications in education, peer review, and data quality evaluation. However, their robustness under prompt injection attacks, where malicious instructions are embedded into the content to manipulate outputs, remains a significant concern. In this work, we explore a frustratingly simple yet effective attack setting to test whether LLMs can be easily misled. Specifically, we evaluate LLMs on basic arithmetic questions (e.g., "What is 3 + 2?") presented as either multiple-choice or true-false judgment problems within PDF files, where hidden prompts are injected into the file. Our results reveal that LLMs are indeed vulnerable to such hidden prompt injection attacks, even in these trivial scenarios, highlighting serious robustness risks for LLM-as-a-judge applications.

http://arxiv.org/abs/2304.10755
Interpretable and Robust AI in EEG Systems: A Survey. (12%)
Xinliang Zhou; Chenyu Liu; Jinan Zhou; Zhongruo Wang; Liming Zhai; Ziyu Jia; Cuntai Guan; Yang Liu
The close coupling of artificial intelligence (AI) and electroencephalography (EEG) has substantially advanced human-computer interaction (HCI) technologies in the AI era. Different from traditional EEG systems, the interpretability and robustness of AI-based EEG systems are becoming particularly crucial. The interpretability clarifies the inner working mechanisms of AI models and thus can gain the trust of users. The robustness reflects the AI's reliability against attacks and perturbations, which is essential for sensitive and fragile EEG signals. Thus the interpretability and robustness of AI in EEG systems have attracted increasing attention, and their research has achieved great progress recently. However, there is still no survey covering recent advances in this field. In this paper, we present the first comprehensive survey and summarize the interpretable and robust AI techniques for EEG systems. Specifically, we first propose a taxonomy of interpretability by characterizing it into three types: backpropagation, perturbation, and inherently interpretable methods. Then we classify the robustness mechanisms into four classes: noise and artifacts, human variability, data acquisition instability, and adversarial attacks. Finally, we identify several critical and unresolved challenges for interpretable and robust AI in EEG systems and further discuss their future directions.

http://arxiv.org/abs/2410.04823
CAT: Concept-level backdoor ATtacks for Concept Bottleneck Models. (11%)
Songning Lai; Jiayu Yang; Yu Huang; Lijie Hu; Tianlang Xue; Zhangyi Hu; Jiaxu Li; Haicheng Liao; Yutao Yue
Despite the transformative impact of deep learning across multiple domains, the inherent opacity of these models has driven the development of Explainable Artificial Intelligence (XAI). Among these efforts, Concept Bottleneck Models (CBMs) have emerged as a key approach to improve interpretability by leveraging high-level semantic information. However, CBMs, like other machine learning models, are susceptible to security threats, particularly backdoor attacks, which can covertly manipulate model behaviors. Understanding that the community has not yet studied the concept level backdoor attack of CBM, because of "Better the devil you know than the devil you don't know.", we introduce CAT (Concept-level Backdoor ATtacks), a methodology that leverages the conceptual representations within CBMs to embed triggers during training, enabling controlled manipulation of model predictions at inference time. An enhanced attack pattern, CAT+, incorporates a correlation function to systematically select the most effective and stealthy concept triggers, thereby optimizing the attack's impact. Our comprehensive evaluation framework assesses both the attack success rate and stealthiness, demonstrating that CAT and CAT+ maintain high performance on clean data while achieving significant targeted effects on backdoored datasets. This work underscores the potential security risks associated with CBMs and provides a robust testing methodology for future security assessments.

http://arxiv.org/abs/2305.14561
NeFT: Negative Feedback Training to Improve Robustness of Compute-In-Memory DNN Accelerators. (1%)
Yifan Qin; Zheyu Yan; Dailin Gan; Jun Xia; Zixuan Pan; Wujie Wen; Xiaobo Sharon Hu; Yiyu Shi
Compute-in-memory accelerators built upon non-volatile memory devices excel in energy efficiency and latency when performing deep neural network (DNN) inference, thanks to their in-situ data processing capability. However, the stochastic nature and intrinsic variations of non-volatile memory devices often result in performance degradation during DNN inference. Introducing these non-ideal device behaviors in DNN training enhances robustness, but drawbacks include limited accuracy improvement, reduced prediction confidence, and convergence issues. This arises from a mismatch between the deterministic training and non-deterministic device variations, as such training, though considering variations, relies solely on the model's final output. In this work, inspired by control theory, we propose Negative Feedback Training (NeFT), a novel concept supported by theoretical analysis, to more effectively capture the multi-scale noisy information throughout the network. We instantiate this concept with two specific instances, oriented variational forward (OVF) and intermediate representation snapshot (IRS). Based on device variation models extracted from measured data, extensive experiments show that our NeFT outperforms existing state-of-the-art methods with up to a 45.08% improvement in inference accuracy while reducing epistemic uncertainty, boosting output confidence, and improving convergence probability. These results underline the generality and practicality of our NeFT framework for increasing the robustness of DNNs against device variations. The source code for these two instances is available at https://github.com/YifanQin-ND/NeFT_CIM

http://arxiv.org/abs/2508.11279
Boosting the Robustness-Accuracy Trade-off of SNNs by Robust Temporal Self-Ensemble. (92%)
Jihang Wang; Dongcheng Zhao; Ruolin Chen; Qian Zhang; Yi Zeng
Spiking Neural Networks (SNNs) offer a promising direction for energy-efficient and brain-inspired computing, yet their vulnerability to adversarial perturbations remains poorly understood. In this work, we revisit the adversarial robustness of SNNs through the lens of temporal ensembling, treating the network as a collection of evolving sub-networks across discrete timesteps. This formulation uncovers two critical but underexplored challenges-the fragility of individual temporal sub-networks and the tendency for adversarial vulnerabilities to transfer across time. To overcome these limitations, we propose Robust Temporal self-Ensemble (RTE), a training framework that improves the robustness of each sub-network while reducing the temporal transferability of adversarial perturbations. RTE integrates both objectives into a unified loss and employs a stochastic sampling strategy for efficient optimization. Extensive experiments across multiple benchmarks demonstrate that RTE consistently outperforms existing training methods in robust-accuracy trade-off. Additional analyses reveal that RTE reshapes the internal robustness landscape of SNNs, leading to more resilient and temporally diversified decision boundaries. Our study highlights the importance of temporal structure in adversarial learning and offers a principled foundation for building robust spiking models.

http://arxiv.org/abs/2508.11341
Semantically Guided Adversarial Testing of Vision Models Using Language Models. (75%)
Katarzyna Filus; Jorge M. Cruz-Duarte
In targeted adversarial attacks on vision models, the selection of the target label is a critical yet often overlooked determinant of attack success. This target label corresponds to the class that the attacker aims to force the model to predict. Now, existing strategies typically rely on randomness, model predictions, or static semantic resources, limiting interpretability, reproducibility, or flexibility. This paper then proposes a semantics-guided framework for adversarial target selection using the cross-modal knowledge transfer from pretrained language and vision-language models. We evaluate several state-of-the-art models (BERT, TinyLLAMA, and CLIP) as similarity sources to select the most and least semantically related labels with respect to the ground truth, forming best- and worst-case adversarial scenarios. Our experiments on three vision models and five attack methods reveal that these models consistently render practical adversarial targets and surpass static lexical databases, such as WordNet, particularly for distant class relationships. We also observe that static testing of target labels offers a preliminary assessment of the effectiveness of similarity sources, \textit{a priori} testing. Our results corroborate the suitability of pretrained models for constructing interpretable, standardized, and scalable adversarial benchmarks across architectures and datasets.

http://arxiv.org/abs/2505.01811
BadPatches: Backdoor Attacks Against Patch-based Mixture of Experts Architectures. (67%)
Cedric Chan; Jona te Lintelo; Stjepan Picek
As Deep Neural Networks (DNNs) continue to require larger amounts of data and computational power, Mixture of Experts (MoE) models have become a popular choice to reduce computational complexity. This popularity increases the importance of considering the security of MoE architectures. Unfortunately, the security of models using a MoE architecture has not yet gained much attention compared to other DNN models. In this work, we investigate the vulnerability of patch-based MoE (pMoE) models for image classification against backdoor attacks. We examine multiple trigger generation methods and Fine-Pruning as a defense. To better understand a pMoE model's vulnerability to backdoor attacks, we investigate which factors affect the model's patch selection. Our work shows that pMoE models are highly susceptible to backdoor attacks. More precisely, we achieve high attack success rates of up to 100% with visible triggers and a 2% poisoning rate, whilst only having a clean accuracy drop of 1.0%. Additionally, we show that pruning itself is ineffective as a defense but that fine-tuning can remove the backdoor almost completely. Our results show that fine-tuning the model for five epochs reduces the attack success rate to 2.1% whilst sacrificing 1.4% accuracy.

http://arxiv.org/abs/2506.05982
MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks. (10%)
Zonglin Wu; Yule Xue; Yaoyao Feng; Xiaolong Wang; Yiren Song
As automated attack techniques rapidly advance, CAPTCHAs remain a critical defense mechanism against malicious bots. However, existing CAPTCHA schemes encompass a diverse range of modalities -- from static distorted text and obfuscated images to interactive clicks, sliding puzzles, and logic-based questions -- yet the community still lacks a unified, large-scale, multimodal benchmark to rigorously evaluate their security robustness. To address this gap, we introduce MCA-Bench, a comprehensive and reproducible benchmarking suite that integrates heterogeneous CAPTCHA types into a single evaluation protocol. Leveraging a shared vision-language model backbone, we fine-tune specialized cracking agents for each CAPTCHA category, enabling consistent, cross-modal assessments. Extensive experiments reveal that MCA-Bench effectively maps the vulnerability spectrum of modern CAPTCHA designs under varied attack settings, and crucially offers the first quantitative analysis of how challenge complexity, interaction depth, and model solvability interrelate. Based on these findings, we propose three actionable design principles and identify key open challenges, laying the groundwork for systematic CAPTCHA hardening, fair benchmarking, and broader community collaboration. Datasets and code are available online.

http://arxiv.org/abs/2508.11432
Robust Convolution Neural ODEs via Contractivity-promoting regularization. (5%)
Muhammad Zakwan; Liang Xu; Giancarlo Ferrari-Trecate
Neural networks can be fragile to input noise and adversarial attacks.
  In this work, we consider Convolutional Neural Ordinary Differential Equations (NODEs), a family of continuous-depth neural networks represented by dynamical systems, and propose to use contraction theory to improve their robustness.
  For a contractive dynamical system two trajectories starting from different initial conditions converge to each other exponentially fast.
  Contractive Convolutional NODEs can enjoy increased robustness as slight perturbations of the features do not cause a significant change in the output.
  Contractivity can be induced during training by using a regularization term involving the Jacobian of the system dynamics.
  To reduce the computational burden, we show that it can also be promoted using carefully selected weight regularization terms for a class of NODEs with slope-restricted activation functions.
  The performance of the proposed regularizers is illustrated through benchmark image classification tasks on MNIST and FashionMNIST datasets, where images are corrupted by different kinds of noise and attacks.

http://arxiv.org/abs/2508.11453
EvoPSF: Online Evolution of Autonomous Driving Models via Planning-State Feedback. (1%)
Jiayue Jin; Lang Qian; Jingyu Zhang; Chuanyu Ju; Liang Song
Recent years have witnessed remarkable progress in autonomous driving, with systems evolving from modular pipelines to end-to-end architectures. However, most existing methods are trained offline and lack mechanisms to adapt to new environments during deployment. As a result, their generalization ability diminishes when faced with unseen variations in real-world driving scenarios. In this paper, we break away from the conventional "train once, deploy forever" paradigm and propose EvoPSF, a novel online Evolution framework for autonomous driving based on Planning-State Feedback. We argue that planning failures are primarily caused by inaccurate object-level motion predictions, and such failures are often reflected in the form of increased planner uncertainty. To address this, we treat planner uncertainty as a trigger for online evolution, using it as a diagnostic signal to initiate targeted model updates. Rather than performing blind updates, we leverage the planner's agent-agent attention to identify the specific objects that the ego vehicle attends to most, which are primarily responsible for the planning failures. For these critical objects, we compute a targeted self-supervised loss by comparing their predicted waypoints from the prediction module with their actual future positions, selected from the perception module's outputs with high confidence scores. This loss is then backpropagated to adapt the model online. As a result, our method improves the model's robustness to environmental changes, leads to more precise motion predictions, and therefore enables more accurate and stable planning behaviors. Experiments on both cross-region and corrupted variants of the nuScenes dataset demonstrate that EvoPSF consistently improves planning performance under challenging conditions.

http://arxiv.org/abs/2508.11742
Assessing User Privacy Leakage in Synthetic Packet Traces: An Attack-Grounded Approach. (1%)
Minhao Jin; Hongyu He; Maria Apostolaki
Current synthetic traffic generators (SynNetGens) promise privacy but lack comprehensive guarantees or empirical validation, even as their fidelity steadily improves. We introduce the first attack-grounded benchmark for assessing the privacy of SynNetGens directly from the traffic they produce. We frame privacy as membership inference at the traffic-source level--a realistic and actionable threat for data holders. To this end, we present TraceBleed, the first attack that exploits behavioral fingerprints across flows using contrastive learning and temporal chunking, outperforming prior membership inference baselines by 172%. Our large-scale study across GAN-, diffusion-, and GPT-based SynNetGens uncovers critical insights: (i) SynNetGens leak user-level information; (ii) differential privacy either fails to stop these attacks or severely degrades fidelity; and (iii) sharing more synthetic data amplifies leakage by 59% on average. Finally, we introduce TracePatch, the first SynNetGen-agnostic defense that combines adversarial ML with SMT constraints to mitigate leakage while preserving fidelity.

http://arxiv.org/abs/2508.10404
Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation. (99%)
Huizhen Shu; Xuying Li; Qirui Wang; Yuji Kosuga; Mengqiu Tian; Zhuo Li
With the rapid proliferation of Natural Language Processing (NLP), especially Large Language Models (LLMs), generating adversarial examples to jailbreak LLMs remains a key challenge for understanding model vulnerabilities and improving robustness. In this context, we propose a new black-box attack method that leverages the interpretability of large models. We introduce the Sparse Feature Perturbation Framework (SFPF), a novel approach for adversarial text generation that utilizes sparse autoencoders to identify and manipulate critical features in text. After using the SAE model to reconstruct hidden layer representations, we perform feature clustering on the successfully attacked texts to identify features with higher activations. These highly activated features are then perturbed to generate new adversarial texts. This selective perturbation preserves the malicious intent while amplifying safety signals, thereby increasing their potential to evade existing defenses. Our method enables a new red-teaming strategy that balances adversarial effectiveness with safety alignment. Experimental results demonstrate that adversarial texts generated by SFPF can bypass state-of-the-art defense mechanisms, revealing persistent vulnerabilities in current NLP systems.However, the method's effectiveness varies across prompts and layers, and its generalizability to other architectures and larger models remains to be validated.

http://arxiv.org/abs/2508.10600
Towards Powerful and Practical Patch Attacks for 2D Object Detection in Autonomous Driving. (98%)
Yuxin Cao; Yedi Zhang; Wentao He; Yifan Liao; Yan Xiao; Chang Li; Zhiyong Huang; Jin Song Dong
Learning-based autonomous driving systems remain critically vulnerable to adversarial patches, posing serious safety and security risks in their real-world deployment. Black-box attacks, notable for their high attack success rate without model knowledge, are especially concerning, with their transferability extensively studied to reduce computational costs compared to query-based attacks. Previous transferability-based black-box attacks typically adopt mean Average Precision (mAP) as the evaluation metric and design training loss accordingly. However, due to the presence of multiple detected bounding boxes and the relatively lenient Intersection over Union (IoU) thresholds, the attack effectiveness of these approaches is often overestimated, resulting in reduced success rates in practical attacking scenarios. Furthermore, patches trained on low-resolution data often fail to maintain effectiveness on high-resolution images, limiting their transferability to autonomous driving datasets. To fill this gap, we propose P$^3$A, a Powerful and Practical Patch Attack framework for 2D object detection in autonomous driving, specifically optimized for high-resolution datasets. First, we introduce a novel metric, Practical Attack Success Rate (PASR), to more accurately quantify attack effectiveness with greater relevance for pedestrian safety. Second, we present a tailored Localization-Confidence Suppression Loss (LCSL) to improve attack transferability under PASR. Finally, to maintain the transferability for high-resolution datasets, we further incorporate the Probabilistic Scale-Preserving Padding (PSPP) into the patch attack pipeline as a data preprocessing step. Extensive experiments show that P$^3$A outperforms state-of-the-art attacks on unseen models and unseen high-resolution datasets, both under the proposed practical IoU-based evaluation metric and the previous mAP-based metrics.

http://arxiv.org/abs/2407.19203
Clean-Label Physical Backdoor Attacks with Data Distillation. (98%)
Thinh Dao; Khoa D Doan; Kok-Seng Wong
Deep Neural Networks (DNNs) are shown to be vulnerable to backdoor poisoning attacks, with most research focusing on digital triggers -- artificial patterns added to test-time inputs to induce targeted misclassification. Physical triggers, which are natural objects embedded in real-world scenes, offer a promising alternative for attackers, as they can activate backdoors in real-time without digital manipulation. However, existing physical backdoor attacks are dirty-label, meaning that attackers must change the labels of poisoned inputs to the target label. The inconsistency between image content and label exposes the attack to human inspection, reducing its stealthiness in real-world settings. To address this limitation, we introduce Clean-Label Physical Backdoor Attack (CLPBA), a new paradigm of physical backdoor attack that does not require label manipulation and trigger injection at the training stage. Instead, the attacker injects imperceptible perturbations into a small number of target class samples to backdoor a model. By framing the attack as a Dataset Distillation problem, we develop three CLPBA variants -- Parameter Matching, Gradient Matching, and Feature Matching -- that craft effective poisons under both linear probing and full-finetuning training settings. In hard scenarios that require backdoor generalizability in the physical world, CLPBA is shown to even surpass Dirty-label attack baselines. We demonstrate the effectiveness of CLPBA via extensive experiments on two collected physical backdoor datasets for facial recognition and animal classification. The code is available in https://github.com/thinh-dao/Clean-Label-Physical-Backdoor-Attacks.

http://arxiv.org/abs/2508.10639
MirGuard: Towards a Robust Provenance-based Intrusion Detection System Against Graph Manipulation Attacks. (83%)
Anyuan Sang; Lu Zhou; Li Yang; Junbo Jia; Huipeng Yang; Pengbin Feng; Jianfeng Ma
Learning-based Provenance-based Intrusion Detection Systems (PIDSes) have become essential tools for anomaly detection in host systems due to their ability to capture rich contextual and structural information, as well as their potential to detect unknown attacks. However, recent studies have shown that these systems are vulnerable to graph manipulation attacks, where attackers manipulate the graph structure to evade detection. While some previous approaches have discussed this type of attack, none have fully addressed it with a robust detection solution, limiting the practical applicability of PIDSes.
  To address this challenge, we propose MirGuard, a robust anomaly detection framework that combines logic-aware multi-view augmentation with contrastive representation learning. Rather than applying arbitrary structural perturbations, MirGuard introduces Logic-Aware Noise Injection (LNI) to generate semantically valid graph views, ensuring that all augmentations preserve the underlying causal semantics of the provenance data. These views are then used in a Logic-Preserving Contrastive Learning framework, which encourages the model to learn representations that are invariant to benign transformations but sensitive to adversarial inconsistencies. Comprehensive evaluations on multiple provenance datasets demonstrate that MirGuard significantly outperforms state-of-the-art detectors in robustness against various graph manipulation attacks without sacrificing detection performance and efficiency. Our work represents the first targeted study to enhance PIDS against such adversarial threats, providing a robust and effective solution to modern cybersecurity challenges.

http://arxiv.org/abs/2508.10243
Pruning and Malicious Injection: A Retraining-Free Backdoor Attack on Transformer Models. (70%)
Taibiao Zhao; Mingxuan Sun; Hao Wang; Xiaobing Chen; Xiangwei Zhou
Transformer models have demonstrated exceptional performance and have become indispensable in computer vision (CV) and natural language processing (NLP) tasks. However, recent studies reveal that transformers are susceptible to backdoor attacks. Prior backdoor attack methods typically rely on retraining with clean data or altering the model architecture, both of which can be resource-intensive and intrusive. In this paper, we propose Head-wise Pruning and Malicious Injection (HPMI), a novel retraining-free backdoor attack on transformers that does not alter the model's architecture. Our approach requires only a small subset of the original data and basic knowledge of the model architecture, eliminating the need for retraining the target transformer. Technically, HPMI works by pruning the least important head and injecting a pre-trained malicious head to establish the backdoor. We provide a rigorous theoretical justification demonstrating that the implanted backdoor resists detection and removal by state-of-the-art defense techniques, under reasonable assumptions. Experimental evaluations across multiple datasets further validate the effectiveness of HPMI, showing that it 1) incurs negligible clean accuracy loss, 2) achieves at least 99.55% attack success rate, and 3) bypasses four advanced defense mechanisms. Additionally, relative to state-of-the-art retraining-dependent attacks, HPMI achieves greater concealment and robustness against diverse defense strategies, while maintaining minimal impact on clean accuracy.

http://arxiv.org/abs/2508.01272
PromptSafe: Gated Prompt Tuning for Safe Text-to-Image Generation. (41%)
Zonglei Jing; Xiao Yang; Xiaoqian Li; Siyuan Liang; Aishan Liu; Mingchuan Zhang; Xianglong Liu
Text-to-image (T2I) models have demonstrated remarkable generative capabilities but remain vulnerable to producing not-safe-for-work (NSFW) content, such as violent or explicit imagery. While recent moderation efforts have introduced soft prompt-guided tuning by appending defensive tokens to the input, these approaches often rely on large-scale curated image-text datasets and apply static, one-size-fits-all defenses at inference time. However, this results not only in high computational cost and degraded benign image quality, but also in limited adaptability to the diverse and nuanced safety requirements of real-world prompts. To address these challenges, we propose PromptSafe, a gated prompt tuning framework that combines a lightweight, text-only supervised soft embedding with an inference-time gated control network. Instead of training on expensive image-text datasets, we first rewrite unsafe prompts into semantically aligned but safe alternatives using an LLM, constructing an efficient text-only training corpus. Based on this, we optimize a universal soft prompt that repels unsafe and attracts safe embeddings during the diffusion denoising process. To avoid over-suppressing benign prompts, we introduce a gated mechanism that adaptively adjusts the defensive strength based on estimated prompt toxicity, thereby aligning defense intensity with prompt risk and ensuring strong protection for harmful inputs while preserving benign generation quality. Extensive experiments across multiple benchmarks and T2I models show that PromptSafe achieves a SOTA unsafe generation rate (2.36%), while preserving high benign fidelity. Furthermore, PromptSafe demonstrates strong generalization to unseen harmful categories, robust transferability across diffusion model architectures, and resilience under adaptive adversarial attacks, highlighting its practical value for safe and scalable deployment.

http://arxiv.org/abs/2508.11053
SHLIME: Foiling adversarial attacks fooling SHAP and LIME. (33%)
Sam Chauhan; Estelle Duguet; Karthik Ramakrishnan; Deventer Hugh Van; Jack Kruger; Ranjan Subbaraman
Post hoc explanation methods, such as LIME and SHAP, provide interpretable insights into black-box classifiers and are increasingly used to assess model biases and generalizability. However, these methods are vulnerable to adversarial manipulation, potentially concealing harmful biases. Building on the work of Slack et al. (2020), we investigate the susceptibility of LIME and SHAP to biased models and evaluate strategies for improving robustness. We first replicate the original COMPAS experiment to validate prior findings and establish a baseline. We then introduce a modular testing framework enabling systematic evaluation of augmented and ensemble explanation approaches across classifiers of varying performance. Using this framework, we assess multiple LIME/SHAP ensemble configurations on out-of-distribution models, comparing their resistance to bias concealment against the original methods. Our results identify configurations that substantially improve bias detection, highlighting their potential for enhancing transparency in the deployment of high-stakes machine learning systems.

http://arxiv.org/abs/2508.10315
A Vision-Language Pre-training Model-Guided Approach for Mitigating Backdoor Attacks in Federated Learning. (11%)
Keke Gai; Dongjue Wang; Jing Yu; Liehuang Zhu; Qi Wu
Existing backdoor defense methods in Federated Learning (FL) rely on the assumption of homogeneous client data distributions or the availability of a clean serve dataset, which limits the practicality and effectiveness. Defending against backdoor attacks under heterogeneous client data distributions while preserving model performance remains a significant challenge. In this paper, we propose a FL backdoor defense framework named CLIP-Fed, which leverages the zero-shot learning capabilities of vision-language pre-training models. By integrating both pre-aggregation and post-aggregation defense strategies, CLIP-Fed overcomes the limitations of Non-IID imposed on defense effectiveness. To address privacy concerns and enhance the coverage of the dataset against diverse triggers, we construct and augment the server dataset using the multimodal large language model and frequency analysis without any client samples. To address class prototype deviations caused by backdoor samples and eliminate the correlation between trigger patterns and target labels, CLIP-Fed aligns the knowledge of the global model and CLIP on the augmented dataset using prototype contrastive loss and Kullback-Leibler divergence. Extensive experiments on representative datasets validate the effectiveness of CLIP-Fed. Compared to state-of-the-art methods, CLIP-Fed achieves an average reduction in ASR, i.e., 2.03\% on CIFAR-10 and 1.35\% on CIFAR-10-LT, while improving average MA by 7.92\% and 0.48\%, respectively.

http://arxiv.org/abs/2508.10491
Contrastive ECOC: Learning Output Codes for Adversarial Defense. (2%)
Che-Yu Chou; Hung-Hsuan Chen
Although one-hot encoding is commonly used for multiclass classification, it is not always the most effective encoding mechanism. Error Correcting Output Codes (ECOC) address multiclass classification by mapping each class to a unique codeword used as a label. Traditional ECOC methods rely on manually designed or randomly generated codebooks, which are labor-intensive and may yield suboptimal, dataset-agnostic results. This paper introduces three models for automated codebook learning based on contrastive learning, allowing codebooks to be learned directly and adaptively from data. Across four datasets, our proposed models demonstrate superior robustness to adversarial attacks compared to two baselines. The source is available at https://github.com/YuChou20/Automated-Codebook-Learning-with-Error-Correcting-Output-Code-Technique.

http://arxiv.org/abs/2508.10598
Oops!... They Stole it Again: Attacks on Split Learning. (1%)
Tanveer Khan; Antonis Michalas
Split Learning (SL) is a collaborative learning approach that improves privacy by keeping data on the client-side while sharing only the intermediate output with a server. However, the distributed nature of SL introduces new security challenges, necessitating a comprehensive exploration of potential attacks. This paper systematically reviews various attacks on SL, classifying them based on factors such as the attacker's role, the type of privacy risks, when data leaks occur, and where vulnerabilities exist. We also analyze existing defense methods, including cryptographic methods, data modification approaches, distributed techniques, and hybrid solutions. Our findings reveal security gaps, highlighting the effectiveness and limitations of existing defenses. By identifying open challenges and future directions, this work provides valuable information to improve SL privacy issues and guide further research.

http://arxiv.org/abs/2508.10637
Processing and acquisition traces in visual encoders: What does CLIP know about your camera? (1%)
Ryan Ramos; Vladan Stojnić; Giorgos Kordopatis-Zilos; Yuta Nakashima; Giorgos Tolias; Noa Garcia
Prior work has analyzed the robustness of visual encoders to image transformations and corruptions, particularly in cases where such alterations are not seen during training. When this occurs, they introduce a form of distribution shift at test time, often leading to performance degradation. The primary focus has been on severe corruptions that, when applied aggressively, distort useful signals necessary for accurate semantic predictions.
  We take a different perspective by analyzing parameters of the image acquisition process and transformations that may be subtle or even imperceptible to the human eye. We find that such parameters are systematically encoded in the learned visual representations and can be easily recovered. More strikingly, their presence can have a profound impact, either positively or negatively, on semantic predictions. This effect depends on whether there is a strong correlation or anti-correlation between semantic labels and these acquisition-based or processing-based labels. Our code and data are available at: https://github.com/ryan-caesar-ramos/visual-encoder-traces

http://arxiv.org/abs/2508.10974
Failures to Surface Harmful Contents in Video Large Language Models. (1%)
Yuxin Cao; Wei Song; Derui Wang; Jingling Xue; Jin Song Dong
Video Large Language Models (VideoLLMs) are increasingly deployed on numerous critical applications, where users rely on auto-generated summaries while casually skimming the video stream. We show that this interaction hides a critical safety gap: if harmful content is embedded in a video, either as full-frame inserts or as small corner patches, state-of-the-art VideoLLMs rarely mention the harmful content in the output, despite its clear visibility to human viewers. A root-cause analysis reveals three compounding design flaws: (1) insufficient temporal coverage resulting from the sparse, uniformly spaced frame sampling used by most leading VideoLLMs, (2) spatial information loss introduced by aggressive token downsampling within sampled frames, and (3) encoder-decoder disconnection, whereby visual cues are only weakly utilized during text generation. Leveraging these insights, we craft three zero-query black-box attacks, aligning with these flaws in the processing pipeline. Our large-scale evaluation across five leading VideoLLMs shows that the harmfulness omission rate exceeds 90% in most cases. Even when harmful content is clearly present in all frames, these models consistently fail to identify it. These results underscore a fundamental vulnerability in current VideoLLMs' designs and highlight the urgent need for sampling strategies, token compression, and decoding mechanisms that guarantee semantic coverage rather than speed alone.

http://arxiv.org/abs/2508.10991
MCP-Guard: A Defense Framework for Model Context Protocol Integrity in Large Language Model Applications. (1%)
Wenpeng Xing; Zhonghao Qi; Yupeng Qin; Yilin Li; Caini Chang; Jiahui Yu; Changting Lin; Zhenzhen Xie; Meng Han
The integration of Large Language Models (LLMs) with external tools via protocols such as the Model Context Protocol (MCP) introduces critical security vulnerabilities, including prompt injection, data exfiltration, and other threats. To counter these challenges, we propose MCP-Guard, a robust, layered defense architecture designed for LLM--tool interactions. MCP-Guard employs a three-stage detection pipeline that balances efficiency with accuracy: it progresses from lightweight static scanning for overt threats and a deep neural detector for semantic attacks, to our fine-tuned E5-based model achieves (96.01) accuracy in identifying adversarial prompts. Finally, a lightweight LLM arbitrator synthesizes these signals to deliver the final decision while minimizing false positives. To facilitate rigorous training and evaluation, we also introduce MCP-AttackBench, a comprehensive benchmark of over 70,000 samples. Sourced from public datasets and augmented by GPT-4, MCP-AttackBench simulates diverse, real-world attack vectors in the MCP format, providing a foundation for future research into securing LLM-tool ecosystems.

http://arxiv.org/abs/2508.10946
IPG: Incremental Patch Generation for Generalized Adversarial Patch Training. (99%)
Wonho Lee; Hyunsik Na; Jisu Lee; Daeseon Choi
The advent of adversarial patches poses a significant challenge to the robustness of AI models, particularly in the domain of computer vision tasks such as object detection. In contradistinction to traditional adversarial examples, these patches target specific regions of an image, resulting in the malfunction of AI models. This paper proposes Incremental Patch Generation (IPG), a method that generates adversarial patches up to 11.1 times more efficiently than existing approaches while maintaining comparable attack performance. The efficacy of IPG is demonstrated by experiments and ablation studies including YOLO's feature distribution visualization and adversarial training results, which show that it produces well-generalized patches that effectively cover a broader range of model vulnerabilities. Furthermore, IPG-generated datasets can serve as a robust knowledge foundation for constructing a robust model, enabling structured representation, advanced reasoning, and proactive defenses in AI security ecosystems. The findings of this study suggest that IPG has considerable potential for future utilization not only in adversarial patch defense but also in real-world applications such as autonomous vehicles, security systems, and medical imaging, where AI models must remain resilient to adversarial attacks in dynamic and high-stakes environments.

http://arxiv.org/abs/2507.01367
3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation. (98%)
Tianrui Lou; Xiaojun Jia; Siyuan Liang; Jiawei Liang; Ming Zhang; Yanjun Xiao; Xiaochun Cao
Physical adversarial attack methods expose the vulnerabilities of deep neural networks and pose a significant threat to safety-critical scenarios such as autonomous driving. Camouflage-based physical attack is a more promising approach compared to the patch-based attack, offering stronger adversarial effectiveness in complex physical environments. However, most prior work relies on mesh priors of the target object and virtual environments constructed by simulators, which are time-consuming to obtain and inevitably differ from the real world. Moreover, due to the limitations of the backgrounds in training images, previous methods often fail to produce multi-view robust adversarial camouflage and tend to fall into sub-optimal solutions. Due to these reasons, prior work lacks adversarial effectiveness and robustness across diverse viewpoints and physical environments. We propose a physical attack framework based on 3D Gaussian Splatting (3DGS), named PGA, which provides rapid and precise reconstruction with few images, along with photo-realistic rendering capabilities. Our framework further enhances cross-view robustness and adversarial effectiveness by preventing mutual and self-occlusion among Gaussians and employing a min-max optimization approach that adjusts the imaging background of each viewpoint, helping the algorithm filter out non-robust adversarial features. Extensive experiments validate the effectiveness and superiority of PGA. Our code is available at:https://github.com/TRLou/PGA.

http://arxiv.org/abs/2508.09603
The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage. (81%)
Skyler Hallinan; Jaehun Jung; Melanie Sclar; Ximing Lu; Abhilasha Ravichander; Sahana Ramnath; Yejin Choi; Sai Praneeth Karimireddy; Niloofar Mireshghallah; Xiang Ren
Membership inference attacks serves as useful tool for fair use of language models, such as detecting potential copyright infringement and auditing data leakage. However, many current state-of-the-art attacks require access to models' hidden states or probability distribution, which prevents investigation into more widely-used, API-access only models like GPT-4. In this work, we introduce N-Gram Coverage Attack, a membership inference attack that relies solely on text outputs from the target model, enabling attacks on completely black-box models. We leverage the observation that models are more likely to memorize and subsequently generate text patterns that were commonly observed in their training data. Specifically, to make a prediction on a candidate member, N-Gram Coverage Attack first obtains multiple model generations conditioned on a prefix of the candidate. It then uses n-gram overlap metrics to compute and aggregate the similarities of these outputs with the ground truth suffix; high similarities indicate likely membership. We first demonstrate on a diverse set of existing benchmarks that N-Gram Coverage Attack outperforms other black-box methods while also impressively achieving comparable or even better performance to state-of-the-art white-box attacks - despite having access to only text outputs. Interestingly, we find that the success rate of our method scales with the attack compute budget - as we increase the number of sequences generated from the target model conditioned on the prefix, attack performance tends to improve. Having verified the accuracy of our method, we use it to investigate previously unstudied closed OpenAI models on multiple domains. We find that more recent models, such as GPT-4o, exhibit increased robustness to membership inference, suggesting an evolving trend toward improved privacy protections.

http://arxiv.org/abs/2412.04510
A Taxonomy of System-Level Attacks on Deep Learning Models in Autonomous Vehicles. (76%)
Masoud Jamshidiyan Tehrani; Jinhan Kim; Rosmael Zidane Lekeufack Foulefack; Alessandro Marchetto; Paolo Tonella
The advent of deep learning and its astonishing performance has enabled its usage in complex systems, including autonomous vehicles. On the other hand, deep learning models are susceptible to mispredictions when small, adversarial changes are introduced into their input. Such mis-predictions can be triggered in the real world and can result in a failure of the entire system. In recent years, a growing number of research works have investigated ways to mount attacks against autonomous vehicles that exploit deep learning components. Such attacks are directed toward elements of the environment where these systems operate and their effectiveness is assessed in terms of system-level failures triggered by them. There has been however no systematic attempt to analyze and categorize such attacks. In this paper, we present the first taxonomy of system-level attacks against autonomous vehicles. We constructed our taxonomy by selecting 21 highly relevant papers, then we tagged them with 12 top-level taxonomy categories and several sub-categories. The taxonomy allowed us to investigate the attack features, the most attacked components and systems, the underlying threat models, and the failure chains from input perturbation to system-level failure. We distilled several lessons for practitioners and identified possible directions for future work for researchers.

http://arxiv.org/abs/2508.09652
Demystifying the Role of Rule-based Detection in AI Systems for Windows Malware Detection. (76%)
Andrea Ponte; Luca Demetrio; Luca Oneto; Ivan Tesfai Ogbu; Battista Biggio; Fabio Roli
Malware detection increasingly relies on AI systems that integrate signature-based detection with machine learning. However, these components are typically developed and combined in isolation, missing opportunities to reduce data complexity and strengthen defenses against adversarial EXEmples, carefully crafted programs designed to evade detection. Hence, in this work we investigate the influence that signature-based detection exerts on model training, when they are included inside the training pipeline. Specifically, we compare models trained on a comprehensive dataset with an AI system whose machine learning component is trained solely on samples not already flagged by signatures. Our results demonstrate improved robustness to both adversarial EXEmples and temporal data drift, although this comes at the cost of a fixed lower bound on false positives, driven by suboptimal rule selection. We conclude by discussing these limitations and outlining how future research could extend AI-based malware detection to include dynamic analysis, thereby further enhancing system resilience.

http://arxiv.org/abs/2506.22557
MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMs. (75%)
Boyuan Chen; Minghao Shao; Abdul Basit; Siddharth Garg; Muhammad Shafique
As large language models (LLMs) grow more capable, they face growing vulnerability to sophisticated jailbreak attacks. While developers invest heavily in alignment finetuning and safety guardrails, researchers continue publishing novel attacks, driving progress through adversarial iteration. This dynamic mirrors a strategic game of continual evolution. However, two major challenges hinder jailbreak development: the high cost of querying top-tier LLMs and the short lifespan of effective attacks due to frequent safety updates. These factors limit cost-efficiency and practical impact of research in jailbreak attacks. To address this, we propose MetaCipher, a low-cost, multi-agent jailbreak framework that generalizes across LLMs with varying safety measures. Using reinforcement learning, MetaCipher is modular and adaptive, supporting extensibility to future strategies. Within as few as 10 queries, MetaCipher achieves state-of-the-art attack success rates on recent malicious prompt benchmarks, outperforming prior jailbreak methods. We conduct a large-scale empirical evaluation across diverse victim models and benchmarks, demonstrating its robustness and adaptability. Warning: This paper contains model outputs that may be offensive or harmful, shown solely to demonstrate jailbreak efficacy.

http://arxiv.org/abs/2404.15611
Model Poisoning Attacks to Federated Learning via Multi-Round Consistency. (54%)
Yueqi Xie; Minghong Fang; Neil Zhenqiang Gong
Model poisoning attacks are critical security threats to Federated Learning (FL). Existing model poisoning attacks suffer from two key limitations: 1) they achieve suboptimal effectiveness when defenses are deployed, and/or 2) they require knowledge of the model updates or local training data on genuine clients. In this work, we make a key observation that their suboptimal effectiveness arises from only leveraging model-update consistency among malicious clients within individual training rounds, making the attack effect self-cancel across training rounds. In light of this observation, we propose PoisonedFL, which enforces multi-round consistency among the malicious clients' model updates while not requiring any knowledge about the genuine clients. Our empirical evaluation on five benchmark datasets shows that PoisonedFL breaks eight state-of-the-art defenses and outperforms seven existing model poisoning attacks. Moreover, we also explore new defenses that are tailored to PoisonedFL, but our results show that we can still adapt PoisonedFL to break them. Our study shows that FL systems are considerably less robust than previously thought, underlining the urgency for the development of new defense mechanisms.

http://arxiv.org/abs/2507.22398
On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations. (33%)
Jordan Vice; Naveed Akhtar; Yansong Gao; Richard Hartley; Ajmal Mian
Vision-Language Models (VLMs) are increasingly used as perceptual modules for visual content reasoning, including through captioning and DeepFake detection. In this work, we expose a critical vulnerability of VLMs when exposed to subtle, structured perturbations in the frequency domain. Specifically, we highlight how these feature transformations undermine authenticity/DeepFake detection and automated image captioning tasks. We design targeted image transformations, operating in the frequency domain to systematically adjust VLM outputs when exposed to frequency-perturbed real and synthetic images. We demonstrate that the perturbation injection method generalizes across five state-of-the-art VLMs which includes different-parameter Qwen2/2.5 and BLIP models. Experimenting across ten real and generated image datasets reveals that VLM judgments are sensitive to frequency-based cues and may not wholly align with semantic content. Crucially, we show that visually-imperceptible spatial frequency transformations expose the fragility of VLMs deployed for automated image captioning and authenticity detection tasks. Our findings under realistic, black-box constraints challenge the reliability of VLMs, underscoring the need for robust multimodal perception systems.

http://arxiv.org/abs/2508.10212
Detecting Untargeted Attacks and Mitigating Unreliable Updates in Federated Learning for Underground Mining Operations. (13%)
Md Sazedur Rahman; Mohamed Elmahallawy; Sanjay Madria; Samuel Frimpong
Underground mining operations rely on distributed sensor networks to collect critical data daily, including mine temperature, toxic gas concentrations, and miner movements for hazard detection and operational decision-making. However, transmitting raw sensor data to a central server for training deep learning models introduces significant privacy risks, potentially exposing sensitive mine-specific information. Federated Learning (FL) offers a transformative solution by enabling collaborative model training while ensuring that raw data remains localized at each mine. Despite its advantages, FL in underground mining faces key challenges: (i) An attacker may compromise a mine's local model by employing techniques such as sign-flipping attacks or additive noise, leading to erroneous predictions; (ii) Low-quality (yet potentially valuable) data, caused by poor lighting conditions or sensor inaccuracies in mines may degrade the FL training process. In response, this paper proposes MineDetect, a defense FL framework that detects and isolates the attacked models while mitigating the impact of mines with low-quality data. MineDetect introduces two key innovations: (i) Detecting attacked models (maliciously manipulated) by developing a history-aware mechanism that leverages local and global averages of gradient updates; (ii) Identifying and eliminating adversarial influences from unreliable models (generated by clients with poor data quality) on the FL training process. Comprehensive simulations across diverse datasets demonstrate that MineDetect outperforms existing methods in both robustness and accuracy, even in challenging non-IID data scenarios. Its ability to counter adversarial influences while maintaining lower computational efficiency makes it a vital advancement for improving safety and operational effectiveness in underground mining.

http://arxiv.org/abs/2508.10949
Perturbed Public Voices (P$^{2}$V): A Dataset for Robust Audio Deepfake Detection. (2%)
Chongyang Gao; Marco Postiglione; Isabel Gortner; Sarit Kraus; V. S. Subrahmanian
Current audio deepfake detectors cannot be trusted. While they excel on controlled benchmarks, they fail when tested in the real world. We introduce Perturbed Public Voices (P$^{2}$V), an IRB-approved dataset capturing three critical aspects of malicious deepfakes: (1) identity-consistent transcripts via LLMs, (2) environmental and adversarial noise, and (3) state-of-the-art voice cloning (2020-2025). Experiments reveal alarming vulnerabilities of 22 recent audio deepfake detectors: models trained on current datasets lose 43% performance when tested on P$^{2}$V, with performance measured as the mean of F1 score on deepfake audio, AUC, and 1-EER. Simple adversarial perturbations induce up to 16% performance degradation, while advanced cloning techniques reduce detectability by 20-30%. In contrast, P$^{2}$V-trained models maintain robustness against these attacks while generalizing to existing datasets, establishing a new benchmark for robust audio deepfake detection. P$^{2}$V will be publicly released upon acceptance by a conference/journal.

http://arxiv.org/abs/2502.12207
PAR-AdvGAN: Improving Adversarial Attack Capability with Progressive Auto-Regression AdvGAN. (99%)
Jiayu Zhang; Zhiyu Zhu; Xinyi Wang; Silin Liao; Zhibo Jin; Flora D. Salim; Huaming Chen
Deep neural networks have demonstrated remarkable performance across various domains. However, they are vulnerable to adversarial examples, which can lead to erroneous predictions. Generative Adversarial Networks (GANs) can leverage the generators and discriminators model to quickly produce high-quality adversarial examples. Since both modules train in a competitive and simultaneous manner, GAN-based algorithms like AdvGAN can generate adversarial examples with better transferability compared to traditional methods. However, the generation of perturbations is usually limited to a single iteration, preventing these examples from fully exploiting the potential of the methods. To tackle this issue, we introduce a novel approach named Progressive Auto-Regression AdvGAN (PAR-AdvGAN). It incorporates an auto-regressive iteration mechanism within a progressive generation network to craft adversarial examples with enhanced attack capability. We thoroughly evaluate our PAR-AdvGAN method with a large-scale experiment, demonstrating its superior performance over various state-of-the-art black-box adversarial attacks, as well as the original AdvGAN.Moreover, PAR-AdvGAN significantly accelerates the adversarial example generation, i.e., achieving the speeds of up to 335.5 frames per second on Inception-v3 model, outperforming the gradient-based transferable attack algorithms. Our code is available at: https://github.com/LMBTough/PAR

http://arxiv.org/abs/2508.08656
Evasive Ransomware Attacks Using Low-level Behavioral Adversarial Examples. (98%)
Manabu Hirano; Ryotaro Kobayashi
Protecting state-of-the-art AI-based cybersecurity defense systems from cyber attacks is crucial. Attackers create adversarial examples by adding small changes (i.e., perturbations) to the attack features to evade or fool the deep learning model. This paper introduces the concept of low-level behavioral adversarial examples and its threat model of evasive ransomware. We formulate the method and the threat model to generate the optimal source code of evasive malware. We then examine the method using the leaked source code of Conti ransomware with the micro-behavior control function. The micro-behavior control function is our test component to simulate changing source code in ransomware; ransomware's behavior can be changed by specifying the number of threads, file encryption ratio, and delay after file encryption at the boot time. We evaluated how much an attacker can control the behavioral features of ransomware using the micro-behavior control function to decrease the detection rate of a ransomware detector.

http://arxiv.org/abs/2508.08920
Exploring Cross-Stage Adversarial Transferability in Class-Incremental Continual Learning. (98%)
Jungwoo Kim; Jong-Seok Lee
Class-incremental continual learning addresses catastrophic forgetting by enabling classification models to preserve knowledge of previously learned classes while acquiring new ones. However, the vulnerability of the models against adversarial attacks during this process has not been investigated sufficiently. In this paper, we present the first exploration of vulnerability to stage-transferred attacks, i.e., an adversarial example generated using the model in an earlier stage is used to attack the model in a later stage. Our findings reveal that continual learning methods are highly susceptible to these attacks, raising a serious security issue. We explain this phenomenon through model similarity between stages and gradual robustness degradation. Additionally, we find that existing adversarial training-based defense methods are not sufficiently effective to stage-transferred attacks. Codes are available at https://github.com/mcml-official/CSAT.

http://arxiv.org/abs/2508.08955
Fre-CW: Targeted Attack on Time Series Forecasting using Frequency Domain Loss. (92%)
Naifu Feng; Lixing Chen; Junhua Tang; Hua Ding; Jianhua Li; Yang Bai
Transformer-based models have made significant progress in time series forecasting. However, a key limitation of deep learning models is their susceptibility to adversarial attacks, which has not been studied enough in the context of time series prediction. In contrast to areas such as computer vision, where adversarial robustness has been extensively studied, frequency domain features of time series data play an important role in the prediction task but have not been sufficiently explored in terms of adversarial attacks. This paper proposes a time series prediction attack algorithm based on frequency domain loss. Specifically, we adapt an attack method originally designed for classification tasks to the prediction field and optimize the adversarial samples using both time-domain and frequency-domain losses. To the best of our knowledge, there is no relevant research on using frequency information for time-series adversarial attacks. Our experimental results show that these current time series prediction models are vulnerable to adversarial attacks, and our approach achieves excellent performance on major time series forecasting datasets.

http://arxiv.org/abs/2508.09275
Constrained Black-Box Attacks Against Multi-Agent Reinforcement Learning. (81%)
Amine Andam; Jamal Bentahar; Mustapha Hedabou
Collaborative multi-agent reinforcement learning (c-MARL) has rapidly evolved, offering state-of-the-art algorithms for real-world applications, including sensitive domains. However, a key challenge to its widespread adoption is the lack of a thorough investigation into its vulnerabilities to adversarial attacks. Existing work predominantly focuses on training-time attacks or unrealistic scenarios, such as access to policy weights or the ability to train surrogate policies. In this paper, we investigate new vulnerabilities under more realistic and constrained conditions, assuming an adversary can only collect and perturb the observations of deployed agents. We also consider scenarios where the adversary has no access at all. We propose simple yet highly effective algorithms for generating adversarial perturbations designed to misalign how victim agents perceive their environment. Our approach is empirically validated on three benchmarks and 22 environments, demonstrating its effectiveness across diverse algorithms and environments. Furthermore, we show that our algorithm is sample-efficient, requiring only 1,000 samples compared to the millions needed by previous methods.

http://arxiv.org/abs/2501.10740
Improving the robustness of neural ODEs with minimal weight perturbation. (67%)
Marinis Arturo De; Nicola Guglielmi; Stefano Sicilia; Francesco Tudisco
We propose a method to enhance the stability of a neural ordinary differential equation (neural ODE) by reducing the maximum error growth subsequent to a perturbation of the initial value. Since the stability depends on the logarithmic norm of the Jacobian matrix associated with the neural ODE, we control the logarithmic norm by perturbing the weight matrices of the neural ODE by a smallest possible perturbation (in Frobenius norm). We do so by engaging an eigenvalue optimisation problem, for which we propose a nested two-level algorithm. For a given perturbation size of the weight matrix, the inner level computes optimal perturbations of that size, while - at the outer level - we tune the perturbation amplitude until we reach the desired uniform stability bound. We embed the proposed algorithm in the training of the neural ODE to improve its robustness to perturbations of the initial value, as adversarial attacks. Numerical experiments on classical image datasets show that an image classifier including a neural ODE in its architecture trained according to our strategy is more stable than the same classifier trained in the classical way, and therefore, it is more robust and less vulnerable to adversarial attacks.

http://arxiv.org/abs/2508.14079
A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy. (64%)
Maxime Heuillet; Rishika Bhagwatkar; Jonas Ngnawé; Yann Pequignot; Alexandre Larouche; Christian Gagné; Irina Rish; Ola Ahmad; Audrey Durand
Deep learning models operating in the image domain are vulnerable to small input perturbations. For years, robustness to such perturbations was pursued by training models from scratch (i.e., with random initializations) using specialized loss objectives. Recently, robust fine-tuning has emerged as a more efficient alternative: instead of training from scratch, pretrained models are adapted to maximize predictive performance and robustness. To conduct robust fine-tuning, practitioners design an optimization strategy that includes the model update protocol (e.g., full or partial) and the specialized loss objective. Additional design choices include the architecture type and size, and the pretrained representation. These design choices affect robust generalization, which is the model's ability to maintain performance when exposed to new and unseen perturbations at test time. Understanding how these design choices influence generalization remains an open question with significant practical implications. In response, we present an empirical study spanning 6 datasets, 40 pretrained architectures, 2 specialized losses, and 3 adaptation protocols, yielding 1,440 training configurations and 7,200 robustness measurements across five perturbation types. To our knowledge, this is the most diverse and comprehensive benchmark of robust fine-tuning to date. While attention-based architectures and robust pretrained representations are increasingly popular, we find that convolutional neural networks pretrained in a supervised manner on large datasets often perform best. Our analysis both confirms and challenges prior design assumptions, highlighting promising research directions and offering practical guidance.

http://arxiv.org/abs/2508.06964
Adversarial Video Promotion Against Text-to-Video Retrieval. (64%)
Qiwei Tian; Chenhao Lin; Zhengyu Zhao; Qian Li; Shuai Liu; Chao Shen
Thanks to the development of cross-modal models, text-to-video retrieval (T2VR) is advancing rapidly, but its robustness remains largely unexamined. Existing attacks against T2VR are designed to push videos away from queries, i.e., suppressing the ranks of videos, while the attacks that pull videos towards selected queries, i.e., promoting the ranks of videos, remain largely unexplored. These attacks can be more impactful as attackers may gain more views/clicks for financial benefits and widespread (mis)information. To this end, we pioneer the first attack against T2VR to promote videos adversarially, dubbed the Video Promotion attack (ViPro). We further propose Modal Refinement (MoRe) to capture the finer-grained, intricate interaction between visual and textual modalities to enhance black-box transferability. Comprehensive experiments cover 2 existing baselines, 3 leading T2VR models, 3 prevailing datasets with over 10k videos, evaluated under 3 scenarios. All experiments are conducted in a multi-target setting to reflect realistic scenarios where attackers seek to promote the video regarding multiple queries simultaneously. We also evaluated our attacks for defences and imperceptibility. Overall, ViPro surpasses other baselines by over $30/10/4\%$ for white/grey/black-box settings on average. Our work highlights an overlooked vulnerability, provides a qualitative analysis on the upper/lower bound of our attacks, and offers insights into potential counterplays. Code will be publicly available at https://github.com/michaeltian108/ViPro.

http://arxiv.org/abs/2508.09320
Exact Verification of Graph Neural Networks with Incremental Constraint Solving. (54%)
Minghao Liu; Chia-Hsuan Lu; Marta Kwiatkowska
Graph neural networks (GNNs) are increasingly employed in high-stakes applications, such as fraud detection or healthcare, but are susceptible to adversarial attacks. A number of techniques have been proposed to provide adversarial robustness guarantees, but support for commonly used aggregation functions in message-passing GNNs is still lacking. In this paper, we develop an exact (sound and complete) verification method for GNNs to compute guarantees against attribute and structural perturbations that involve edge addition or deletion, subject to budget constraints. Focusing on node classification tasks, our method employs constraint solving with bound tightening, and iteratively solves a sequence of relaxed constraint satisfaction problems while relying on incremental solving capabilities of solvers to improve efficiency. We implement GNNev, a versatile solver for message-passing neural networks, which supports three aggregation functions, sum, max and mean, with the latter two considered here for the first time. Extensive experimental evaluation of GNNev on two standard benchmarks (Cora and CiteSeer) and two real-world fraud datasets (Amazon and Yelp) demonstrates its usability and effectiveness, as well as superior performance compared to existing {exact verification} tools on sum-aggregated node classification tasks.

http://arxiv.org/abs/2508.09230
Cowpox: Towards the Immunity of VLM-based Multi-Agent Systems. (13%)
Yutong Wu; Jie Zhang; Yiming Li; Chao Zhang; Qing Guo; Nils Lukas; Tianwei Zhang
Vision Language Model (VLM)-based agents are stateful, autonomous entities capable of perceiving and interacting with their environments through vision and language. Multi-agent systems comprise specialized agents who collaborate to solve a (complex) task. A core security property is robustness, stating that the system should maintain its integrity under adversarial attacks. However, the design of existing multi-agent systems lacks the robustness consideration, as a successful exploit against one agent can spread and infect other agents to undermine the entire system's assurance. To address this, we propose a new defense approach, Cowpox, to provably enhance the robustness of multi-agent systems. It incorporates a distributed mechanism, which improves the recovery rate of agents by limiting the expected number of infections to other agents. The core idea is to generate and distribute a special cure sample that immunizes an agent against the attack before exposure and helps recover the already infected agents. We demonstrate the effectiveness of Cowpox empirically and provide theoretical robustness guarantees.

http://arxiv.org/abs/2508.09060
Developing a Transferable Federated Network Intrusion Detection System. (1%)
Abu Shafin Mohammad Mahdee Jameel; Shreya Ghosh; Aly El Gamal
Intrusion Detection Systems (IDS) are a vital part of a network-connected device. In this paper, we develop a deep learning based intrusion detection system that is deployed in a distributed setup across devices connected to a network. Our aim is to better equip deep learning models against unknown attacks using knowledge from known attacks. To this end, we develop algorithms to maximize the number of transferability relationships. We propose a Convolutional Neural Network (CNN) model, along with two algorithms that maximize the number of relationships observed. One is a two step data pre-processing stage, and the other is a Block-Based Smart Aggregation (BBSA) algorithm. The proposed system succeeds in achieving superior transferability performance while maintaining impressive local detection rates. We also show that our method is generalizable, exhibiting transferability potential across datasets and even with different backbones. The code for this work can be found at https://github.com/ghosh64/tabfidsv2.

http://arxiv.org/abs/2508.09021
Attacks and Defenses Against LLM Fingerprinting. (1%)
Kevin Kurian; Ethan Holland; Sean Oesch
As large language models are increasingly deployed in sensitive environments, fingerprinting attacks pose significant privacy and security risks. We present a study of LLM fingerprinting from both offensive and defensive perspectives. Our attack methodology uses reinforcement learning to automatically optimize query selection, achieving better fingerprinting accuracy with only 3 queries compared to randomly selecting 3 queries from the same pool. Our defensive approach employs semantic-preserving output filtering through a secondary LLM to obfuscate model identity while maintaining semantic integrity. The defensive method reduces fingerprinting accuracy across tested models while preserving output quality. These contributions show the potential to improve fingerprinting tools capabilities while providing practical mitigation strategies against fingerprinting attacks.

http://arxiv.org/abs/2502.18862
One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMs. (1%)
Jacob Dunefsky; Arman Cohan
Steering vectors (SVs) have emerged as a promising approach for interpreting and controlling LLMs, but current methods typically require large contrastive datasets that are often impractical to construct and may capture spurious correlations. We propose directly optimizing SVs through gradient descent on a single training example, and systematically investigate how these SVs generalize. We consider several SV optimization techniques and find that the resulting SVs effectively mediate safety-relevant behaviors in multiple models. Indeed, in experiments on an alignment-faking model, we are able to optimize one-shot SVs that induce harmful behavior on benign examples and whose negations suppress harmful behavior on malign examples. And in experiments on refusal suppression, we demonstrate that one-shot optimized SVs can transfer across inputs, yielding a Harmbench attack success rate of 96.9%. Furthermore, we extend work on "emergent misalignment" and show that SVs optimized to induce a model to write vulnerable code cause the model to respond harmfully on unrelated open-ended prompts. Finally, we use one-shot SV optimization to investigate how an instruction-tuned LLM recovers from outputting false information, and find that this ability is independent of the model's explicit verbalization that the information was false. Overall, our findings suggest that optimizing SVs on a single example can mediate a wide array of misaligned behaviors in LLMs. Code can be found at https://github.com/jacobdunefsky/one-shot-steering-repro and https://github.com/jacobdunefsky/one-shot-steering-misalignment.

http://arxiv.org/abs/2508.08031
IPBA: Imperceptible Perturbation Backdoor Attack in Federated Self-Supervised Learning. (80%)
Jiayao Wang; Yang Song; Zhendong Zhao; Jiale Zhang; Qilin Wu; Junwu Zhu; Dongfang Zhao
Federated self-supervised learning (FSSL) combines the advantages of decentralized modeling and unlabeled representation learning, serving as a cutting-edge paradigm with strong potential for scalability and privacy preservation. Although FSSL has garnered increasing attention, research indicates that it remains vulnerable to backdoor attacks. Existing methods generally rely on visually obvious triggers, which makes it difficult to meet the requirements for stealth and practicality in real-world deployment. In this paper, we propose an imperceptible and effective backdoor attack method against FSSL, called IPBA. Our empirical study reveals that existing imperceptible triggers face a series of challenges in FSSL, particularly limited transferability, feature entanglement with augmented samples, and out-of-distribution properties. These issues collectively undermine the effectiveness and stealthiness of traditional backdoor attacks in FSSL. To overcome these challenges, IPBA decouples the feature distributions of backdoor and augmented samples, and introduces Sliced-Wasserstein distance to mitigate the out-of-distribution properties of backdoor samples, thereby optimizing the trigger generation process. Our experimental results on several FSSL scenarios and datasets show that IPBA significantly outperforms existing backdoor attack methods in performance and exhibits strong robustness under various defense mechanisms.

http://arxiv.org/abs/2508.09218
Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity. (67%)
Zuoou Li; Weitong Zhang; Jingyuan Wang; Shuyuan Zhang; Wenjia Bai; Bernhard Kainz; Mengyun Qiao
Multimodal large language models (MLLMs) are widely used in vision-language reasoning tasks. However, their vulnerability to adversarial prompts remains a serious concern, as safety mechanisms often fail to prevent the generation of harmful outputs. Although recent jailbreak strategies report high success rates, many responses classified as "successful" are actually benign, vague, or unrelated to the intended malicious goal. This mismatch suggests that current evaluation standards may overestimate the effectiveness of such attacks. To address this issue, we introduce a four-axis evaluation framework that considers input on-topicness, input out-of-distribution (OOD) intensity, output harmfulness, and output refusal rate. This framework identifies truly effective jailbreaks. In a substantial empirical study, we reveal a structural trade-off: highly on-topic prompts are frequently blocked by safety filters, whereas those that are too OOD often evade detection but fail to produce harmful content. However, prompts that balance relevance and novelty are more likely to evade filters and trigger dangerous output. Building on this insight, we develop a recursive rewriting strategy called Balanced Structural Decomposition (BSD). The approach restructures malicious prompts into semantically aligned sub-tasks, while introducing subtle OOD signals and visual cues that make the inputs harder to detect. BSD was tested across 13 commercial and open-source MLLMs, where it consistently led to higher attack success rates, more harmful outputs, and fewer refusals. Compared to previous methods, it improves success rates by $67\%$ and harmfulness by $21\%$, revealing a previously underappreciated weakness in current multimodal safety systems.

http://arxiv.org/abs/2508.07873
EFU: Enforcing Federated Unlearning via Functional Encryption. (50%)
Samaneh Mohammadi; Vasileios Tsouvalas; Iraklis Symeonidis; Ali Balador; Tanir Ozcelebi; Francesco Flammini; Nirvana Meratnia
Federated unlearning (FU) algorithms allow clients in federated settings to exercise their ''right to be forgotten'' by removing the influence of their data from a collaboratively trained model. Existing FU methods maintain data privacy by performing unlearning locally on the client-side and sending targeted updates to the server without exposing forgotten data; yet they often rely on server-side cooperation, revealing the client's intent and identity without enforcement guarantees - compromising autonomy and unlearning privacy. In this work, we propose EFU (Enforced Federated Unlearning), a cryptographically enforced FU framework that enables clients to initiate unlearning while concealing its occurrence from the server. Specifically, EFU leverages functional encryption to bind encrypted updates to specific aggregation functions, ensuring the server can neither perform unauthorized computations nor detect or skip unlearning requests. To further mask behavioral and parameter shifts in the aggregated model, we incorporate auxiliary unlearning losses based on adversarial examples and parameter importance regularization. Extensive experiments show that EFU achieves near-random accuracy on forgotten data while maintaining performance comparable to full retraining across datasets and neural architectures - all while concealing unlearning intent from the server. Furthermore, we demonstrate that EFU is agnostic to the underlying unlearning algorithm, enabling secure, function-hiding, and verifiable unlearning for any client-side FU mechanism that issues targeted updates.

http://arxiv.org/abs/2508.08521
VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models. (16%)
Mansi Phute; Ravikumar Balakrishnan
Vision Language Models (VLMs) are increasingly being used in a broad range of applications, bringing their security and behavioral control to the forefront. While existing approaches for behavioral control or output redirection, like system prompting in VLMs, are easily detectable and often ineffective, activation-based steering vectors require invasive runtime access to model internals--incompatible with API-based services and closed-source deployments. We introduce VISOR (Visual Input-based Steering for Output Redirection), a novel method that achieves sophisticated behavioral control through optimized visual inputs alone. By crafting universal steering images that induce target activation patterns, VISOR enables practical deployment across all VLM serving modalities while remaining imperceptible compared to explicit textual instructions. We validate VISOR on LLaVA-1.5-7B across three critical alignment tasks: refusal, sycophancy and survival instinct. A single 150KB steering image matches steering vector performance within 1-2% for positive behavioral shifts while dramatically exceeding it for negative steering--achieving up to 25% shifts from baseline compared to steering vectors' modest changes. Unlike system prompting (3-4% shifts), VISOR provides robust bidirectional control while maintaining 99.9% performance on 14,000 unrelated MMLU tasks. Beyond eliminating runtime overhead and model access requirements, VISOR exposes a critical security vulnerability: adversaries can achieve sophisticated behavioral manipulation through visual channels alone, bypassing text-based defenses. Our work fundamentally re-imagines multimodal model control and highlights the urgent need for defenses against visual steering attacks.

http://arxiv.org/abs/2508.08127
BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attacks. (11%)
Rui Miao; Yixin Liu; Yili Wang; Xu Shen; Yue Tan; Yiwei Dai; Shirui Pan; Xin Wang
The security of LLM-based multi-agent systems (MAS) is critically threatened by propagation vulnerability, where malicious agents can distort collective decision-making through inter-agent message interactions. While existing supervised defense methods demonstrate promising performance, they may be impractical in real-world scenarios due to their heavy reliance on labeled malicious agents to train a supervised malicious detection model. To enable practical and generalizable MAS defenses, in this paper, we propose BlindGuard, an unsupervised defense method that learns without requiring any attack-specific labels or prior knowledge of malicious behaviors. To this end, we establish a hierarchical agent encoder to capture individual, neighborhood, and global interaction patterns of each agent, providing a comprehensive understanding for malicious agent detection. Meanwhile, we design a corruption-guided detector that consists of directional noise injection and contrastive learning, allowing effective detection model training solely on normal agent behaviors. Extensive experiments show that BlindGuard effectively detects diverse attack types (i.e., prompt injection, memory poisoning, and tool attack) across MAS with various communication patterns while maintaining superior generalizability compared to supervised baselines. The code is available at: https://github.com/MR9812/BlindGuard.

http://arxiv.org/abs/2508.08029
Robust Anomaly Detection in O-RAN: Leveraging LLMs against Data Manipulation Attacks. (4%)
Thusitha Dayaratne; Ngoc Duy Pham; Viet Vo; Shangqi Lai; Sharif Abuadbba; Hajime Suzuki; Xingliang Yuan; Carsten Rudolph
The introduction of 5G and the Open Radio Access Network (O-RAN) architecture has enabled more flexible and intelligent network deployments. However, the increased complexity and openness of these architectures also introduce novel security challenges, such as data manipulation attacks on the semi-standardised Shared Data Layer (SDL) within the O-RAN platform through malicious xApps. In particular, malicious xApps can exploit this vulnerability by introducing subtle Unicode-wise alterations (hypoglyphs) into the data that are being used by traditional machine learning (ML)-based anomaly detection methods. These Unicode-wise manipulations can potentially bypass detection and cause failures in anomaly detection systems based on traditional ML, such as AutoEncoders, which are unable to process hypoglyphed data without crashing. We investigate the use of Large Language Models (LLMs) for anomaly detection within the O-RAN architecture to address this challenge. We demonstrate that LLM-based xApps maintain robust operational performance and are capable of processing manipulated messages without crashing. While initial detection accuracy requires further improvements, our results highlight the robustness of LLMs to adversarial attacks such as hypoglyphs in input data. There is potential to use their adaptability through prompt engineering to further improve the accuracy, although this requires further research. Additionally, we show that LLMs achieve low detection latency (under 0.07 seconds), making them suitable for Near-Real-Time (Near-RT) RIC deployments.

http://arxiv.org/abs/2508.08043
False Reality: Uncovering Sensor-induced Human-VR Interaction Vulnerability. (1%)
Yancheng Jiang; Yan Jiang; Ruochen Zhou; Yi-Chao Chen; Xiaoyu Ji; Wenyuan Xu
Virtual Reality (VR) techniques, serving as the bridge between the real and virtual worlds, have boomed and are widely used in manufacturing, remote healthcare, gaming, etc. Specifically, VR systems offer users immersive experiences that include both perceptions and actions. Various studies have demonstrated that attackers can manipulate VR software to influence users' interactions, including perception and actions. However, such attacks typically require strong access and specialized expertise. In this paper, we are the first to present a systematic analysis of physical attacks against VR systems and introduce False Reality, a new attack threat to VR devices without requiring access to or modification of their software. False Reality disturbs VR system services by tampering with sensor measurements, and further spoofing users' perception even inducing harmful actions, e.g., inducing dizziness or causing users to crash into obstacles, by exploiting perceptual and psychological effects. We formalize these threats through an attack pathway framework and validate three representative pathways via physical experiments and user studies on five commercial VR devices. Finally, we further propose a defense prototype to mitigate such threats. Our findings shall provide valuable insights for enhancing the security and resilience of future VR systems.

http://arxiv.org/abs/2508.10039
Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries. (98%)
Wenqiang Wang; Yan Xiao; Hao Lin; Yangshijie Zhang; Xiaochun Cao
Current multi-task adversarial text attacks rely on abundant access to shared internal features and numerous queries, often limited to a single task type. As a result, these attacks are less effective against practical scenarios involving black-box feedback APIs, limited queries, or multiple task types. To bridge this gap, we propose \textbf{C}luster and \textbf{E}nsemble \textbf{M}ulti-task Text Adversarial \textbf{A}ttack (\textbf{CEMA}), an effective black-box attack that exploits the transferability of adversarial texts across different tasks. CEMA simplifies complex multi-task scenarios by using a \textit{deep-level substitute model} trained in a \textit{plug-and-play} manner for text classification, enabling attacks without mimicking the victim model. This approach requires only a few queries for training, converting multi-task attacks into classification attacks and allowing attacks across various tasks.
  CEMA generates multiple adversarial candidates using different text classification methods and selects the one that most effectively attacks substitute models.
  In experiments involving multi-task models with two, three, or six tasks--spanning classification, translation, summarization, and text-to-image generation--CEMA demonstrates significant attack success with as few as 100 queries. Furthermore, CEMA can target commercial APIs (e.g., Baidu and Google Translate), large language models (e.g., ChatGPT 4o), and image-generation models (e.g., Stable Diffusion V2), showcasing its versatility and effectiveness in real-world applications.

http://arxiv.org/abs/2508.10038
Certifiably robust malware detectors by design. (98%)
Pierre-Francois Gimenez; Sarath Sivaprasad; Mario Fritz
Malware analysis involves analyzing suspicious software to detect malicious payloads. Static malware analysis, which does not require software execution, relies increasingly on machine learning techniques to achieve scalability. Although such techniques obtain very high detection accuracy, they can be easily evaded with adversarial examples where a few modifications of the sample can dupe the detector without modifying the behavior of the software. Unlike other domains, such as computer vision, creating an adversarial example of malware without altering its functionality requires specific transformations. We propose a new model architecture for certifiably robust malware detection by design. In addition, we show that every robust detector can be decomposed into a specific structure, which can be applied to learn empirically robust malware detectors, even on fragile features. Our framework ERDALT is based on this structure. We compare and validate these approaches with machine-learning-based malware detection methods, allowing for robust detection with limited reduction of detection performance.

http://arxiv.org/abs/2508.07281
Representation Understanding via Activation Maximization. (81%)
Hongbo Zhu; Angelo Cangelosi
Understanding internal feature representations of deep neural networks (DNNs) is a fundamental step toward model interpretability. Inspired by neuroscience methods that probe biological neurons using visual stimuli, recent deep learning studies have employed Activation Maximization (AM) to synthesize inputs that elicit strong responses from artificial neurons. In this work, we propose a unified feature visualization framework applicable to both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). Unlike prior efforts that predominantly focus on the last output-layer neurons in CNNs, we extend feature visualization to intermediate layers as well, offering deeper insights into the hierarchical structure of learned feature representations. Furthermore, we investigate how activation maximization can be leveraged to generate adversarial examples, revealing potential vulnerabilities and decision boundaries of DNNs. Our experiments demonstrate the effectiveness of our approach in both traditional CNNs and modern ViT, highlighting its generalizability and interpretive value.

http://arxiv.org/abs/2508.07458
Towards Unveiling Predictive Uncertainty Vulnerabilities in the Context of the Right to Be Forgotten. (56%)
Wei Qian; Chenxu Zhao; Yangyi Li; Wenqian Ye; Mengdi Huai
Currently, various uncertainty quantification methods have been proposed to provide certainty and probability estimates for deep learning models' label predictions. Meanwhile, with the growing demand for the right to be forgotten, machine unlearning has been extensively studied as a means to remove the impact of requested sensitive data from a pre-trained model without retraining the model from scratch. However, the vulnerabilities of such generated predictive uncertainties with regard to dedicated malicious unlearning attacks remain unexplored. To bridge this gap, for the first time, we propose a new class of malicious unlearning attacks against predictive uncertainties, where the adversary aims to cause the desired manipulations of specific predictive uncertainty results. We also design novel optimization frameworks for our attacks and conduct extensive experiments, including black-box scenarios. Notably, our extensive experiments show that our attacks are more effective in manipulating predictive uncertainties than traditional attacks that focus on label misclassifications, and existing defenses against conventional attacks are ineffective against our attacks.

http://arxiv.org/abs/2508.07139
A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detection. (13%)
Ivan Zhang
Ensuring LLM alignment is critical to information security as AI models become increasingly widespread and integrated in society. Unfortunately, many defenses against adversarial attacks and jailbreaking on LLMs cannot adapt quickly to new attacks, degrade model responses to benign prompts, or introduce significant barriers to scalable implementation. To mitigate these challenges, we introduce a real-time, self-tuning (RTST) moderator framework to defend against adversarial attacks while maintaining a lightweight training footprint. We empirically evaluate its effectiveness using Google's Gemini models against modern, effective jailbreaks. Our results demonstrate the advantages of an adaptive, minimally intrusive framework for jailbreak defense over traditional fine-tuning or classifier models.

http://arxiv.org/abs/2508.07263
Fading the Digital Ink: A Universal Black-Box Attack Framework for 3DGS Watermarking Systems. (3%)
Qingyuan Zeng; Shu Jiang; Jiajing Lin; Zhenzhong Wang; Kay Chen Tan; Min Jiang
With the rise of 3D Gaussian Splatting (3DGS), a variety of digital watermarking techniques, embedding either 1D bitstreams or 2D images, are used for copyright protection. However, the robustness of these watermarking techniques against potential attacks remains underexplored. This paper introduces the first universal black-box attack framework, the Group-based Multi-objective Evolutionary Attack (GMEA), designed to challenge these watermarking systems. We formulate the attack as a large-scale multi-objective optimization problem, balancing watermark removal with visual quality. In a black-box setting, we introduce an indirect objective function that blinds the watermark detector by minimizing the standard deviation of features extracted by a convolutional network, thus rendering the feature maps uninformative. To manage the vast search space of 3DGS models, we employ a group-based optimization strategy to partition the model into multiple, independent sub-optimization problems. Experiments demonstrate that our framework effectively removes both 1D and 2D watermarks from mainstream 3DGS watermarking methods while maintaining high visual fidelity. This work reveals critical vulnerabilities in existing 3DGS copyright protection schemes and calls for the development of more robust watermarking systems.

http://arxiv.org/abs/2508.10031
Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMs. (98%)
Jinhwa Kim; Ian G. Harris
While Large Language Models (LLMs) have shown significant advancements in performance, various jailbreak attacks have posed growing safety and ethical risks. Malicious users often exploit adversarial context to deceive LLMs, prompting them to generate responses to harmful queries. In this study, we propose a new defense mechanism called Context Filtering model, an input pre-processing method designed to filter out untrustworthy and unreliable context while identifying the primary prompts containing the real user intent to uncover concealed malicious intent. Given that enhancing the safety of LLMs often compromises their helpfulness, potentially affecting the experience of benign users, our method aims to improve the safety of the LLMs while preserving their original performance. We evaluate the effectiveness of our model in defending against jailbreak attacks through comparative analysis, comparing our approach with state-of-the-art defense mechanisms against six different attacks and assessing the helpfulness of LLMs under these defenses. Our model demonstrates its ability to reduce the Attack Success Rates of jailbreak attacks by up to 88% while maintaining the original LLMs' performance, achieving state-of-the-art Safety and Helpfulness Product results. Notably, our model is a plug-and-play method that can be applied to all LLMs, including both white-box and black-box models, to enhance their safety without requiring any fine-tuning of the models themselves. We will make our model publicly available for research purposes.

http://arxiv.org/abs/2508.06913
Model-Agnostic Sentiment Distribution Stability Analysis for Robust LLM-Generated Texts Detection. (12%)
Siyuan Li; Xi Lin; Guangyan Li; Zehao Liu; Aodu Wulianghai; Li Ding; Jun Wu; Jianhua Li
The rapid advancement of large language models (LLMs) has resulted in increasingly sophisticated AI-generated content, posing significant challenges in distinguishing LLM-generated text from human-written language. Existing detection methods, primarily based on lexical heuristics or fine-tuned classifiers, often suffer from limited generalizability and are vulnerable to paraphrasing, adversarial perturbations, and cross-domain shifts. In this work, we propose SentiDetect, a model-agnostic framework for detecting LLM-generated text by analyzing the divergence in sentiment distribution stability. Our method is motivated by the empirical observation that LLM outputs tend to exhibit emotionally consistent patterns, whereas human-written texts display greater emotional variability. To capture this phenomenon, we define two complementary metrics: sentiment distribution consistency and sentiment distribution preservation, which quantify stability under sentiment-altering and semantic-preserving transformations. We evaluate SentiDetect on five diverse datasets and a range of advanced LLMs,including Gemini-1.5-Pro, Claude-3, GPT-4-0613, and LLaMa-3.3. Experimental results demonstrate its superiority over state-of-the-art baselines, with over 16% and 11% F1 score improvements on Gemini-1.5-Pro and GPT-4-0613, respectively. Moreover, SentiDetect also shows greater robustness to paraphrasing, adversarial attacks, and text length variations, outperforming existing detectors in challenging scenarios.

http://arxiv.org/abs/2508.07066
Membership Inference Attacks with False Discovery Rate Control. (3%)
Chenxu Zhao; Wei Qian; Aobo Chen; Mengdi Huai
Recent studies have shown that deep learning models are vulnerable to membership inference attacks (MIAs), which aim to infer whether a data record was used to train a target model or not. To analyze and study these vulnerabilities, various MIA methods have been proposed. Despite the significance and popularity of MIAs, existing works on MIAs are limited in providing guarantees on the false discovery rate (FDR), which refers to the expected proportion of false discoveries among the identified positive discoveries. However, it is very challenging to ensure the false discovery rate guarantees, because the underlying distribution is usually unknown, and the estimated non-member probabilities often exhibit interdependence. To tackle the above challenges, in this paper, we design a novel membership inference attack method, which can provide the guarantees on the false discovery rate. Additionally, we show that our method can also provide the marginal probability guarantee on labeling true non-member data as member data. Notably, our method can work as a wrapper that can be seamlessly integrated with existing MIA methods in a post-hoc manner, while also providing the FDR control. We perform the theoretical analysis for our method. Extensive experiments in various settings (e.g., the black-box setting and the lifelong learning setting) are also conducted to verify the desirable performance of our method.

http://arxiv.org/abs/2508.07053
SPARE: Securing Progressive Web Applications Against Unauthorized Replications. (2%)
Sajib Talukder; Nur Imtiazul Haque; Khandakar Ashrafi Akbar
WebView applications are widely used in mobile applications to display web content directly within the app, enhancing user engagement by eliminating the need to open an external browser and providing a seamless experience. Progressive Web Applications (PWAs) further improve usability by combining the accessibility of web apps with the speed, offline capabilities, and responsiveness of native applications. However, malicious developers can exploit this technology by duplicating PWA web links to create counterfeit native apps, monetizing through user diversion. This unethical practice poses significant risks to users and the original application developers, underscoring the need for robust security measures to prevent unauthorized replication. Considering the one-way communication of Trusted Web Activity (a method for integrating web content into Android applications) and PWAs, we propose a query parameter-based practical security solution to defend against or mitigate such attacks. We analyze the vulnerabilities of our proposed security solution to assess its effectiveness and introduce advanced measures to address any identified weaknesses, presenting a comprehensive defense framework. As part of our work, we developed a prototype web application that secures PWAs from replication by embedding a combination of Unix timestamps and device identifiers into the query parameters. We evaluate the effectiveness of this defense strategy by simulating an advanced attack scenario. Additionally, we created a realistic dataset reflecting mobile app user behavior, modeled using a Zipfian distribution, to validate our framework.

http://arxiv.org/abs/2508.07115
Sensory robustness through top-down feedback and neural stochasticity in recurrent vision models. (2%)
Antonino Greco; Marco D'Alessandro; Karl J. Friston; Giovanni Pezzulo; Markus Siegel
Biological systems leverage top-down feedback for visual processing, yet most artificial vision models succeed in image classification using purely feedforward or recurrent architectures, calling into question the functional significance of descending cortical pathways. Here, we trained convolutional recurrent neural networks (ConvRNN) on image classification in the presence or absence of top-down feedback projections to elucidate the specific computational contributions of those feedback pathways. We found that ConvRNNs with top-down feedback exhibited remarkable speed-accuracy trade-off and robustness to noise perturbations and adversarial attacks, but only when they were trained with stochastic neural variability, simulated by randomly silencing single units via dropout. By performing detailed analyses to identify the reasons for such benefits, we observed that feedback information substantially shaped the representational geometry of the post-integration layer, combining the bottom-up and top-down streams, and this effect was amplified by dropout. Moreover, feedback signals coupled with dropout optimally constrained network activity onto a low-dimensional manifold and encoded object information more efficiently in out-of-distribution regimes, with top-down information stabilizing the representational dynamics at the population level. Together, these findings uncover a dual mechanism for resilient sensory coding. On the one hand, neural stochasticity prevents unit-level co-adaptation albeit at the cost of more chaotic dynamics. On the other hand, top-down feedback harnesses high-level information to stabilize network activity on compact low-dimensional manifolds.

http://arxiv.org/abs/2508.10035
Neural Network-Based Detection and Multi-Class Classification of FDI Attacks in Smart Grid Home Energy Systems. (1%)
Varsha Sen; Biswash Basnet
False Data Injection Attacks (FDIAs) pose a significant threat to smart grid infrastructures, particularly Home Area Networks (HANs), where real-time monitoring and control are highly adopted. Owing to the comparatively less stringent security controls and widespread availability of HANs, attackers view them as an attractive entry point to manipulate aggregated demand patterns, which can ultimately propagate and disrupt broader grid operations. These attacks undermine the integrity of smart meter data, enabling malicious actors to manipulate consumption values without activating conventional alarms, thereby creating serious vulnerabilities across both residential and utility-scale infrastructures. This paper presents a machine learning-based framework for both the detection and classification of FDIAs using residential energy data. A real-time detection is provided by the lightweight Artificial Neural Network (ANN), which works by using the most vital features of energy consumption, cost, and time context. For the classification of different attack types, a Bidirectional LSTM is trained to recognize normal, trapezoidal, and sigmoid attack shapes through learning sequential dependencies in the data. A synthetic time-series dataset was generated to emulate realistic household behaviour. Experimental results demonstrate that the proposed models are effective in identifying and classifying FDIAs, offering a scalable solution for enhancing grid resilience at the edge. This work contributes toward building intelligent, data-driven defence mechanisms that strengthen smart grid cybersecurity from residential endpoints.

http://arxiv.org/abs/2508.06837
Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Models. (1%)
Shiqian Zhao; Chong Wang; Yiming Li; Yihao Huang; Wenjie Qu; Siew-Kei Lam; Yi Xie; Kangjie Chen; Jie Zhang; Tianwei Zhang
Text-to-Image (T2I) models, represented by DALL$\cdot$E and Midjourney, have gained huge popularity for creating realistic images. The quality of these images relies on the carefully engineered prompts, which have become valuable intellectual property. While skilled prompters showcase their AI-generated art on markets to attract buyers, this business incidentally exposes them to \textit{prompt stealing attacks}. Existing state-of-the-art attack techniques reconstruct the prompts from a fixed set of modifiers (i.e., style descriptions) with model-specific training, which exhibit restricted adaptability and effectiveness to diverse showcases (i.e., target images) and diffusion models.
  To alleviate these limitations, we propose Prometheus, a training-free, proxy-in-the-loop, search-based prompt-stealing attack, which reverse-engineers the valuable prompts of the showcases by interacting with a local proxy model. It consists of three innovative designs. First, we introduce dynamic modifiers, as a supplement to static modifiers used in prior works. These dynamic modifiers provide more details specific to the showcases, and we exploit NLP analysis to generate them on the fly. Second, we design a contextual matching algorithm to sort both dynamic and static modifiers. This offline process helps reduce the search space of the subsequent step. Third, we interact with a local proxy model to invert the prompts with a greedy search algorithm. Based on the feedback guidance, we refine the prompt to achieve higher fidelity. The evaluation results show that Prometheus successfully extracts prompts from popular platforms like PromptBase and AIFrog against diverse victim models, including Midjourney, Leonardo.ai, and DALL$\cdot$E, with an ASR improvement of 25.0\%. We also validate that Prometheus is resistant to extensive potential defenses, further highlighting its severity in practice.

http://arxiv.org/abs/2508.06127
SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures. (93%)
Yi Qin; Rui Wang; Tao Huang; Tong Xiao; Liping Jing
While the Segment Anything Model (SAM) transforms interactive segmentation with zero-shot abilities, its inherent vulnerabilities present a single-point risk, potentially leading to the failure of numerous downstream applications. Proactively evaluating these transferable vulnerabilities is thus imperative. Prior adversarial attacks on SAM often present limited transferability due to insufficient exploration of common weakness across domains. To address this, we propose Vertex-Refining Simplicial Complex Attack (VeSCA), a novel method that leverages only the encoder of SAM for generating transferable adversarial examples. Specifically, it achieves this by explicitly characterizing the shared vulnerable regions between SAM and downstream models through a parametric simplicial complex. Our goal is to identify such complexes within adversarially potent regions by iterative vertex-wise refinement. A lightweight domain re-adaptation strategy is introduced to bridge domain divergence using minimal reference data during the initialization of simplicial complex. Ultimately, VeSCA generates consistently transferable adversarial examples through random simplicial complex sampling. Extensive experiments demonstrate that VeSCA achieves performance improved by 12.7% compared to state-of-the-art methods across three downstream model categories across five domain-specific datasets. Our findings further highlight the downstream model risks posed by SAM's vulnerabilities and emphasize the urgency of developing more robust foundation models.

http://arxiv.org/abs/2412.05734
LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage. (92%)
Yuzhou Nie; Zhun Wang; Ye Yu; Xian Wu; Xuandong Zhao; Wenbo Guo; Dawn Song
Recent studies have discovered that large language models (LLM) may be ``fooled'' to output private information, including training data, system prompts, and personally identifiable information, under carefully crafted adversarial prompts. Existing red-teaming approaches for privacy leakage either rely on manual efforts or focus solely on system prompt extraction, making them ineffective for severe risks of training data leakage. We propose LeakAgent, a novel black-box red-teaming framework for LLM privacy leakage. Our framework trains an open-source LLM through reinforcement learning as the attack agent to generate adversarial prompts for both training data extraction and system prompt extraction. To achieve this, we propose a novel reward function to provide effective and fine-grained rewards and design novel mechanisms to balance exploration and exploitation during learning and enhance the diversity of adversarial prompts. Through extensive evaluations, we first show that LeakAgent significantly outperforms existing rule-based approaches in training data extraction and automated methods in system prompt leakage. We also demonstrate the effectiveness of LeakAgent in extracting system prompts from real-world applications in OpenAI's GPT Store. We further demonstrate LeakAgent's effectiveness in evading the existing guardrail defense and its helpfulness in enabling better safety alignment. Finally, we validate our customized designs through a detailed ablation study. We release our code here https://github.com/rucnyz/LeakAgent.

http://arxiv.org/abs/2508.06153
SLIP: Soft Label Mechanism and Key-Extraction-Guided CoT-based Defense Against Instruction Backdoor in APIs. (89%)
Zhengxian Wu; Juan Wen; Wanli Peng; Haowei Chang; Yinghan Zhou; Yiming Xue
With the development of customized large language model (LLM) agents, a new threat of black-box backdoor attacks has emerged, where malicious instructions are injected into hidden system prompts. These attacks easily bypass existing defenses that rely on white-box access, posing a serious security challenge. To address this, we propose SLIP, a Soft Label mechanism and key-extraction-guided CoT-based defense against Instruction backdoors in APIs. SLIP is designed based on two key insights. First, to counteract the model's oversensitivity to triggers, we propose a Key-extraction-guided Chain-of-Thought (KCoT). Instead of only considering the single trigger or the input sentence, KCoT prompts the agent to extract task-relevant key phrases. Second, to guide the LLM toward correct answers, our proposed Soft Label Mechanism (SLM) prompts the agent to quantify the semantic correlation between key phrases and candidate answers. Crucially, to mitigate the influence of residual triggers or misleading content in phrases extracted by KCoT, which typically causes anomalous scores, SLM excludes anomalous scores deviating significantly from the mean and subsequently averages the remaining scores to derive a more reliable semantic representation. Extensive experiments on classification and question-answer (QA) tasks demonstrate that SLIP is highly effective, reducing the average attack success rate (ASR) from 90.2% to 25.13% while maintaining high accuracy on clean data and outperforming state-of-the-art defenses. Our code are available in https://github.com/CAU-ISS-Lab/Backdoor-Attack-Defense-LLMs/tree/main/SLIP.

http://arxiv.org/abs/2508.06394
When AIOps Become "AI Oops": Subverting LLM-driven IT Operations via Telemetry Manipulation. (69%)
Dario Pasquini; Evgenios M. Kornaropoulos; Giuseppe Ateniese; Omer Akgul; Athanasios Theocharis; Petros Efstathopoulos
AI for IT Operations (AIOps) is transforming how organizations manage complex software systems by automating anomaly detection, incident diagnosis, and remediation. Modern AIOps solutions increasingly rely on autonomous LLM-based agents to interpret telemetry data and take corrective actions with minimal human intervention, promising faster response times and operational cost savings.
  In this work, we perform the first security analysis of AIOps solutions, showing that, once again, AI-driven automation comes with a profound security cost. We demonstrate that adversaries can manipulate system telemetry to mislead AIOps agents into taking actions that compromise the integrity of the infrastructure they manage. We introduce techniques to reliably inject telemetry data using error-inducing requests that influence agent behavior through a form of adversarial reward-hacking; plausible but incorrect system error interpretations that steer the agent's decision-making. Our attack methodology, AIOpsDoom, is fully automated--combining reconnaissance, fuzzing, and LLM-driven adversarial input generation--and operates without any prior knowledge of the target system.
  To counter this threat, we propose AIOpsShield, a defense mechanism that sanitizes telemetry data by exploiting its structured nature and the minimal role of user-generated content. Our experiments show that AIOpsShield reliably blocks telemetry-based attacks without affecting normal agent performance.
  Ultimately, this work exposes AIOps as an emerging attack vector for system compromise and underscores the urgent need for security-aware AIOps design.

http://arxiv.org/abs/2502.05341
Neural Encrypted State Transduction for Ransomware Classification: A Novel Approach Using Cryptographic Flow Residuals. (67%)
Barnaby Fortescue; Edmund Hawksmoor; Alistair Wetherington; Frederick Marlowe; Kevin Pekepok
Encrypted behavioral patterns provide a unique avenue for classifying complex digital threats without reliance on explicit feature extraction, enabling detection frameworks to remain effective even when conventional static and behavioral methodologies fail. A novel approach based on Neural Encrypted State Transduction (NEST) is introduced to analyze cryptographic flow residuals and classify threats through their encrypted state transitions, mitigating evasion tactics employed through polymorphic and obfuscated attack strategies. The mathematical formulation of NEST leverages transduction principles to map state transitions dynamically, enabling high-confidence classification without requiring direct access to decrypted execution traces. Experimental evaluations demonstrate that the proposed framework achieves improved detection accuracy across multiple ransomware families while exhibiting resilience against adversarial perturbations and previously unseen attack variants. The model maintains competitive processing efficiency, offering a practical balance between classification performance and computational resource constraints, making it suitable for large-scale security deployments. Comparative assessments reveal that NEST consistently outperforms baseline classification models, particularly in detecting ransomware samples employing delayed encryption, entropy-based obfuscation, and memory-resident execution techniques. The capacity to generalize across diverse execution environments reinforces the applicability of encrypted transduction methodologies in adversarial classification tasks beyond conventional malware detection pipelines. The integration of residual learning mechanisms within the transduction layers further enhances classification robustness, minimizing both false positives and misclassification rates across varied operational contexts.

http://arxiv.org/abs/2508.06073
ProvX: Generating Counterfactual-Driven Attack Explanations for Provenance-Based Detection. (62%)
Weiheng Wu; Wei Qiao; Teng Li; Yebo Feng; Zhuo Ma; Jianfeng Ma; Yang Liu
Provenance graph-based intrusion detection systems are deployed on hosts to defend against increasingly severe Advanced Persistent Threat. Using Graph Neural Networks to detect these threats has become a research focus and has demonstrated exceptional performance. However, the widespread adoption of GNN-based security models is limited by their inherent black-box nature, as they fail to provide security analysts with any verifiable explanations for model predictions or any evidence regarding the model's judgment in relation to real-world attacks. To address this challenge, we propose ProvX, an effective explanation framework for exlaining GNN-based security models on provenance graphs. ProvX introduces counterfactual explanation logic, seeking the minimal structural subset within a graph predicted as malicious that, when perturbed, can subvert the model's original prediction. We innovatively transform the discrete search problem of finding this critical subgraph into a continuous optimization task guided by a dual objective of prediction flipping and distance minimization. Furthermore, a Staged Solidification strategy is incorporated to enhance the precision and stability of the explanations. We conducted extensive evaluations of ProvX on authoritative datasets. The experimental results demonstrate that ProvX can locate critical graph structures that are highly relevant to real-world attacks and achieves an average explanation necessity of 51.59\%, with these metrics outperforming current SOTA explainers. Furthermore, we explore and provide a preliminary validation of a closed-loop Detection-Explanation-Feedback enhancement framework, demonstrating through experiments that the explanation results from ProvX can guide model optimization, effectively enhancing its robustness against adversarial attacks.

http://arxiv.org/abs/2501.01872
Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions. (62%)
Rachneet Sachdeva; Rima Hazra; Iryna Gurevych
Large language models, despite extensive alignment with human values and ethical principles, remain vulnerable to sophisticated jailbreak attacks that exploit their reasoning abilities. Existing safety measures often detect overt malicious intent but fail to address subtle, reasoning-driven vulnerabilities. In this work, we introduce POATE (Polar Opposite query generation, Adversarial Template construction, and Elaboration), a novel jailbreak technique that harnesses contrastive reasoning to provoke unethical responses. POATE crafts semantically opposing intents and integrates them with adversarial templates, steering models toward harmful outputs with remarkable subtlety. We conduct extensive evaluation across six diverse language model families of varying parameter sizes to demonstrate the robustness of the attack, achieving significantly higher attack success rates (~44%) compared to existing methods. To counter this, we propose Intent-Aware CoT and Reverse Thinking CoT, which decompose queries to detect malicious intent and reason in reverse to evaluate and reject harmful responses. These methods enhance reasoning robustness and strengthen the model's defense against adversarial exploits.

http://arxiv.org/abs/2508.06601
Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs. (41%)
Kyle O'Brien; Stephen Casper; Quentin Anthony; Tomek Korbak; Robert Kirk; Xander Davies; Ishan Mishra; Geoffrey Irving; Yarin Gal; Stella Biderman
Open-weight AI systems offer unique benefits, including enhanced transparency, open research, and decentralized access. However, they are vulnerable to tampering attacks which can efficiently elicit harmful behaviors by modifying weights or activations. Currently, there is not yet a robust science of open-weight model risk management. Existing safety fine-tuning methods and other post-training techniques have struggled to make LLMs resistant to more than a few dozen steps of adversarial fine-tuning. In this paper, we investigate whether filtering text about dual-use topics from training data can prevent unwanted capabilities and serve as a more tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable data filtering and show that it offers a tractable and effective method for minimizing biothreat proxy knowledge in LLMs. We pretrain multiple 6.9B-parameter models from scratch and find that they exhibit substantial resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M tokens of biothreat-related text -- outperforming existing post-training baselines by over an order of magnitude -- with no observed degradation to unrelated capabilities. However, while filtered models lack internalized dangerous knowledge, we find that they can still leverage such information when it is provided in context (e.g., via search tool augmentation), demonstrating a need for a defense-in-depth approach. Overall, these findings help to establish pretraining data curation as a promising layer of defense for open-weight AI systems.

http://arxiv.org/abs/2411.14110
Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications. (38%)
Changyue Jiang; Xudong Pan; Geng Hong; Chenfu Bao; Yang Chen; Min Yang
Retrieval-Augmented Generation (RAG) expands the knowledge boundary of large language models (LLMs) by integrating external knowledge bases, whose construction is often time-consuming and laborious. If an adversary extracts the knowledge base verbatim, it not only severely infringes the owner's intellectual property but also enables the adversary to replicate the application's functionality for unfair competition. Previous works on knowledge base extraction are limited either by low extraction coverage (usually less than 4%) in query-based attacks or by impractical assumptions of white-box access in embedding-based optimization methods. In this work, we propose CopyBreakRAG, an agent-based black-box attack that reasons from feedback and adaptively generates new adversarial queries for progressive extraction. By balancing exploration and exploitation through curiosity-driven queries and feedback-guided query refinement, our method overcomes the limitations of prior approaches and achieves significantly higher extraction coverage in realistic black-box settings. Experimental results show that CopyBreakRAG outperforms the state-of-the-art black-box approach by 45% on average in terms of chunk extraction ratio from applications built with mainstream RAG frameworks, and extracts over 70% of the data from the knowledge base in applications on commercial platforms including OpenAI's GPTs and ByteDance's Coze when essential protection is in place.

http://arxiv.org/abs/2508.10029
Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs. (12%)
Wenpeng Xing; Mohan Li; Chunqiang Hu; Haitao XuNingyu Zhang; Bo Lin; Meng Han
Large language models (LLMs) demonstrate impressive capabilities in various language tasks but are susceptible to jailbreak attacks that circumvent their safety alignments. This paper introduces Latent Fusion Jailbreak (LFJ), a representation-based attack that interpolates hidden states from harmful and benign query pairs to elicit prohibited responses. LFJ begins by selecting query pairs with high thematic and syntactic similarity, then performs gradient-guided interpolation at influential layers and tokens, followed by optimization to balance attack success, output fluency, and computational efficiency. Evaluations on models such as Vicuna and LLaMA-2 across benchmarks like AdvBench and MaliciousInstruct yield an average attack success rate (ASR) of 94.01%, outperforming existing methods. To mitigate LFJ, we propose an adversarial training defense that fine-tunes models on interpolated examples, reducing ASR by over 80% without degrading performance on benign inputs. Ablation studies validate the importance of query pair selection, hidden state interpolation components, and optimization strategies in LFJ's effectiveness.

http://arxiv.org/abs/2501.15509
FIT-Print: Towards False-claim-resistant Model Ownership Verification via Targeted Fingerprint. (10%)
Shuo Shao; Haozhe Zhu; Hongwei Yao; Yiming Li; Tianwei Zhang; Zhan Qin
Model fingerprinting is a widely adopted approach to safeguard the intellectual property rights of open-source models by preventing their unauthorized reuse. It is promising and convenient since it does not necessitate modifying the protected model. In this paper, we revisit existing fingerprinting methods and reveal that they are vulnerable to false claim attacks where adversaries falsely assert ownership of any third-party model. We demonstrate that this vulnerability mostly stems from their untargeted nature, where they generally compare the outputs of given samples on different models instead of the similarities to specific references. Motivated by these findings, we propose a targeted fingerprinting paradigm (i.e., FIT-Print) to counteract false claim attacks. Specifically, FIT-Print transforms the fingerprint into a targeted signature via optimization. Building on the principles of FIT-Print, we develop bit-wise and list-wise black-box model fingerprinting methods, i.e., FIT-ModelDiff and FIT-LIME, which exploit the distance between model outputs and the feature attribution of specific samples as the fingerprint, respectively. Extensive experiments on benchmark models and datasets verify the effectiveness, conferrability, and resistance to false claim attacks of our FIT-Print.

http://arxiv.org/abs/2508.06059
Fact2Fiction: Targeted Poisoning Attack to Agentic Fact-checking System. (4%)
Haorui He; Yupeng Li; Bin Benjamin Zhu; Dacheng Wen; Reynold Cheng; Francis C. M. Lau
State-of-the-art fact-checking systems combat misinformation at scale by employing autonomous LLM-based agents to decompose complex claims into smaller sub-claims, verify each sub-claim individually, and aggregate the partial results to produce verdicts with justifications (explanatory rationales for the verdicts). The security of these systems is crucial, as compromised fact-checkers, which tend to be easily underexplored, can amplify misinformation. This work introduces Fact2Fiction, the first poisoning attack framework targeting such agentic fact-checking systems. Fact2Fiction mirrors the decomposition strategy and exploits system-generated justifications to craft tailored malicious evidences that compromise sub-claim verification. Extensive experiments demonstrate that Fact2Fiction achieves 8.9\%--21.2\% higher attack success rates than state-of-the-art attacks across various poisoning budgets. Fact2Fiction exposes security weaknesses in current fact-checking systems and highlights the need for defensive countermeasures.

http://arxiv.org/abs/2508.06656
Towards Robust Red-Green Watermarking for Autoregressive Image Generators. (2%)
Denis Lukovnikov; Andreas Müller; Erwin Quiring; Asja Fischer
In-generation watermarking for detecting and attributing generated content has recently been explored for latent diffusion models (LDMs), demonstrating high robustness. However, the use of in-generation watermarks in autoregressive (AR) image models has not been explored yet. AR models generate images by autoregressively predicting a sequence of visual tokens that are then decoded into pixels using a vector-quantized decoder. Inspired by red-green watermarks for large language models, we examine token-level watermarking schemes that bias the next-token prediction based on prior tokens. We find that a direct transfer of these schemes works in principle, but the detectability of the watermarks decreases considerably under common image perturbations. As a remedy, we propose two novel watermarking methods that rely on visual token clustering to assign similar tokens to the same set. Firstly, we investigate a training-free approach that relies on a cluster lookup table, and secondly, we finetune VAE encoders to predict token clusters directly from perturbed images. Overall, our experiments show that cluster-level watermarks improve robustness against perturbations and regeneration attacks while preserving image quality. Cluster classification further boosts watermark detectability, outperforming a set of baselines. Moreover, our methods offer fast verification runtime, comparable to lightweight post-hoc watermarking methods.

http://arxiv.org/abs/2502.03687
Conditional Diffusion Models are Medical Image Classifiers that Provide Explainability and Uncertainty for Free. (1%)
Gian Mario Favero; Parham Saremi; Emily Kaczmarek; Brennan Nichyporuk; Tal Arbel
Discriminative classifiers have become a foundational tool in deep learning for medical imaging, excelling at learning separable features of complex data distributions. However, these models often need careful design, augmentation, and training techniques to ensure safe and reliable deployment. Recently, diffusion models have become synonymous with generative modeling in 2D. These models showcase robustness across a range of tasks including natural image classification, where classification is performed by comparing reconstruction errors across images generated for each possible conditioning input. This work presents the first exploration of the potential of class conditional diffusion models for 2D medical image classification. First, we develop a novel majority voting scheme shown to improve the performance of medical diffusion classifiers. Next, extensive experiments on the CheXpert and ISIC Melanoma skin cancer datasets demonstrate that foundation and trained-from-scratch diffusion models achieve competitive performance against SOTA discriminative classifiers without the need for explicit supervision. In addition, we show that diffusion classifiers are intrinsically explainable, and can be used to quantify the uncertainty of their predictions, increasing their trustworthiness and reliability in safety-critical, clinical contexts. Further information is available on our project page: https://faverogian.github.io/med-diffusion-classifier.github.io/.

http://arxiv.org/abs/2508.06346
Introducing Fractional Classification Loss for Robust Learning with Noisy Labels. (1%)
Mert Can Kurucu; Tufan Kumbasar; İbrahim Eksin; Müjde Güzelkaya
Robust loss functions are crucial for training deep neural networks in the presence of label noise, yet existing approaches require extensive, dataset-specific hyperparameter tuning. In this work, we introduce Fractional Classification Loss (FCL), an adaptive robust loss that automatically calibrates its robustness to label noise during training. Built within the active-passive loss framework, FCL employs the fractional derivative of the Cross-Entropy (CE) loss as its active component and the Mean Absolute Error (MAE) as its passive loss component. With this formulation, we demonstrate that the fractional derivative order $μ$ spans a family of loss functions that interpolate between MAE-like robustness and CE-like fast convergence. Furthermore, we integrate $μ$ into the gradient-based optimization as a learnable parameter and automatically adjust it to optimize the trade-off between robustness and convergence speed. We reveal that FCL's unique property establishes a critical trade-off that enables the stable learning of $μ$: lower log penalties on difficult or mislabeled examples improve robustness but impose higher penalties on easy or clean data, reducing model confidence in them. Consequently, FCL can dynamically reshape its loss landscape to achieve effective classification performance under label noise. Extensive experiments on benchmark datasets show that FCL achieves state-of-the-art results without the need for manual hyperparameter tuning.

http://arxiv.org/abs/2505.07258
No Query, No Access. (99%)
Wenqiang Wang; Siyuan Liang; Yangshijie Zhang; Xiaojun Jia; Hao Lin; Xiaochun Cao
Textual adversarial attacks mislead NLP models, including Large Language Models (LLMs), by subtly modifying text. While effective, existing attacks often require knowledge of the victim model, extensive queries, or access to training data, limiting real-world feasibility. To overcome these constraints, we introduce the \textbf{Victim Data-based Adversarial Attack (VDBA)}, which operates using only victim texts. To prevent access to the victim model, we create a shadow dataset with publicly available pre-trained models and clustering methods as a foundation for developing substitute models. To address the low attack success rate (ASR) due to insufficient information feedback, we propose the hierarchical substitution model design, generating substitute models to mitigate the failure of a single substitute model at the decision boundary.
  Concurrently, we use diverse adversarial example generation, employing various attack methods to generate and select the adversarial example with better similarity and attack effectiveness. Experiments on the Emotion and SST5 datasets show that VDBA outperforms state-of-the-art methods, achieving an ASR improvement of 52.08\% while significantly reducing attack queries to 0. More importantly, we discover that VDBA poses a significant threat to LLMs such as Qwen2 and the GPT family, and achieves the highest ASR of 45.99% even without access to the API, confirming that advanced NLP models still face serious security risks. Our codes can be found at https://anonymous.4open.science/r/VDBA-Victim-Data-based-Adversarial-Attack-36EC/

http://arxiv.org/abs/2508.03783
Probing and Enhancing the Robustness of GNN-based QEC Decoders with Reinforcement Learning. (99%)
Ryota Ikeda
Graph Neural Networks (GNNs) have emerged as a powerful, data-driven approach for Quantum Error Correction (QEC) decoding, capable of learning complex noise characteristics directly from syndrome data. However, the robustness of these decoders against subtle, adversarial perturbations remains a critical open question. This work introduces a novel framework to systematically probe the vulnerabilities of a GNN decoder using a reinforcement learning (RL) agent. The RL agent is trained as an adversary with the goal of finding minimal syndrome modifications that cause the decoder to misclassify. We apply this framework to a Graph Attention Network (GAT) decoder trained on experimental surface code data from Google Quantum AI. Our results show that the RL agent can successfully identify specific, critical vulnerabilities, achieving a high attack success rate with a minimal number of bit flips. Furthermore, we demonstrate that the decoder's robustness can be significantly enhanced through adversarial training, where the model is retrained on the adversarial examples generated by the RL agent. This iterative process of automated vulnerability discovery and targeted retraining presents a promising methodology for developing more reliable and robust neural network decoders for fault-tolerant quantum computing.

http://arxiv.org/abs/2508.05489
Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification. (93%)
Samuel Räber; Till Aczel; Andreas Plesner; Roger Wattenhofer
Previous work has suggested that preprocessing images through lossy compression can defend against adversarial perturbations, but comprehensive attack evaluations have been lacking. In this paper, we construct strong white-box and adaptive attacks against various compression models and identify a critical challenge for attackers: high realism in reconstructed images significantly increases attack difficulty. Through rigorous evaluation across multiple attack scenarios, we demonstrate that compression models capable of producing realistic, high-fidelity reconstructions are substantially more resistant to our attacks. In contrast, low-realism compression models can be broken. Our analysis reveals that this is not due to gradient masking. Rather, realistic reconstructions maintaining distributional alignment with natural images seem to offer inherent robustness. This work highlights a significant obstacle for future adversarial attacks and suggests that developing more effective techniques to overcome realism represents an essential challenge for comprehensive security evaluation.

http://arxiv.org/abs/2508.05167
PhysPatch: A Physically Realizable and Transferable Adversarial Patch Attack for Multimodal Large Language Models-based Autonomous Driving Systems. (91%)
Qi Guo; Xiaojun Jia; Shanmin Pang; Simeng Qin; Lin Wang; Ju Jia; Yang Liu; Qing Guo
Multimodal Large Language Models (MLLMs) are becoming integral to autonomous driving (AD) systems due to their strong vision-language reasoning capabilities. However, MLLMs are vulnerable to adversarial attacks, particularly adversarial patch attacks, which can pose serious threats in real-world scenarios. Existing patch-based attack methods are primarily designed for object detection models and perform poorly when transferred to MLLM-based systems due to the latter's complex architectures and reasoning abilities. To address these limitations, we propose PhysPatch, a physically realizable and transferable adversarial patch framework tailored for MLLM-based AD systems. PhysPatch jointly optimizes patch location, shape, and content to enhance attack effectiveness and real-world applicability. It introduces a semantic-based mask initialization strategy for realistic placement, an SVD-based local alignment loss with patch-guided crop-resize to improve transferability, and a potential field-based mask refinement method. Extensive experiments across open-source, commercial, and reasoning-capable MLLMs demonstrate that PhysPatch significantly outperforms prior methods in steering MLLM-based AD systems toward target-aligned perception and planning outputs. Moreover, PhysPatch consistently places adversarial patches in physically feasible regions of AD scenes, ensuring strong real-world applicability and deployability.

http://arxiv.org/abs/2508.00649
Revisiting Adversarial Patch Defenses on Object Detectors: Unified Evaluation, Large-Scale Dataset, and New Insights. (80%)
Junhao Zheng; Jiahao Sun; Chenhao Lin; Zhengyu Zhao; Chen Ma; Chong Zhang; Cong Wang; Qian Wang; Chao Shen
Developing reliable defenses against patch attacks on object detectors has attracted increasing interest. However, we identify that existing defense evaluations lack a unified and comprehensive framework, resulting in inconsistent and incomplete assessments of current methods. To address this issue, we revisit 11 representative defenses and present the first patch defense benchmark, involving 2 attack goals, 13 patch attacks, 11 object detectors, and 4 diverse metrics. This leads to the large-scale adversarial patch dataset with 94 types of patches and 94,000 images. Our comprehensive analyses reveal new insights: (1) The difficulty in defending against naturalistic patches lies in the data distribution, rather than the commonly believed high frequencies. Our new dataset with diverse patch distributions can be used to improve existing defenses by 15.09% AP@0.5. (2) The average precision of the attacked object, rather than the commonly pursued patch detection accuracy, shows high consistency with defense performance. (3) Adaptive attacks can substantially bypass existing defenses, and defenses with complex/stochastic models or universal patch properties are relatively robust. We hope that our analyses will serve as guidance on properly evaluating patch attacks/defenses and advancing their design. Code and dataset are available at https://github.com/Gandolfczjh/APDE, where we will keep integrating new attacks/defenses.

http://arxiv.org/abs/2508.05516
FS-IQA: Certified Feature Smoothing for Robust Image Quality Assessment. (74%)
Ekaterina Shumitskaya; Dmitriy Vatolin; Anastasia Antsiferova
We propose a novel certified defense method for Image Quality Assessment (IQA) models based on randomized smoothing with noise applied in the feature space rather than the input space. Unlike prior approaches that inject Gaussian noise directly into input images, often degrading visual quality, our method preserves image fidelity while providing robustness guarantees. To formally connect noise levels in the feature space with corresponding input-space perturbations, we analyze the maximum singular value of the backbone network's Jacobian. Our approach supports both full-reference (FR) and no-reference (NR) IQA models without requiring any architectural modifications, suitable for various scenarios. It is also computationally efficient, requiring a single backbone forward pass per image. Compared to previous methods, it reduces inference time by 99.5% without certification and by 20.6% when certification is applied. We validate our method with extensive experiments on two benchmark datasets, involving six widely-used FR and NR IQA models and comparisons against five state-of-the-art certified defenses. Our results demonstrate consistent improvements in correlation with subjective quality scores by up to 30.9%.

http://arxiv.org/abs/2508.05404
NT-ML: Backdoor Defense via Non-target Label Training and Mutual Learning. (69%)
Wenjie Huo; Katinka Wolter
Recent studies have shown that deep neural networks (DNNs) are vulnerable to backdoor attacks, where a designed trigger is injected into the dataset, causing erroneous predictions when activated. In this paper, we propose a novel defense mechanism, Non-target label Training and Mutual Learning (NT-ML), which can successfully restore the poisoned model under advanced backdoor attacks. NT aims to reduce the harm of poisoned data by retraining the model with the outputs of the standard training. At this stage, a teacher model with high accuracy on clean data and a student model with higher confidence in correct prediction on poisoned data are obtained. Then, the teacher and student can learn the strengths from each other through ML to obtain a purified student model. Extensive experiments show that NT-ML can effectively defend against 6 backdoor attacks with a small number of clean samples, and outperforms 5 state-of-the-art backdoor defenses.

http://arxiv.org/abs/2508.05600
Non-omniscient backdoor injection with a single poison sample: Proving the one-poison hypothesis for linear regression and linear classification. (26%)
Thorsten Peinemann; Paula Arnold; Sebastian Berndt; Thomas Eisenbarth; Esfandiar Mohammadi
Backdoor injection attacks are a threat to machine learning models that are trained on large data collected from untrusted sources; these attacks enable attackers to inject malicious behavior into the model that can be triggered by specially crafted inputs. Prior work has established bounds on the success of backdoor attacks and their impact on the benign learning task, however, an open question is what amount of poison data is needed for a successful backdoor attack. Typical attacks either use few samples, but need much information about the data points or need to poison many data points.
  In this paper, we formulate the one-poison hypothesis: An adversary with one poison sample and limited background knowledge can inject a backdoor with zero backdooring-error and without significantly impacting the benign learning task performance. Moreover, we prove the one-poison hypothesis for linear regression and linear classification. For adversaries that utilize a direction that is unused by the benign data distribution for the poison sample, we show that the resulting model is functionally equivalent to a model where the poison was excluded from training. We build on prior work on statistical backdoor learning to show that in all other cases, the impact on the benign learning task is still limited. We also validate our theoretical results experimentally with realistic benchmark data sets.

http://arxiv.org/abs/2508.05409
From Detection to Correction: Backdoor-Resilient Face Recognition via Vision-Language Trigger Detection and Noise-Based Neutralization. (26%)
Farah Wahida; M. A. P. Chamikara; Yashothara Shanmugarasa; Mohan Baruwal Chhetri; Thilina Ranbaduge; Ibrahim Khalil
Biometric systems, such as face recognition systems powered by deep neural networks (DNNs), rely on large and highly sensitive datasets. Backdoor attacks can subvert these systems by manipulating the training process. By inserting a small trigger, such as a sticker, make-up, or patterned mask, into a few training images, an adversary can later present the same trigger during authentication to be falsely recognized as another individual, thereby gaining unauthorized access. Existing defense mechanisms against backdoor attacks still face challenges in precisely identifying and mitigating poisoned images without compromising data utility, which undermines the overall reliability of the system. We propose a novel and generalizable approach, TrueBiometric: Trustworthy Biometrics, which accurately detects poisoned images using a majority voting mechanism leveraging multiple state-of-the-art large vision language models. Once identified, poisoned samples are corrected using targeted and calibrated corrective noise. Our extensive empirical results demonstrate that TrueBiometric detects and corrects poisoned images with 100\% accuracy without compromising accuracy on clean images. Compared to existing state-of-the-art approaches, TrueBiometric offers a more practical, accurate, and effective solution for mitigating backdoor attacks in face recognition systems.

http://arxiv.org/abs/2503.09336
Stealthy Patch-Wise Backdoor Attack in 3D Point Cloud via Curvature Awareness. (15%)
Yu Feng; Dingxin Zhang; Runkai Zhao; Yong Xia; Heng Huang; Weidong Cai
Backdoor attacks pose a severe threat to deep neural networks (DNNs) by implanting hidden backdoors that can be activated with predefined triggers to manipulate model behaviors maliciously. Existing 3D point cloud backdoor attacks primarily rely on sample-wise global modifications, which suffer from low imperceptibility. Although optimization can improve stealthiness, optimizing sample-wise triggers significantly increases computational cost. To address these limitations, we propose the Stealthy Patch-Wise Backdoor Attack (SPBA), the first patch-wise backdoor attack framework for 3D point clouds. Specifically, SPBA decomposes point clouds into local patches and employs a curvature-based imperceptibility score to guide trigger injection into visually less sensitive patches. By optimizing a unified patch-wise trigger that perturbs spectral features of selected patches, SPBA significantly enhances optimization efficiency while maintaining high stealthiness. Extensive experiments on ModelNet40 and ShapeNetPart further demonstrate that SPBA surpasses prior state-of-the-art backdoor attacks in both attack effectiveness and resistance to defense methods.

http://arxiv.org/abs/2508.05414
Physical Adversarial Camouflage through Gradient Calibration and Regularization. (10%)
Jiawei Liang; Siyuan Liang; Jianjie Huang; Chenxi Si; Ming Zhang; Xiaochun Cao
The advancement of deep object detectors has greatly affected safety-critical fields like autonomous driving. However, physical adversarial camouflage poses a significant security risk by altering object textures to deceive detectors. Existing techniques struggle with variable physical environments, facing two main challenges: 1) inconsistent sampling point densities across distances hinder the gradient optimization from ensuring local continuity, and 2) updating texture gradients from multiple angles causes conflicts, reducing optimization stability and attack effectiveness. To address these issues, we propose a novel adversarial camouflage framework based on gradient optimization. First, we introduce a gradient calibration strategy, which ensures consistent gradient updates across distances by propagating gradients from sparsely to unsampled texture points. Additionally, we develop a gradient decorrelation method, which prioritizes and orthogonalizes gradients based on loss values, enhancing stability and effectiveness in multi-angle optimization by eliminating redundant or conflicting updates. Extensive experimental results on various detection models, angles and distances show that our method significantly exceeds the state of the art, with an average increase in attack success rate (ASR) of 13.46% across distances and 11.03% across angles. Furthermore, empirical evaluation in real-world scenarios highlights the need for more robust system design.

http://arxiv.org/abs/2508.05087
JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering. (9%)
Renmiao Chen; Shiyao Cui; Xuancheng Huang; Chengwei Pan; Victor Shea-Jay Huang; QingLin Zhang; Xuan Ouyang; Zhexin Zhang; Hongning Wang; Minlie Huang
Jailbreak attacks against multimodal large language Models (MLLMs) are a significant research focus. Current research predominantly focuses on maximizing attack success rate (ASR), often overlooking whether the generated responses actually fulfill the attacker's malicious intent. This oversight frequently leads to low-quality outputs that bypass safety filters but lack substantial harmful content. To address this gap, we propose JPS, \underline{J}ailbreak MLLMs with collaborative visual \underline{P}erturbation and textual \underline{S}teering, which achieves jailbreaks via corporation of visual image and textually steering prompt. Specifically, JPS utilizes target-guided adversarial image perturbations for effective safety bypass, complemented by "steering prompt" optimized via a multi-agent system to specifically guide LLM responses fulfilling the attackers' intent. These visual and textual components undergo iterative co-optimization for enhanced performance. To evaluate the quality of attack outcomes, we propose the Malicious Intent Fulfillment Rate (MIFR) metric, assessed using a Reasoning-LLM-based evaluator. Our experiments show JPS sets a new state-of-the-art in both ASR and MIFR across various MLLMs and benchmarks, with analyses confirming its efficacy. Codes are available at \href{https://github.com/thu-coai/JPS}{https://github.com/thu-coai/JPS}. \color{warningcolor}{Warning: This paper contains potentially sensitive contents.}

http://arxiv.org/abs/2508.00387
STF: Shallow-Level Temporal Feedback to Enhance Spiking Transformers. (1%)
Zeqi Zheng; Zizheng Zhu; Yingchao Yu; Yanchen Huang; Changze Lv; Junfeng Tang; Zhaofei Yu; Yaochu Jin
Transformer-based Spiking Neural Networks (SNNs) suffer from a great performance gap compared to floating-point \mbox{Artificial} Neural Networks (ANNs) due to the binary nature of spike trains. Recent efforts have introduced deep-level feedback loops to transmit high-level semantic information to narrow this gap. However, these designs often span \mbox{multiple} deep layers, resulting in costly feature transformations, higher parameter overhead, increased energy consumption, and longer inference latency. To address this issue, we propose Shallow-level Temporal Feedback (STF), a lightweight plug-and-play module for the encoding layer, which consists of Temporal-Spatial Position Embedding (TSPE) and Temporal Feedback (TF). Extensive experiments show that STF consistently improves performance across various Transformer-based SNN backbones on static datasets, including CIFAR-10, CIFAR-100, and ImageNet-1K, under different spike timestep settings. Further analysis reveals that STF enhances the diversity of spike patterns, which is key to performance gain. Moreover, evaluations on adversarial robustness and temporal sensitivity confirm that STF outperforms direct coding and its variants, highlighting its potential as a new spike encoding scheme for static scenarios. Our code will be released upon acceptance.

http://arxiv.org/abs/2508.05544
Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees. (1%)
Guang Yang; Xinyang Liu
Large Language Models (LLMs) have shown remarkable progress in multiple-choice question answering (MCQA), but their inherent unreliability, such as hallucination and overconfidence, limits their application in high-risk domains. To address this, we propose a frequency-based uncertainty quantification method under black-box settings, leveraging conformal prediction (CP) to ensure provable coverage guarantees. Our approach involves multiple independent samplings of the model's output distribution for each input, with the most frequent sample serving as a reference to calculate predictive entropy (PE). Experimental evaluations across six LLMs and four datasets (MedMCQA, MedQA, MMLU, MMLU-Pro) demonstrate that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions, as measured by AUROC. Furthermore, the method effectively controls the empirical miscoverage rate under user-specified risk levels, validating that sampling frequency can serve as a viable substitute for logit-based probabilities in black-box scenarios. This work provides a distribution-free model-agnostic framework for reliable uncertainty quantification in MCQA with guaranteed coverage, enhancing the trustworthiness of LLMs in practical applications.

http://arxiv.org/abs/2508.05689
Boosting Adversarial Transferability via Residual Perturbation Attack. (99%)
Jinjia Peng; Zeze Tao; Huibing Wang; Meng Wang; Yang Wang
Deep neural networks are susceptible to adversarial examples while suffering from incorrect predictions via imperceptible perturbations. Transfer-based attacks create adversarial examples for surrogate models and transfer these examples to target models under black-box scenarios. Recent studies reveal that adversarial examples in flat loss landscapes exhibit superior transferability to alleviate overfitting on surrogate models. However, the prior arts overlook the influence of perturbation directions, resulting in limited transferability. In this paper, we propose a novel attack method, named Residual Perturbation Attack (ResPA), relying on the residual gradient as the perturbation direction to guide the adversarial examples toward the flat regions of the loss function. Specifically, ResPA conducts an exponential moving average on the input gradients to obtain the first moment as the reference gradient, which encompasses the direction of historical gradients. Instead of heavily relying on the local flatness that stems from the current gradients as the perturbation direction, ResPA further considers the residual between the current gradient and the reference gradient to capture the changes in the global perturbation direction. The experimental results demonstrate the better transferability of ResPA than the existing typical transfer-based attack methods, while the transferability can be further improved by combining ResPA with the current input transformation methods. The code is available at https://github.com/ZezeTao/ResPA.

http://arxiv.org/abs/2505.03646
GRILL: Gradient Signal Restoration in Ill-Conditioned Layers to Enhance Adversarial Attacks on Autoencoders. (99%)
Chethan Krishnamurthy Ramanaik; Arjun Roy; Tobias Callies; Eirini Ntoutsi
Adversarial robustness of deep autoencoders (AEs) remains relatively unexplored, even though their non-invertible nature poses distinct challenges. Existing attack algorithms during the optimization of imperceptible, norm-bounded adversarial perturbations to maximize output damage in AEs, often stop at sub-optimal attacks. We observe that the adversarial loss gradient vanishes when backpropagated through ill-conditioned layers. This issue arises from near-zero singular values in the Jacobians of these layers, which weaken the gradient signal during optimization. We introduce GRILL, a technique that locally restores gradient signals in ill-conditioned layers, enabling more effective norm-bounded attacks. Through extensive experiments on different architectures of popular AEs, under both sample-specific and universal attack setups, and across standard and adaptive attack settings, we show that our method significantly increases the effectiveness of our adversarial attacks, enabling a more rigorous evaluation of AE robustness.

http://arxiv.org/abs/2507.21483
NCCR: to Evaluate the Robustness of Neural Networks and Adversarial Examples. (99%)
Shi Pu; Fu Song; Wenjie Wang
Neural networks have received a lot of attention recently, and related security issues have come with it. Many studies have shown that neural networks are vulnerable to adversarial examples that have been artificially perturbed with modification, which is too small to be distinguishable by human perception. Different attacks and defenses have been proposed to solve these problems, but there is little research on evaluating the robustness of neural networks and their inputs. In this work, we propose a metric called the neuron cover change rate (NCCR) to measure the ability of deep learning models to resist attacks and the stability of adversarial examples. NCCR monitors alterations in the output of specifically chosen neurons when the input is perturbed, and networks with a smaller degree of variation are considered to be more robust. The results of the experiment on image recognition and the speaker recognition model show that our metrics can provide a good assessment of the robustness of neural networks or their inputs. It can also be used to detect whether an input is adversarial or not, as adversarial examples are always less robust.

http://arxiv.org/abs/2502.10452
Quaternion-Hadamard Network: A Novel Defense Against Adversarial Attacks with a New Dataset. (99%)
Vladimir Frants; Sos Agaian
This paper addresses the vulnerability of deep-learning models designed for rain, snow, and haze removal. Despite enhancing image quality in adverse weather, these models are susceptible to adversarial attacks that compromise their effectiveness. Traditional defenses such as adversarial training and model distillation often require extensive retraining, making them costly and impractical for real-world deployment. While denoising and super-resolution techniques can aid image classification models, they impose high computational demands and introduce visual artifacts that hinder image processing tasks. We propose a model-agnostic defense against first-order white-box adversarial attacks using the Quaternion-Hadamard Network (QHNet) to tackle these challenges. White-box attacks are particularly difficult to defend against since attackers have full access to the model's architecture, weights, and training procedures. Our defense introduces the Quaternion Hadamard Denoising Convolutional Block (QHDCB) and the Quaternion Denoising Residual Block (QDRB), leveraging polynomial thresholding. QHNet incorporates these blocks within an encoder-decoder architecture, enhanced by feature refinement, to effectively neutralize adversarial noise. Additionally, we introduce the Adversarial Weather Conditions Vision Dataset (AWCVD), created by applying first-order gradient attacks on state-of-the-art weather removal techniques in scenarios involving haze, rain streaks, and snow. Using PSNR and SSIM metrics, we demonstrate that QHNet significantly enhances the robustness of low-level computer vision models against adversarial attacks compared with state-of-the-art denoising and super-resolution techniques. The source code and dataset will be released alongside the final version of this paper.

http://arxiv.org/abs/2508.04894
Adversarial Attacks and Defenses on Graph-aware Large Language Models (LLMs). (99%)
Iyiola E. Olatunji; Franziska Boenisch; Jing Xu; Adam Dziedzic
Large Language Models (LLMs) are increasingly integrated with graph-structured data for tasks like node classification, a domain traditionally dominated by Graph Neural Networks (GNNs). While this integration leverages rich relational information to improve task performance, their robustness against adversarial attacks remains unexplored. We take the first step to explore the vulnerabilities of graph-aware LLMs by leveraging existing adversarial attack methods tailored for graph-based models, including those for poisoning (training-time attacks) and evasion (test-time attacks), on two representative models, LLAGA (Chen et al. 2024) and GRAPHPROMPTER (Liu et al. 2024). Additionally, we discover a new attack surface for LLAGA where an attacker can inject malicious nodes as placeholders into the node sequence template to severely degrade its performance. Our systematic analysis reveals that certain design choices in graph encoding can enhance attack success, with specific findings that: (1) the node sequence template in LLAGA increases its vulnerability; (2) the GNN encoder used in GRAPHPROMPTER demonstrates greater robustness; and (3) both approaches remain susceptible to imperceptible feature perturbation attacks. Finally, we propose an end-to-end defense framework GALGUARD, that combines an LLM-based feature correction module to mitigate feature-level perturbations and adapted GNN defenses to protect against structural attacks.

http://arxiv.org/abs/2508.11667
Assessing Representation Stability for Transformer Models. (98%)
Bryan E. Tuck; Rakesh M. Verma
Adversarial text attacks remain a persistent threat to transformer models, yet existing defenses are typically attack-specific or require costly model retraining. We introduce Representation Stability (RS), a model-agnostic detection framework that identifies adversarial examples by measuring how embedding representations change when important words are masked. RS first ranks words using importance heuristics, then measures embedding sensitivity to masking top-k critical words, and processes the resulting patterns with a BiLSTM detector. Experiments show that adversarially perturbed words exhibit disproportionately high masking sensitivity compared to naturally important words. Across three datasets, three attack types, and two victim models, RS achieves over 88% detection accuracy and demonstrates competitive performance compared to existing state-of-the-art methods, often at lower computational cost. Using Normalized Discounted Cumulative Gain (NDCG) to measure perturbation identification quality, we reveal that gradient-based ranking outperforms attention and random selection approaches, with identification quality correlating with detection performance for word-level attacks. RS also generalizes well to unseen datasets, attacks, and models without retraining, providing a practical solution for adversarial text detection.

http://arxiv.org/abs/2505.16888
CAIN: Hijacking LLM-Humans Conversations via Malicious System Prompts. (86%)
Viet Pham; Thai Le
Large language models (LLMs) have advanced many applications, but are also known to be vulnerable to adversarial attacks. In this work, we introduce a novel security threat: hijacking AI-human conversations by manipulating LLMs' system prompts to produce malicious answers only to specific targeted questions (e.g., "Who should I vote for US President?", "Are Covid vaccines safe?"), while behaving benignly on others. This attack is detrimental as it can enable malicious actors to exercise large-scale information manipulation by spreading harmful but benign-looking system prompts online. To demonstrate such an attack, we develop CAIN, an algorithm that can automatically curate such harmful system prompts for a specific target question in a black-box setting or without the need to access the LLM's parameters. Evaluated on both open-source and commercial LLMs, CAIN demonstrates significant adversarial impact. In untargeted attacks or forcing LLMs to output incorrect answers, CAIN achieves up to 40% F1 degradation on targeted questions while preserving high accuracy on benign inputs. For targeted attacks or forcing LLMs to output specific harmful answers, CAIN achieves over 70% F1 scores on these targeted responses with minimal impact on benign questions. Our results highlight the critical need for enhanced robustness measures to safeguard the integrity and safety of LLMs in real-world applications. All source code will be publicly available.

http://arxiv.org/abs/2507.06043
CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations. (67%)
Xiaohu Li; Yunfeng Ning; Zepeng Bao; Mayi Xu; Jianhao Chen; Tieyun Qian
Security alignment enables the Large Language Model (LLM) to gain the protection against malicious queries, but various jailbreak attack methods reveal the vulnerability of this security mechanism. Previous studies have isolated LLM jailbreak attacks and defenses. We analyze the security protection mechanism of the LLM, and propose a framework that combines attack and defense. Our method is based on the linearly separable property of LLM intermediate layer embedding, as well as the essence of jailbreak attack, which aims to embed harmful problems and transfer them to the safe area. We utilize generative adversarial network (GAN) to learn the security judgment boundary inside the LLM to achieve efficient jailbreak attack and defense. The experimental results indicate that our method achieves an average jailbreak success rate of 88.85\% across three popular LLMs, while the defense success rate on the state-of-the-art jailbreak dataset reaches an average of 84.17\%. This not only validates the effectiveness of our approach but also sheds light on the internal security mechanisms of LLMs, offering new insights for enhancing model security The code and data are available at https://github.com/NLPGM/CAVGAN.

http://arxiv.org/abs/2508.04276
A Few Words Can Distort Graphs: Knowledge Poisoning Attacks on Graph-based Retrieval-Augmented Generation of Large Language Models. (61%)
Jiayi Wen; Tianxin Chen; Zhirun Zheng; Cheng Huang
Graph-based Retrieval-Augmented Generation (GraphRAG) has recently emerged as a promising paradigm for enhancing large language models (LLMs) by converting raw text into structured knowledge graphs, improving both accuracy and explainability. However, GraphRAG relies on LLMs to extract knowledge from raw text during graph construction, and this process can be maliciously manipulated to implant misleading information. Targeting this attack surface, we propose two knowledge poisoning attacks (KPAs) and demonstrate that modifying only a few words in the source text can significantly change the constructed graph, poison the GraphRAG, and severely mislead downstream reasoning. The first attack, named Targeted KPA (TKPA), utilizes graph-theoretic analysis to locate vulnerable nodes in the generated graphs and rewrites the corresponding narratives with LLMs, achieving precise control over specific question-answering (QA) outcomes with a success rate of 93.1\%, while keeping the poisoned text fluent and natural. The second attack, named Universal KPA (UKPA), exploits linguistic cues such as pronouns and dependency relations to disrupt the structural integrity of the generated graph by altering globally influential words. With fewer than 0.05\% of full text modified, the QA accuracy collapses from 95\% to 50\%. Furthermore, experiments show that state-of-the-art defense methods fail to detect these attacks, highlighting that securing GraphRAG pipelines against knowledge poisoning remains largely unexplored.

http://arxiv.org/abs/2502.13053
Evaluating the Robustness of Multimodal Agents Against Active Environmental Injection Attacks. (45%)
Yurun Chen; Xavier Hu; Keting Yin; Juncheng Li; Shengyu Zhang
As researchers continue to optimize AI agents for more effective task execution within operating systems, they often overlook a critical security concern: the ability of these agents to detect "impostors" within their environment. Through an analysis of the agents' operational context, we identify a significant threat-attackers can disguise malicious attacks as environmental elements, injecting active disturbances into the agents' execution processes to manipulate their decision-making. We define this novel threat as the Active Environment Injection Attack (AEIA). Focusing on the interaction mechanisms of the Android OS, we conduct a risk assessment of AEIA and identify two critical security vulnerabilities: (1) Adversarial content injection in multimodal interaction interfaces, where attackers embed adversarial instructions within environmental elements to mislead agent decision-making; and (2) Reasoning gap vulnerabilities in the agent's task execution process, which increase susceptibility to AEIA attacks during reasoning. To evaluate the impact of these vulnerabilities, we propose AEIA-MN, an attack scheme that exploits interaction vulnerabilities in mobile operating systems to assess the robustness of MLLM-based agents. Experimental results show that even advanced MLLMs are highly vulnerable to this attack, achieving a maximum attack success rate of 93% on the AndroidWorld benchmark by combining two vulnerabilities.

http://arxiv.org/abs/2508.04064
FLAT: Latent-Driven Arbitrary-Target Backdoor Attacks in Federated Learning. (10%)
Tuan Nguyen; Khoa D Doan; Kok-Seng Wong
Federated learning (FL) is vulnerable to backdoor attacks, yet most existing methods are limited by fixed-pattern or single-target triggers, making them inflexible and easier to detect. We propose FLAT (FL Arbitrary-Target Attack), a novel backdoor attack that leverages a latent-driven conditional autoencoder to generate diverse, target-specific triggers as needed. By introducing a latent code, FLAT enables the creation of visually adaptive and highly variable triggers, allowing attackers to select arbitrary targets without retraining and to evade conventional detection mechanisms. Our approach unifies attack success, stealth, and diversity within a single framework, introducing a new level of flexibility and sophistication to backdoor attacks in FL. Extensive experiments show that FLAT achieves high attack success and remains robust against advanced FL defenses. These results highlight the urgent need for new defense strategies to address latent-driven, multi-target backdoor threats in federated settings.

http://arxiv.org/abs/2508.05691
AuthPrint: Fingerprinting Generative Models Against Malicious Model Providers. (8%)
Kai Yao; Marc Juarez
Generative models are increasingly adopted in high-stakes domains, yet current deployments offer no mechanisms to verify the origin of model outputs. We address this gap by extending model fingerprinting techniques beyond the traditional collaborative setting to one where the model provider may act adversarially. To our knowledge, this is the first work to evaluate fingerprinting for provenance attribution under such a threat model. The methods rely on a trusted verifier that extracts secret fingerprints from the model's output space, unknown to the provider, and trains a model to predict and verify them. Our empirical evaluation shows that our methods achieve near-zero FPR@95%TPR for instances of GAN and diffusion models, even when tested on small modifications to the original architecture and training data. Moreover, the methods remain robust against adversarial attacks that actively modify the outputs to bypass detection. Source codes are available at https://github.com/PSMLab/authprint.

http://arxiv.org/abs/2508.04576
ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judges. (5%)
Yue Zhou; Yi Chang; Yuan Wu
Reasoning is a critical capability of multimodal large language models (MLLMs) for solving complex multimodal tasks, and judging the correctness of reasoning steps is crucial for improving this capability. Recently, MLLM-based process judges (MPJs) have been widely used to assess the correctness of reasoning steps in multimodal tasks. Therefore, evaluating MPJs is important for identifying their limitations and guiding future improvements. However, existing benchmarks for MPJs mainly focus on tasks such as step correctness classification and reasoning process search, while overlooking a key aspect: whether the confidence scores produced by MPJs at the step level are reliable. To address this gap, we propose ConfProBench, the first comprehensive benchmark designed to systematically evaluate the reliability of step-level confidence scores generated by MPJs. Our benchmark constructs three types of adversarially perturbed reasoning steps: Synonym Substitution, Syntactic Transformation, and Image Perturbation, to test the robustness of MPJ confidence under perturbations. In addition, we introduce three novel evaluation metrics: Confidence Robustness Score (CRS), Confidence Sensitivity Score (CSS), and Confidence Calibration Score (CCS), which evaluate robustness, sensitivity, and calibration, respectively. We evaluate 14 state-of-the-art MLLMs, including both proprietary and open-source models. Experiments reveal limitations in current MPJs' confidence performance and offer competitive baselines to support future research.

http://arxiv.org/abs/2508.04155
Evaluating Selective Encryption Against Gradient Inversion Attacks. (3%)
Jiajun Gu; Yuhang Yao; Shuaiqi Wang; Carlee Joe-Wong
Gradient inversion attacks pose significant privacy threats to distributed training frameworks such as federated learning, enabling malicious parties to reconstruct sensitive local training data from gradient communications between clients and an aggregation server during the aggregation process. While traditional encryption-based defenses, such as homomorphic encryption, offer strong privacy guarantees without compromising model utility, they often incur prohibitive computational overheads. To mitigate this, selective encryption has emerged as a promising approach, encrypting only a subset of gradient data based on the data's significance under a certain metric. However, there have been few systematic studies on how to specify this metric in practice. This paper systematically evaluates selective encryption methods with different significance metrics against state-of-the-art attacks. Our findings demonstrate the feasibility of selective encryption in reducing computational overhead while maintaining resilience against attacks. We propose a distance-based significance analysis framework that provides theoretical foundations for selecting critical gradient elements for encryption. Through extensive experiments on different model architectures (LeNet, CNN, BERT, GPT-2) and attack types, we identify gradient magnitude as a generally effective metric for protection against optimization-based gradient inversions. However, we also observe that no single selective encryption strategy is universally optimal across all attack scenarios, and we provide guidelines for choosing appropriate strategies for different model architectures and privacy requirements.

http://arxiv.org/abs/2409.11026
Prompt Obfuscation for Large Language Models. (3%)
David Pape; Sina Mavali; Thorsten Eisenhofer; Lea Schönherr
System prompts that include detailed instructions to describe the task performed by the underlying LLM can easily transform foundation models into tools and services with minimal overhead. They are often considered intellectual property, similar to the code of a software product, because of their crucial impact on the utility. However, extracting system prompts is easily possible. As of today, there is no effective countermeasure to prevent the stealing of system prompts, and all safeguarding efforts could be evaded. In this work, we propose an alternative to conventional system prompts. We introduce prompt obfuscation to prevent the extraction of the system prompt with little overhead. The core idea is to find a representation of the original system prompt that leads to the same functionality, while the obfuscated system prompt does not contain any information that allows conclusions to be drawn about the original system prompt. We evaluate our approach by comparing our obfuscated prompt output with the output of the original prompt, using eight distinct metrics to measure the lexical, character-level, and semantic similarity. We show that the obfuscated version is constantly on par with the original one. We further perform three different deobfuscation attacks with varying attacker knowledge--covering both black-box and white-box conditions--and show that in realistic attack scenarios an attacker is unable to extract meaningful information. Overall, we demonstrate that prompt obfuscation is an effective mechanism to safeguard the intellectual property of a system prompt while maintaining the same utility as the original prompt.

http://arxiv.org/abs/2508.04094
Isolate Trigger: Detecting and Eradicating Evade-Adaptive Backdoors. (2%)
Chengrui Sun; Hua Zhang; Haoran Gao; Zian Tian; Jianjin Zhao; qi Li; Hongliang Zhu; Zongliang Shen; Shang Wang; Anmin Fu
All current detection of backdoor attacks on deep learning models fall under the category of a non essential features(NEF), which focus on fighting against simple and efficient vertical class backdoor -- trigger is small, few and not overlapping with the source. Evade-adaptive backdoor (EAB) attacks have evaded NEF detection and improved training efficiency. We introduces a precise, efficient and universal detection and defense framework coined as Isolate Trigger (IsTr). IsTr aims to find the hidden trigger by breaking the barrier of the source features. Therefore, it investigates the essence of backdoor triggering, and uses Steps and Differential-Middle-Slice as components to update past theories of distance and gradient. IsTr also plays a positive role in the model, whether the backdoor exists. For example, accurately find and repair the wrong identification caused by deliberate or unintentional training in automatic driving. Extensive experiments on robustness scross various tasks, including MNIST, facial recognition, and traffic sign recognition, confirm the high efficiency, generality and precision of the IsTr. We rigorously evaluated the effectiveness of the IsTr against a series of six EAB attacks, including Badnets, Sin-Wave, Multi-trigger, SSBAs, CASSOCK, HCB. None of these countermeasures evade, even when attacks are combined and the trigger and source overlap.

http://arxiv.org/abs/2508.04818
Single-Step Reconstruction-Free Anomaly Detection and Segmentation via Diffusion Models. (1%)
Mehrdad Moradi; Marco Grasso; Bianca Maria Colosimo; Kamran Paynabar
Generative models have demonstrated significant success in anomaly detection and segmentation over the past decade. Recently, diffusion models have emerged as a powerful alternative, outperforming previous approaches such as GANs and VAEs. In typical diffusion-based anomaly detection, a model is trained on normal data, and during inference, anomalous images are perturbed to a predefined intermediate step in the forward diffusion process. The corresponding normal image is then reconstructed through iterative reverse sampling.
  However, reconstruction-based approaches present three major challenges: (1) the reconstruction process is computationally expensive due to multiple sampling steps, making real-time applications impractical; (2) for complex or subtle patterns, the reconstructed image may correspond to a different normal pattern rather than the original input; and (3) Choosing an appropriate intermediate noise level is challenging because it is application-dependent and often assumes prior knowledge of anomalies, an assumption that does not hold in unsupervised settings.
  We introduce Reconstruction-free Anomaly Detection with Attention-based diffusion models in Real-time (RADAR), which overcomes the limitations of reconstruction-based anomaly detection. Unlike current SOTA methods that reconstruct the input image, RADAR directly produces anomaly maps from the diffusion model, improving both detection accuracy and computational efficiency. We evaluate RADAR on real-world 3D-printed material and the MVTec-AD dataset. Our approach surpasses state-of-the-art diffusion-based and statistical machine learning models across all key metrics, including accuracy, precision, recall, and F1 score. Specifically, RADAR improves F1 score by 7% on MVTec-AD and 13% on the 3D-printed material dataset compared to the next best model.
  Code available at: https://github.com/mehrdadmoradi124/RADAR

http://arxiv.org/abs/2508.04204
ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments. (1%)
Yuquan Wang; Mi Zhang; Yining Wang; Geng Hong; Xiaoyu You; Min Yang
Large Reasoning Models (LRMs) have demonstrated impressive performance in reasoning-intensive tasks, but they remain vulnerable to harmful content generation, particularly in the mid-to-late steps of their reasoning processes. Existing defense mechanisms, however, rely on costly fine-tuning and additional expert knowledge, which restricts their scalability. In this work, we propose ReasoningGuard, an inference-time safeguard for LRMs, which injects timely safety aha moments to steer harmless while helpful reasoning processes. Leveraging the model's internal attention behavior, our approach accurately identifies critical points in the reasoning path, and triggers spontaneous, safety-oriented reflection. To safeguard both the subsequent reasoning steps and the final answers, we further implement a scaling sampling strategy during the decoding phase, selecting the optimal reasoning path. Inducing minimal extra inference cost, ReasoningGuard effectively mitigates three types of jailbreak attacks, including the latest ones targeting the reasoning process of LRMs. Our approach outperforms seven existing safeguards, achieving state-of-the-art safety defenses while effectively avoiding the common exaggerated safety issues.

http://arxiv.org/abs/2508.02997
CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors. (1%)
Sri Durga Sai Sowmya Kadali; Evangelos E. Papalexakis
The widespread use of Large Language Models (LLMs) in many applications marks a significant advance in research and practice. However, their complexity and hard-to-understand nature make them vulnerable to attacks, especially jailbreaks designed to produce harmful responses. To counter these threats, developing strong detection methods is essential for the safe and reliable use of LLMs. This paper studies this detection problem using the Contextual Co-occurrence Matrix, a structure recognized for its efficacy in data-scarce environments. We propose a novel method leveraging the latent space characteristics of Contextual Co-occurrence Matrices and Tensors for the effective identification of adversarial and jailbreak prompts. Our evaluations show that this approach achieves a notable F1 score of 0.83 using only 0.5% of labeled prompts, which is a 96.6% improvement over baselines. This result highlights the strength of our learned patterns, especially when labeled data is scarce. Our method is also significantly faster, speedup ranging from 2.3 to 128.4 times compared to the baseline models. To support future research and reproducibility, we have made our implementation publicly available.

http://arxiv.org/abs/2508.04196
Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models. (1%)
Siddhant Panpatil; Hiskias Dingeto; Haon Park
Despite significant advances in alignment techniques, we demonstrate that state-of-the-art language models remain vulnerable to carefully crafted conversational scenarios that can induce various forms of misalignment without explicit jailbreaking. Through systematic manual red-teaming with Claude-4-Opus, we discovered 10 successful attack scenarios, revealing fundamental vulnerabilities in how current alignment methods handle narrative immersion, emotional pressure, and strategic framing. These scenarios successfully elicited a range of misaligned behaviors, including deception, value drift, self-preservation, and manipulative reasoning, each exploiting different psychological and contextual vulnerabilities. To validate generalizability, we distilled our successful manual attacks into MISALIGNMENTBENCH, an automated evaluation framework that enables reproducible testing across multiple models. Cross-model evaluation of our 10 scenarios against five frontier LLMs revealed an overall 76% vulnerability rate, with significant variations: GPT-4.1 showed the highest susceptibility (90%), while Claude-4-Sonnet demonstrated greater resistance (40%). Our findings demonstrate that sophisticated reasoning capabilities often become attack vectors rather than protective mechanisms, as models can be manipulated into complex justifications for misaligned behavior. This work provides (i) a detailed taxonomy of conversational manipulation patterns and (ii) a reusable evaluation framework. Together, these findings expose critical gaps in current alignment strategies and highlight the need for robustness against subtle, scenario-based manipulation in future AI systems.

http://arxiv.org/abs/2508.03213
The Power of Many: Synergistic Unification of Diverse Augmentations for Efficient Adversarial Robustness. (99%)
Wang Yu-Hang; Shiwei Li; Jianxiang Liao; Li Bohan; Jian Liu; Wenfei Yin
Adversarial perturbations pose a significant threat to deep learning models. Adversarial Training (AT), the predominant defense method, faces challenges of high computational costs and a degradation in standard performance. While data augmentation offers an alternative path, existing techniques either yield limited robustness gains or incur substantial training overhead. Therefore, developing a defense mechanism that is both highly efficient and strongly robust is of paramount importance.In this work, we first conduct a systematic analysis of existing augmentation techniques, revealing that the synergy among diverse strategies -- rather than any single method -- is crucial for enhancing robustness. Based on this insight, we propose the Universal Adversarial Augmenter (UAA) framework, which is characterized by its plug-and-play nature and training efficiency. UAA decouples the expensive perturbation generation process from model training by pre-computing a universal transformation offline, which is then used to efficiently generate unique adversarial perturbations for each sample during training.Extensive experiments conducted on multiple benchmarks validate the effectiveness of UAA. The results demonstrate that UAA establishes a new state-of-the-art (SOTA) for data-augmentation-based adversarial defense strategies , without requiring the online generation of adversarial examples during training. This framework provides a practical and efficient pathway for building robust models,Our code is available in the supplementary materials.

http://arxiv.org/abs/2409.00029
Attack Anything: Blind DNNs via Universal Background Adversarial Attack. (99%)
Jiawei Lian; Shaohui Mei; Xiaofei Wang; Yi Wang; Lefan Wang; Yingjie Lu; Mingyang Ma; Lap-Pui Chau
It has been widely substantiated that deep neural networks (DNNs) are susceptible and vulnerable to adversarial perturbations. Existing studies mainly focus on performing attacks by corrupting targeted objects (physical attack) or images (digital attack), which is intuitively acceptable and understandable in terms of the attack's effectiveness. In contrast, our focus lies in conducting background adversarial attacks in both digital and physical domains, without causing any disruptions to the targeted objects themselves. Specifically, an effective background adversarial attack framework is proposed to attack anything, by which the attack efficacy generalizes well between diverse objects, models, and tasks. Technically, we approach the background adversarial attack as an iterative optimization problem, analogous to the process of DNN learning. Besides, we offer a theoretical demonstration of its convergence under a set of mild but sufficient conditions. To strengthen the attack efficacy and transferability, we propose a new ensemble strategy tailored for adversarial perturbations and introduce an improved smooth constraint for the seamless connection of integrated perturbations. We conduct comprehensive and rigorous experiments in both digital and physical domains across various objects, models, and tasks, demonstrating the effectiveness of attacking anything of the proposed method. The findings of this research substantiate the significant discrepancy between human and machine vision on the value of background variations, which play a far more critical role than previously recognized, necessitating a reevaluation of the robustness and reliability of DNNs. The code will be publicly available at https://github.com/JiaweiLian/Attack_Anything

http://arxiv.org/abs/2508.05677
Adversarial Attacks on Reinforcement Learning-based Medical Questionnaire Systems: Input-level Perturbation Strategies and Medical Constraint Validation. (99%)
Peizhuo Liu
RL-based medical questionnaire systems have shown great potential in medical scenarios. However, their safety and robustness remain unresolved. This study performs a comprehensive evaluation on adversarial attack methods to identify and analyze their potential vulnerabilities. We formulate the diagnosis process as a Markov Decision Process (MDP), where the state is the patient responses and unasked questions, and the action is either to ask a question or to make a diagnosis. We implemented six prevailing major attack methods, including the Fast Gradient Signed Method (FGSM), Projected Gradient Descent (PGD), Carlini & Wagner Attack (C&W) attack, Basic Iterative Method (BIM), DeepFool, and AutoAttack, with seven epsilon values each. To ensure the generated adversarial examples remain clinically plausible, we developed a comprehensive medical validation framework consisting of 247 medical constraints, including physiological bounds, symptom correlations, and conditional medical constraints. We achieved a 97.6% success rate in generating clinically plausible adversarial samples. We performed our experiment on the National Health Interview Survey (NHIS) dataset (https://www.cdc.gov/nchs/nhis/), which consists of 182,630 samples, to predict the participant's 4-year mortality rate. We evaluated our attacks on the AdaptiveFS framework proposed in arXiv:2004.00994. Our results show that adversarial attacks could significantly impact the diagnostic accuracy, with attack success rates ranging from 33.08% (FGSM) to 64.70% (AutoAttack). Our work has demonstrated that even under strict medical constraints on the input, such RL-based medical questionnaire systems still show significant vulnerabilities.

http://arxiv.org/abs/2508.02987
Adversarial Attention Perturbations for Large Object Detection Transformers. (98%)
Zachary Yahn; Selim Furkan Tekin; Fatih Ilhan; Sihao Hu; Tiansheng Huang; Yichang Xu; Margaret Loper; Ling Liu
Adversarial perturbations are useful tools for exposing vulnerabilities in neural networks. Existing adversarial perturbation methods for object detection are either limited to attacking CNN-based detectors or weak against transformer-based detectors. This paper presents an Attention-Focused Offensive Gradient (AFOG) attack against object detection transformers. By design, AFOG is neural-architecture agnostic and effective for attacking both large transformer-based object detectors and conventional CNN-based detectors with a unified adversarial attention framework. This paper makes three original contributions. First, AFOG utilizes a learnable attention mechanism that focuses perturbations on vulnerable image regions in multi-box detection tasks, increasing performance over non-attention baselines by up to 30.6%. Second, AFOG's attack loss is formulated by integrating two types of feature loss through learnable attention updates with iterative injection of adversarial perturbations. Finally, AFOG is an efficient and stealthy adversarial perturbation method. It probes the weak spots of detection transformers by adding strategically generated and visually imperceptible perturbations which can cause well-trained object detection models to fail. Extensive experiments conducted with twelve large detection transformers on COCO demonstrate the efficacy of AFOG. Our empirical results also show that AFOG outperforms existing attacks on transformer-based and CNN-based object detectors by up to 83% with superior speed and imperceptibility. Code is available at https://github.com/zacharyyahn/AFOG.

http://arxiv.org/abs/2411.05189
Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens. (98%)
Usman Anwar; Oswald Johannes Von; Louis Kirsch; David Krueger; Spencer Frei
In this work, we make two contributions towards understanding of in-context learning of linear models by transformers. First, we investigate the adversarial robustness of in-context learning in transformers to hijacking attacks -- a type of adversarial attacks in which the adversary's goal is to manipulate the prompt to force the transformer to generate a specific output. We show that both linear transformers and transformers with GPT-2 architectures are vulnerable to such hijacking attacks. However, adversarial robustness to such attacks can be significantly improved through adversarial training -- done either at the pretraining or finetuning stage -- and can generalize to stronger attack models. Our second main contribution is a comparative analysis of adversarial vulnerabilities across transformer models and other algorithms for learning linear models. This reveals two novel findings. First, adversarial attacks transfer poorly between larger transformer models trained from different seeds despite achieving similar in-distribution performance. This suggests that transformers of the same architecture trained according to the same recipe may implement different in-context learning algorithms for the same task. Second, we observe that attacks do not transfer well between classical learning algorithms for linear models (single-step gradient descent and ordinary least squares) and transformers. This suggests that there could be qualitative differences between the in-context learning algorithms that transformers implement and these traditional algorithms.

http://arxiv.org/abs/2508.03780
Are Inherently Interpretable Models More Robust? A Study In Music Emotion Recognition. (98%)
Katharina Hoedt; Arthur Flexer; Gerhard Widmer
One of the desired key properties of deep learning models is the ability to generalise to unseen samples. When provided with new samples that are (perceptually) similar to one or more training samples, deep learning models are expected to produce correspondingly similar outputs. Models that succeed in predicting similar outputs for similar inputs are often called robust. Deep learning models, on the other hand, have been shown to be highly vulnerable to minor (adversarial) perturbations of the input, which manage to drastically change a model's output and simultaneously expose its reliance on spurious correlations. In this work, we investigate whether inherently interpretable deep models, i.e., deep models that were designed to focus more on meaningful and interpretable features, are more robust to irrelevant perturbations in the data, compared to their black-box counterparts. We test our hypothesis by comparing the robustness of an interpretable and a black-box music emotion recognition (MER) model when challenged with adversarial examples. Furthermore, we include an adversarially trained model, which is optimised to be more robust, in the comparison. Our results indicate that inherently more interpretable models can indeed be more robust than their black-box counterparts, and achieve similar levels of robustness as adversarially trained models, at lower computational cost.

http://arxiv.org/abs/2507.08343
Towards Imperceptible JPEG Image Hiding: Multi-range Representations-driven Adversarial Stego Generation. (82%)
Junxue Yang; Xin Liao; Weixuan Tang; Jianhua Yang; Zheng Qin
Image hiding fully explores the hidden potential of deep learning-based models, aiming to conceal image-level messages within cover images and reveal them from stego images to achieve covert communication. Existing hiding schemes are easily detected by the naked eyes or steganalyzers due to the cover type confined to the spatial domain, single-range feature extraction and attacks, and insufficient loss constraints. To address these issues, we propose a multi-range representations-driven adversarial stego generation framework called MRAG for JPEG image hiding. This design stems from the fact that steganalyzers typically combine local-range and global-range information to better capture hidden traces. Specifically, MRAG integrates the local-range characteristic of the convolution and the global-range modeling of the transformer. Meanwhile, a features angle-norm disentanglement loss is designed to launch multi-range representations-driven feature-level adversarial attacks. It computes the adversarial loss between covers and stegos based on the surrogate steganalyzer's classified features, i.e., the features before the last fully connected layer. Under the dual constraints of features angle and norm, MRAG can delicately encode the concatenation of cover and secret into subtle adversarial perturbations from local and global ranges relevant to steganalysis. Therefore, the resulting stego can achieve visual and steganalysis imperceptibility. Moreover, coarse-grained and fine-grained frequency decomposition operations are devised to transform the input, introducing multi-grained information. Extensive experiments demonstrate that MRAG can achieve state-of-the-art performance.

http://arxiv.org/abs/2508.03307
BDFirewall: Towards Effective and Expeditiously Black-Box Backdoor Defense in MLaaS. (81%)
Ye Li; Chengcheng Zhu; Yanchao Zhao; Jiale Zhang
In this paper, we endeavor to address the challenges of backdoor attacks countermeasures in black-box scenarios, thereby fortifying the security of inference under MLaaS. We first categorize backdoor triggers from a new perspective, i.e., their impact on the patched area, and divide them into: high-visibility triggers (HVT), semi-visibility triggers (SVT), and low-visibility triggers (LVT). Based on this classification, we propose a progressive defense framework, BDFirewall, that removes these triggers from the most conspicuous to the most subtle, without requiring model access. First, for HVTs, which create the most significant local semantic distortions, we identify and eliminate them by detecting these salient differences. We then restore the patched area to mitigate the adverse impact of such removal process. The localized purification designed for HVTs is, however, ineffective against SVTs, which globally perturb benign features. We therefore model an SVT-poisoned input as a mixture of a trigger and benign features, where we unconventionally treat the benign features as "noise". This formulation allows us to reconstruct SVTs by applying a denoising process that removes these benign "noise" features. The SVT-free input is then obtained by subtracting the reconstructed trigger. Finally, to neutralize the nearly imperceptible but fragile LVTs, we introduce lightweight noise to disrupt the trigger pattern and then apply DDPM to restore any collateral impact on clean features. Comprehensive experiments demonstrate that our method outperforms state-of-the-art defenses. Compared with baselines, BDFirewall reduces the Attack Success Rate (ASR) by an average of 33.25%, improving poisoned sample accuracy (PA) by 29.64%, and achieving up to a 111x speedup in inference time. Code will be made publicly available upon acceptance.

http://arxiv.org/abs/2508.03365
When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs. (74%)
Bodam Kim; Hiskias Dingeto; Taeyoun Kwon; Dasol Choi; DongGeon Lee; Haon Park; JaeHoon Lee; Jongho Shin
As large language models become increasingly integrated into daily life, audio has emerged as a key interface for human-AI interaction. However, this convenience also introduces new vulnerabilities, making audio a potential attack surface for adversaries. Our research introduces WhisperInject, a two-stage adversarial audio attack framework that can manipulate state-of-the-art audio language models to generate harmful content. Our method uses imperceptible perturbations in audio inputs that remain benign to human listeners. The first stage uses a novel reward-based optimization method, Reinforcement Learning with Projected Gradient Descent (RL-PGD), to guide the target model to circumvent its own safety protocols and generate harmful native responses. This native harmful response then serves as the target for Stage 2, Payload Injection, where we use Projected Gradient Descent (PGD) to optimize subtle perturbations that are embedded into benign audio carriers, such as weather queries or greeting messages. Validated under the rigorous StrongREJECT, LlamaGuard, as well as Human Evaluation safety evaluation framework, our experiments demonstrate a success rate exceeding 86% across Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Phi-4-Multimodal. Our work demonstrates a new class of practical, audio-native threats, moving beyond theoretical exploits to reveal a feasible and covert method for manipulating AI behavior.

http://arxiv.org/abs/2504.21054
FFCBA: Feature-based Full-target Clean-label Backdoor Attacks. (73%)
Yangxu Yin; Honglong Chen; Yudong Gao; Peng Sun; Liantao Wu; Zhe Li; Weifeng Liu
Backdoor attacks pose a significant threat to deep neural networks, as backdoored models would misclassify poisoned samples with specific triggers into target classes while maintaining normal performance on clean samples. Among these, multi-target backdoor attacks can simultaneously target multiple classes. However, existing multi-target backdoor attacks all follow the dirty-label paradigm, where poisoned samples are mislabeled, and most of them require an extremely high poisoning rate. This makes them easily detectable by manual inspection. In contrast, clean-label attacks are more stealthy, as they avoid modifying the labels of poisoned samples. However, they generally struggle to achieve stable and satisfactory attack performance and often fail to scale effectively to multi-target attacks. To address this issue, we propose the Feature-based Full-target Clean-label Backdoor Attacks (FFCBA) which consists of two paradigms: Feature-Spanning Backdoor Attacks (FSBA) and Feature-Migrating Backdoor Attacks (FMBA). FSBA leverages class-conditional autoencoders to generate noise triggers that align perturbed in-class samples with the original category's features, ensuring the effectiveness, intra-class consistency, inter-class specificity and natural-feature correlation of triggers. While FSBA supports swift and efficient attacks, its cross-model attack capability is relatively weak. FMBA employs a two-stage class-conditional autoencoder training process that alternates between using out-of-class samples and in-class samples. This allows FMBA to generate triggers with strong target-class features, making it highly effective for cross-model attacks. We conduct experiments on multiple datasets and models, the results show that FFCBA achieves outstanding attack performance and maintains desirable robustness against the state-of-the-art backdoor defenses.

http://arxiv.org/abs/2508.03949
Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code. (68%)
Md. Abdul Awal; Mrigank Rochan; Chanchal K. Roy
Transformer-based language models for code have shown remarkable performance in various software analytics tasks, but their adoption is hindered by high computational costs, slow inference speeds, and substantial environmental impact. Model compression techniques such as pruning, quantization, and knowledge distillation have gained traction in addressing these challenges. However, the impact of these strategies on the robustness of compressed language models for code in adversarial scenarios remains poorly understood. Understanding how these compressed models behave under adversarial attacks is essential for their safe and effective deployment in real-world applications. To bridge this knowledge gap, we conduct a comprehensive evaluation of how common compression strategies affect the adversarial robustness of compressed models. We assess the robustness of compressed versions of three widely used language models for code across three software analytics tasks, using six evaluation metrics and four commonly used classical adversarial attacks. Our findings indicate that compressed models generally maintain comparable performance to their uncompressed counterparts. However, when subjected to adversarial attacks, compressed models exhibit significantly reduced robustness. These results reveal a trade-off between model size reduction and adversarial robustness, underscoring the need for careful consideration when deploying compressed models in security-critical software applications. Our study highlights the need for further research into compression strategies that strike a balance between computational efficiency and adversarial robustness, which is essential for deploying reliable language models for code in real-world software applications.

http://arxiv.org/abs/2508.03110
Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation. (67%)
Zizhong Li; Haopeng Zhang; Jiawei Zhang
While large language models (LLMs) have achieved remarkable success in providing trustworthy responses for knowledge-intensive tasks, they still face critical limitations such as hallucinations and outdated knowledge. To address these issues, the retrieval-augmented generation (RAG) framework enhances LLMs with access to external knowledge via a retriever, enabling more accurate and real-time outputs about the latest events. However, this integration brings new security vulnerabilities: the risk that malicious content in the external database can be retrieved and used to manipulate model outputs. Although prior work has explored attacks on RAG systems, existing approaches either rely heavily on access to the retriever or fail to jointly consider both retrieval and generation stages, limiting their effectiveness, particularly in black-box scenarios. To overcome these limitations, we propose Token-level Precise Attack on the RAG (TPARAG), a novel framework that targets both white-box and black-box RAG systems. TPARAG leverages a lightweight white-box LLM as an attacker to generate and iteratively optimize malicious passages at the token level, ensuring both retrievability and high attack success in generation. Extensive experiments on open-domain QA datasets demonstrate that TPARAG consistently outperforms previous approaches in retrieval-stage and end-to-end attack effectiveness. These results further reveal critical vulnerabilities in RAG pipelines and offer new insights into improving their robustness.

http://arxiv.org/abs/2508.03067
Untraceable DeepFakes via Traceable Fingerprint Elimination. (64%)
Jiewei Lai; Lan Zhang; Chen Tang; Pengcheng Sun; Xinming Wang; Yunhao Wang
Recent advancements in DeepFakes attribution technologies have significantly enhanced forensic capabilities, enabling the extraction of traces left by generative models (GMs) in images, making DeepFakes traceable back to their source GMs. Meanwhile, several attacks have attempted to evade attribution models (AMs) for exploring their limitations, calling for more robust AMs. However, existing attacks fail to eliminate GMs' traces, thus can be mitigated by defensive measures. In this paper, we identify that untraceable DeepFakes can be achieved through a multiplicative attack, which can fundamentally eliminate GMs' traces, thereby evading AMs even enhanced with defensive measures. We design a universal and black-box attack method that trains an adversarial model solely using real data, applicable for various GMs and agnostic to AMs. Experimental results demonstrate the outstanding attack capability and universal applicability of our method, achieving an average attack success rate (ASR) of 97.08\% against 6 advanced AMs on DeepFakes generated by 9 GMs. Even in the presence of defensive mechanisms, our method maintains an ASR exceeding 72.39\%. Our work underscores the potential challenges posed by multiplicative attacks and highlights the need for more robust AMs.

http://arxiv.org/abs/2508.03221
BadBlocks: Low-Cost and Stealthy Backdoor Attacks Tailored for Text-to-Image Diffusion Models. (47%)
Yu Pan; Jiahao Chen; Lin Wang; Bingrong Dai; Yi Du
In recent years,Diffusion models have achieved remarkable progress in the field of image generation.However,recent studies have shown that diffusion models are susceptible to backdoor attacks,in which attackers can manipulate the output by injecting covert triggers such as specific visual patterns or textual phrases into the training dataset.Fortunately,with the continuous advancement of defense techniques,defenders have become increasingly capable of identifying and mitigating most backdoor attacks using visual inspection and neural network-based detection methods.However,in this paper,we identify a novel type of backdoor threat that is more lightweight and covert than existing approaches,which we name BadBlocks,requires only about 30\% of the computational resources and 20\% GPU time typically needed by previous backdoor attacks,yet it successfully injects backdoors and evades the most advanced defense frameworks.BadBlocks enables attackers to selectively contaminate specific blocks within the UNet architecture of diffusion models while maintaining normal functionality in the remaining components.Experimental results demonstrate that BadBlocks achieves a high attack success rate (ASR) and low perceptual quality loss (as measured by FID Score),even under extremely constrained computational resources and GPU time.Moreover,BadBlocks is able to bypass existing defense frameworks,especially the attention-based backdoor detection method, highlighting it as a novel and noteworthy threat.Ablation studies further demonstrate that effective backdoor injection does not require fine-tuning the entire network and highlight the pivotal role of certain neural network layers in backdoor mapping.Overall,BadBlocks significantly reduces the barrier to conducting backdoor attacks in all aspects.It enables attackers to inject backdoors into large-scale diffusion models even using consumer-grade GPUs.

http://arxiv.org/abs/2508.03209
GeoShield: Safeguarding Geolocation Privacy from Vision-Language Models via Adversarial Perturbations. (47%)
Xinwei Liu; Xiaojun Jia; Yuan Xun; Simeng Qin; Xiaochun Cao
Vision-Language Models (VLMs) such as GPT-4o now demonstrate a remarkable ability to infer users' locations from public shared images, posing a substantial risk to geoprivacy. Although adversarial perturbations offer a potential defense, current methods are ill-suited for this scenario: they often perform poorly on high-resolution images and low perturbation budgets, and may introduce irrelevant semantic content. To address these limitations, we propose GeoShield, a novel adversarial framework designed for robust geoprivacy protection in real-world scenarios. GeoShield comprises three key modules: a feature disentanglement module that separates geographical and non-geographical information, an exposure element identification module that pinpoints geo-revealing regions within an image, and a scale-adaptive enhancement module that jointly optimizes perturbations at both global and local levels to ensure effectiveness across resolutions. Extensive experiments on challenging benchmarks show that GeoShield consistently surpasses prior methods in black-box settings, achieving strong privacy protection with minimal impact on visual or semantic quality. To our knowledge, this work is the first to explore adversarial perturbations for defending against geolocation inference by advanced VLMs, providing a practical and effective solution to escalating privacy concerns.

http://arxiv.org/abs/2508.03864
Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety. (31%)
Zhenyu Pan; Yiting Zhang; Yutong Zhang; Jianshu Zhang; Haozheng Luo; Yuwei Han; Dennis Wu; Hong-Yu Chen; Philip S. Yu; Manling Li; Han Liu
Multi-agent systems (MAS) built on multimodal large language models exhibit strong collaboration and performance. However, their growing openness and interaction complexity pose serious risks, notably jailbreak and adversarial attacks. Existing defenses typically rely on external guard modules, such as dedicated safety agents, to handle unsafe behaviors. Unfortunately, this paradigm faces two challenges: (1) standalone agents offer limited protection, and (2) their independence leads to single-point failure-if compromised, system-wide safety collapses. Naively increasing the number of guard agents further raises cost and complexity. To address these challenges, we propose Evo-MARL, a novel multi-agent reinforcement learning (MARL) framework that enables all task agents to jointly acquire defensive capabilities. Rather than relying on external safety modules, Evo-MARL trains each agent to simultaneously perform its primary function and resist adversarial threats, ensuring robustness without increasing system overhead or single-node failure. Furthermore, Evo-MARL integrates evolutionary search with parameter-sharing reinforcement learning to co-evolve attackers and defenders. This adversarial training paradigm internalizes safety mechanisms and continually enhances MAS performance under co-evolving threats. Experiments show that Evo-MARL reduces attack success rates by up to 22% while boosting accuracy by up to 5% on reasoning tasks-demonstrating that safety and utility can be jointly improved.

http://arxiv.org/abs/2508.06325
Anti-Tamper Protection for Unauthorized Individual Image Generation. (16%)
Zelin Li; Ruohan Zong; Yifan Liu; Ruichen Yao; Yaokun Liu; Yang Zhang; Dong Wang
With the advancement of personalized image generation technologies, concerns about forgery attacks that infringe on portrait rights and privacy are growing. To address these concerns, protection perturbation algorithms have been developed to disrupt forgery generation. However, the protection algorithms would become ineffective when forgery attackers apply purification techniques to bypass the protection. To address this issue, we present a novel approach, Anti-Tamper Perturbation (ATP). ATP introduces a tamper-proof mechanism within the perturbation. It consists of protection and authorization perturbations, where the protection perturbation defends against forgery attacks, while the authorization perturbation detects purification-based tampering. Both protection and authorization perturbations are applied in the frequency domain under the guidance of a mask, ensuring that the protection perturbation does not disrupt the authorization perturbation. This design also enables the authorization perturbation to be distributed across all image pixels, preserving its sensitivity to purification-based tampering. ATP demonstrates its effectiveness in defending forgery attacks across various attack settings through extensive experiments, providing a robust solution for protecting individuals' portrait rights and privacy. Our code is available at: https://github.com/Seeyn/Anti-Tamper-Perturbation .

http://arxiv.org/abs/2410.19794
DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks for Image Analysis. (10%)
Zohreh Aghababaeyan; Manel Abdellatif; Lionel Briand; Ramesh S
Deep Neural Networks (DNNs) are increasingly deployed across applications. However, ensuring their reliability remains a challenge, and in many situations, alternative models with similar functionality and accuracy are available. Traditional accuracy-based evaluations often fail to capture behavioral differences between models, especially with limited test datasets, making it difficult to select or combine models effectively. Differential testing addresses this by generating test inputs that expose discrepancies in DNN model behavior. However, existing approaches face significant limitations: many rely on model internals or are constrained by available seed inputs. To address these challenges, we propose DiffGAN, a black-box test image generation approach for differential testing of DNN models. DiffGAN leverages a Generative Adversarial Network (GAN) and the Non-dominated Sorting Genetic Algorithm II to generate diverse and valid triggering inputs that reveal behavioral discrepancies between models. DiffGAN employs two custom fitness functions, focusing on diversity and divergence, to guide the exploration of the GAN input space and identify discrepancies between models' outputs. By strategically searching this space, DiffGAN generates inputs with specific features that trigger differences in model behavior. DiffGAN is black-box, making it applicable in more situations. We evaluate DiffGAN on eight DNN model pairs trained on widely used image datasets. Our results show DiffGAN significantly outperforms a SOTA baseline, generating four times more triggering inputs, with greater diversity and validity, within the same budget. Additionally, the generated inputs improve the accuracy of a machine learning-based model selection mechanism, which selects the best-performing model based on input characteristics and can serve as a smart output voting mechanism when using alternative models.

http://arxiv.org/abs/2508.03098
Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation. (3%)
Haoran Wang; Xiongxiao Xu; Baixiang Huang; Kai Shu
Retrieval-Augmented Generation (RAG) enhances the factual accuracy of large language models (LLMs) by conditioning outputs on external knowledge sources. However, when retrieval involves private or sensitive data, RAG systems are susceptible to extraction attacks that can leak confidential information through generated responses. We propose Privacy-Aware Decoding (PAD), a lightweight, inference-time defense that adaptively injects calibrated Gaussian noise into token logits during generation. PAD integrates confidence-based screening to selectively protect high-risk tokens, efficient sensitivity estimation to minimize unnecessary noise, and context-aware noise calibration to balance privacy with generation quality. A \renyi Differential Privacy (RDP) accountant rigorously tracks cumulative privacy loss, enabling explicit per-response $(\varepsilon, δ)$-DP guarantees for sensitive outputs. Unlike prior approaches requiring retraining or corpus-level filtering, PAD is model-agnostic and operates entirely at decoding time with minimal computational overhead. Experiments on three real-world datasets demonstrate that PAD substantially reduces private information leakage while preserving response utility, outperforming existing retrieval- and post-processing-based defenses. Our work takes an important step toward mitigating privacy risks in RAG via decoding strategies, paving the way for universal and scalable privacy solutions in sensitive domains. Our code is available: https://github.com/wang2226/PAD.

http://arxiv.org/abs/2508.01365
ConfGuard: A Simple and Effective Backdoor Detection for Large Language Models. (3%)
Zihan Wang; Rui Zhang; Hongwei Li; Wenshu Fan; Wenbo Jiang; Qingchuan Zhao; Guowen Xu
Backdoor attacks pose a significant threat to Large Language Models (LLMs), where adversaries can embed hidden triggers to manipulate LLM's outputs. Most existing defense methods, primarily designed for classification tasks, are ineffective against the autoregressive nature and vast output space of LLMs, thereby suffering from poor performance and high latency. To address these limitations, we investigate the behavioral discrepancies between benign and backdoored LLMs in output space. We identify a critical phenomenon which we term sequence lock: a backdoored model generates the target sequence with abnormally high and consistent confidence compared to benign generation. Building on this insight, we propose ConfGuard, a lightweight and effective detection method that monitors a sliding window of token confidences to identify sequence lock. Extensive experiments demonstrate ConfGuard achieves a near 100\% true positive rate (TPR) and a negligible false positive rate (FPR) in the vast majority of cases. Crucially, the ConfGuard enables real-time detection almost without additional latency, making it a practical backdoor defense for real-world LLM deployments.

http://arxiv.org/abs/2508.03006
Seeing It Before It Happens: In-Generation NSFW Detection for Diffusion-Based Text-to-Image Models. (1%)
Fan Yang; Yihao Huang; Jiayi Zhu; Ling Shi; Geguang Pu; Jin Song Dong; Kailong Wang
Diffusion-based text-to-image (T2I) models enable high-quality image generation but also pose significant risks of misuse, particularly in producing not-safe-for-work (NSFW) content. While prior detection methods have focused on filtering prompts before generation or moderating images afterward, the in-generation phase of diffusion models remains largely unexplored for NSFW detection. In this paper, we introduce In-Generation Detection (IGD), a simple yet effective approach that leverages the predicted noise during the diffusion process as an internal signal to identify NSFW content. This approach is motivated by preliminary findings suggesting that the predicted noise may capture semantic cues that differentiate NSFW from benign prompts, even when the prompts are adversarially crafted. Experiments conducted on seven NSFW categories show that IGD achieves an average detection accuracy of 91.32% over naive and adversarial NSFW prompts, outperforming seven baseline methods.

http://arxiv.org/abs/2508.00756
LeakyCLIP: Extracting Training Data from CLIP. (1%)
Yunhao Chen; Shujie Wang; Xin Wang; Xingjun Ma
Understanding the memorization and privacy leakage risks in Contrastive Language--Image Pretraining (CLIP) is critical for ensuring the security of multimodal models. Recent studies have demonstrated the feasibility of extracting sensitive training examples from diffusion models, with conditional diffusion models exhibiting a stronger tendency to memorize and leak information. In this work, we investigate data memorization and extraction risks in CLIP through the lens of CLIP inversion, a process that aims to reconstruct training images from text prompts. To this end, we introduce \textbf{LeakyCLIP}, a novel attack framework designed to achieve high-quality, semantically accurate image reconstruction from CLIP embeddings. We identify three key challenges in CLIP inversion: 1) non-robust features, 2) limited visual semantics in text embeddings, and 3) low reconstruction fidelity. To address these challenges, LeakyCLIP employs 1) adversarial fine-tuning to enhance optimization smoothness, 2) linear transformation-based embedding alignment, and 3) Stable Diffusion-based refinement to improve fidelity. Empirical results demonstrate the superiority of LeakyCLIP, achieving over 358% improvement in Structural Similarity Index Measure (SSIM) for ViT-B-16 compared to baseline methods on LAION-2B subset. Furthermore, we uncover a pervasive leakage risk, showing that training data membership can even be successfully inferred from the metrics of low-fidelity reconstructions. Our work introduces a practical method for CLIP inversion while offering novel insights into the nature and scope of privacy risks in multimodal models.

http://arxiv.org/abs/2508.03832
Generating Inputs for Grammar Mining using Dynamic Symbolic Execution. (1%)
Andreas Pointner; Josef Pichler; Herbert Prähofer
A vast number of software systems include components that parse and process structured input. In addition to programming languages, which are analyzed by compilers or interpreters, there are numerous components that process standardized or proprietary data formats of varying complexity. Even if such components were initially developed and tested based on a specification, such as a grammar, numerous modifications and adaptations over the course of software evolution can make it impossible to precisely determine which inputs they actually accept.  
In this situation, grammar mining can be used to reconstruct the specification in the form of a grammar. Established approaches already produce useful results, provided that sufficient input data is available to fully cover the input language. However, achieving this completeness is a major challenge. In practice, only input data recorded during the operation of the software systems is available. If this data is used for grammar mining, the resulting grammar reflects only the actual processed inputs but not the complete grammar of the input language accepted by the software component. As a result, edge cases or previously supported features that no longer appear in the available input data are missing from the generated grammar.  
This work addresses this challenge by introducing a novel approach for the automatic generation of inputs for grammar mining. Although input generators have already been used for fuzz testing, it remains unclear whether they are also suitable for grammar miners. Building on the grammar miner Mimid, this work presents a fully automated approach to input generation. The approach leverages Dynamic Symbolic Execution (DSE) and extends it with two mechanisms to overcome the limitations of DSE regarding structured input parsers. First, the search for new inputs is guided by an iterative expansion that starts with a single-character input and gradually extends it. Second, input generation is structured into a novel three-phase approach, which separates the generation of inputs for parser functions.  
The proposed method was evaluated against a diverse set of eleven benchmark applications from the existing literature. Results demonstrate that the approach achieves precision and recall for extracted grammars close to those derived from state-of-the-art grammar miners such as Mimid. Notably, it successfully uncovers subtle features and edge cases in parsers that are typically missed by such grammar miners. The effectiveness of the method is supported by empirical evidence, showing that it can achieve high performance in various domains without requiring prior input samples.  
This contribution is significant for researchers and practitioners in software engineering, offering an automated, scalable, and precise solution for grammar mining. By eliminating the need for manual input generation, the approach not only reduces workload but also enhances the robustness and comprehensiveness of the extracted grammars. Following this approach, software engineers can reconstruct specification from existing (legacy) parsers.

http://arxiv.org/abs/2508.02186
Failure Cases Are Better Learned But Boundary Says Sorry: Facilitating Smooth Perception Change for Accuracy-Robustness Trade-Off in Adversarial Training. (96%)
Yanyun Wang; Li Liu
Adversarial Training (AT) is one of the most effective methods to train robust Deep Neural Networks (DNNs). However, AT creates an inherent trade-off between clean accuracy and adversarial robustness, which is commonly attributed to the more complicated decision boundary caused by the insufficient learning of hard adversarial samples. In this work, we reveal a counterintuitive fact for the first time: From the perspective of perception consistency, hard adversarial samples that can still attack the robust model after AT are already learned better than those successfully defended. Thus, different from previous views, we argue that it is rather the over-sufficient learning of hard adversarial samples that degrades the decision boundary and contributes to the trade-off problem. Specifically, the excessive pursuit of perception consistency would force the model to view the perturbations as noise and ignore the information within them, which should have been utilized to induce a smoother perception transition towards the decision boundary to support its establishment to an appropriate location. In response, we define a new AT objective named Robust Perception, encouraging the model perception to change smoothly with input perturbations, based on which we propose a novel Robust Perception Adversarial Training (RPAT) method, effectively mitigating the current accuracy-robustness trade-off. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with ResNet-18, PreActResNet-18, and WideResNet-34-10 demonstrate the effectiveness of our method beyond four common baselines and 12 state-of-the-art (SOTA) works. The code is available at https://github.com/FlaAI/RPAT.

http://arxiv.org/abs/2508.02835
Defending Against Knowledge Poisoning Attacks During Retrieval-Augmented Generation. (80%)
Kennedy Edemacu; Vinay M. Shashidhar; Micheal Tuape; Dan Abudu; Beakcheol Jang; Jong Wook Kim
Retrieval-Augmented Generation (RAG) has emerged as a powerful approach to boost the capabilities of large language models (LLMs) by incorporating external, up-to-date knowledge sources. However, this introduces a potential vulnerability to knowledge poisoning attacks, where attackers can compromise the knowledge source to mislead the generation model. One such attack is the PoisonedRAG in which the injected adversarial texts steer the model to generate an attacker-chosen response to a target question. In this work, we propose novel defense methods, FilterRAG and ML-FilterRAG, to mitigate the PoisonedRAG attack. First, we propose a new property to uncover distinct properties to differentiate between adversarial and clean texts in the knowledge data source. Next, we employ this property to filter out adversarial texts from clean ones in the design of our proposed approaches. Evaluation of these methods using benchmark datasets demonstrate their effectiveness, with performances close to those of the original RAG systems.

http://arxiv.org/abs/2508.05671
DINA: A Dual Defense Framework Against Internal Noise and External Attacks in Natural Language Processing. (69%)
Ko-Wei Chuang; Hen-Hsen Huang; Tsai-Yen Li
As large language models (LLMs) and generative AI become increasingly integrated into customer service and moderation applications, adversarial threats emerge from both external manipulations and internal label corruption. In this work, we identify and systematically address these dual adversarial threats by introducing DINA (Dual Defense Against Internal Noise and Adversarial Attacks), a novel unified framework tailored specifically for NLP. Our approach adapts advanced noisy-label learning methods from computer vision and integrates them with adversarial training to simultaneously mitigate internal label sabotage and external adversarial perturbations. Extensive experiments conducted on a real-world dataset from an online gaming service demonstrate that DINA significantly improves model robustness and accuracy compared to baseline models. Our findings not only highlight the critical necessity of dual-threat defenses but also offer practical strategies for safeguarding NLP systems in realistic adversarial scenarios, underscoring broader implications for fair and responsible AI deployment.

http://arxiv.org/abs/2508.15778
Towards Stealthy and Effective Backdoor Attacks on Lane Detection: A Naturalistic Data Poisoning Approach. (67%)
Yifan Liao; Yuxin Cao; Yedi Zhang; Wentao He; Yan Xiao; Xianglong Du; Zhiyong Huang; Jin Song Dong
Deep learning-based lane detection (LD) plays a critical role in autonomous driving and advanced driver assistance systems. However, its vulnerability to backdoor attacks presents a significant security concern. Existing backdoor attack methods on LD often exhibit limited practical utility due to the artificial and conspicuous nature of their triggers. To address this limitation and investigate the impact of more ecologically valid backdoor attacks on LD models, we examine the common data poisoning attack and introduce DBALD, a novel diffusion-based data poisoning framework for generating naturalistic backdoor triggers. DBALD comprises two key components: optimal trigger position finding and stealthy trigger generation. Given the insight that attack performance varies depending on the trigger position, we propose a heatmap-based method to identify the optimal trigger location, with gradient analysis to generate attack-specific heatmaps. A region-based editing diffusion process is then applied to synthesize visually plausible triggers within the most susceptible regions identified previously. Furthermore, to ensure scene integrity and stealthy attacks, we introduce two loss strategies: one for preserving lane structure and another for maintaining the consistency of the driving scene. Consequently, compared to existing attack methods, DBALD achieves both a high attack success rate and superior stealthiness. Extensive experiments on 4 mainstream LD models show that DBALD exceeds state-of-the-art methods, with an average success rate improvement of +10.87% and significantly enhanced stealthiness. The experimental results highlight significant practical challenges in ensuring model robustness against real-world backdoor threats in LD.

http://arxiv.org/abs/2508.01987
Controllable and Stealthy Shilling Attacks via Dispersive Latent Diffusion. (50%)
Shutong Qiao; Wei Yuan; Junliang Yu; Tong Chen; Quoc Viet Hung Nguyen; Hongzhi Yin
Recommender systems (RSs) are now fundamental to various online platforms, but their dependence on user-contributed data leaves them vulnerable to shilling attacks that can manipulate item rankings by injecting fake users. Although widely studied, most existing attack models fail to meet two critical objectives simultaneously: achieving strong adversarial promotion of target items while maintaining realistic behavior to evade detection. As a result, the true severity of shilling threats that manage to reconcile the two objectives remains underappreciated. To expose this overlooked vulnerability, we present DLDA, a diffusion-based attack framework that can generate highly effective yet indistinguishable fake users by enabling fine-grained control over target promotion. Specifically, DLDA operates in a pre-aligned collaborative embedding space, where it employs a conditional latent diffusion process to iteratively synthesize fake user profiles with precise target item control. To evade detection, DLDA introduces a dispersive regularization mechanism that promotes variability and realism in generated behavioral patterns. Extensive experiments on three real-world datasets and five popular RS models demonstrate that, compared to prior attacks, DLDA consistently achieves stronger item promotion while remaining harder to detect. These results highlight that modern RSs are more vulnerable than previously recognized, underscoring the urgent need for more robust defenses.

http://arxiv.org/abs/2508.02116
SUAD: Solid-Channel Ultrasound Injection Attack and Defense to Voice Assistants. (22%)
Chao Liu; Zhezheng Zhu; Hao Chen; Zhe Chen; Kaiwen Guo; Penghao Wang; Jun Luo
As a versatile AI application, voice assistants (VAs) have become increasingly popular, but are vulnerable to security threats. Attackers have proposed various inaudible attacks, but are limited by cost, distance, or LoS. Therefore, we propose \name~Attack, a long-range, cross-barrier, and interference-free inaudible voice attack via solid channels. We begin by thoroughly analyzing the dispersion effect in solid channels, revealing its unique impact on signal propagation. To avoid distortions in voice commands, we design a modular command generation model that parameterizes attack distance, victim audio, and medium dispersion features to adapt to variations in the solid-channel state. Additionally, we propose SUAD Defense, a universal defense that uses ultrasonic perturbation signals to block inaudible voice attacks (IVAs) without impacting normal speech. Since the attack can occur at arbitrary frequencies and times, we propose a training method that randomizes both time and frequency to generate perturbation signals that break ultrasonic commands. Notably, the perturbation signal is modulated to an inaudible frequency without affecting the functionality of voice commands for VAs. Experiments on six smartphones have shown that SUAD Attack achieves activation success rates above 89.8% and SUAD Defense blocks IVAs with success rates exceeding 98%.

http://arxiv.org/abs/2508.02115
Coward: Toward Practical Proactive Federated Backdoor Defense via Collision-based Watermark. (15%)
Wenjie Li; Siying Gu; Yiming Li; Kangjie Chen; Zhili Chen; Tianwei Zhang; Shu-Tao Xia; Dacheng Tao
Backdoor detection is currently the mainstream defense against backdoor attacks in federated learning (FL), where malicious clients upload poisoned updates that compromise the global model and undermine the reliability of FL deployments. Existing backdoor detection techniques fall into two categories, including passive and proactive ones, depending on whether the server proactively modifies the global model. However, both have inherent limitations in practice: passive defenses are vulnerable to common non-i.i.d. data distributions and random participation of FL clients, whereas current proactive defenses suffer inevitable out-of-distribution (OOD) bias because they rely on backdoor co-existence effects. To address these issues, we introduce a new proactive defense, dubbed Coward, inspired by our discovery of multi-backdoor collision effects, in which consecutively planted, distinct backdoors significantly suppress earlier ones. In general, we detect attackers by evaluating whether the server-injected, conflicting global watermark is erased during local training rather than retained. Our method preserves the advantages of proactive defenses in handling data heterogeneity (\ie, non-i.i.d. data) while mitigating the adverse impact of OOD bias through a revised detection mechanism. Extensive experiments on benchmark datasets confirm the effectiveness of Coward and its resilience to potential adaptive attacks. The code for our method would be available at https://github.com/still2009/cowardFL.

http://arxiv.org/abs/2508.02110
Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools. (10%)
Kanghua Mo; Li Hu; Yucheng Long; Zhihao Li
Large language model (LLM) agents have demonstrated remarkable capabilities in complex reasoning and decision-making by leveraging external tools. However, this tool-centric paradigm introduces a previously underexplored attack surface: adversaries can manipulate tool metadata -- such as names, descriptions, and parameter schemas -- to influence agent behavior. We identify this as a new and stealthy threat surface that allows malicious tools to be preferentially selected by LLM agents, without requiring prompt injection or access to model internals. To demonstrate and exploit this vulnerability, we propose the Attractive Metadata Attack (AMA), a black-box in-context learning framework that generates highly attractive but syntactically and semantically valid tool metadata through iterative optimization. Our attack integrates seamlessly into standard tool ecosystems and requires no modification to the agent's execution framework. Extensive experiments across ten realistic, simulated tool-use scenarios and a range of popular LLM agents demonstrate consistently high attack success rates (81\%-95\%) and significant privacy leakage, with negligible impact on primary task execution. Moreover, the attack remains effective even under prompt-level defenses and structured tool-selection protocols such as the Model Context Protocol, revealing systemic vulnerabilities in current agent architectures. These findings reveal that metadata manipulation constitutes a potent and stealthy attack surface, highlighting the need for execution-level security mechanisms that go beyond prompt-level defenses.

http://arxiv.org/abs/2508.02136
FedLAD: A Linear Algebra Based Data Poisoning Defence for Federated Learning. (9%)
Qi Xiong; Hai Dong; Nasrin Sohrabi; Zahir Tari
Sybil attacks pose a significant threat to federated learning, as malicious nodes can collaborate and gain a majority, thereby overwhelming the system. Therefore, it is essential to develop countermeasures that ensure the security of federated learning environments. We present a novel defence method against targeted data poisoning, which is one of the types of Sybil attacks, called Linear Algebra-based Detection (FedLAD). Unlike existing approaches, such as clustering and robust training, which struggle in situations where malicious nodes dominate, FedLAD models the federated learning aggregation process as a linear problem, transforming it into a linear algebra optimisation challenge. This method identifies potential attacks by extracting the independent linear combinations from the original linear combinations, effectively filtering out redundant and malicious elements. Extensive experimental evaluations demonstrate the effectiveness of FedLAD compared to five well-established defence methods: Sherpa, CONTRA, Median, Trimmed Mean, and Krum. Using tasks from both image classification and natural language processing, our experiments confirm that FedLAD is robust and not dependent on specific application settings. The results indicate that FedLAD effectively protects federated learning systems across a broad spectrum of malicious node ratios. Compared to baseline defence methods, FedLAD maintains a low attack success rate for malicious nodes when their ratio ranges from 0.2 to 0.8. Additionally, it preserves high model accuracy when the malicious node ratio is between 0.2 and 0.5. These findings underscore FedLAD's potential to enhance both the reliability and performance of federated learning systems in the face of data poisoning attacks.

http://arxiv.org/abs/2508.02530
Understanding the Risks of Asphalt Art on the Reliability of Surveillance Perception Systems. (9%)
Jin Ma; Abyad Enan; Long Cheng; Mashrur Chowdhury
Artistic crosswalks featuring asphalt art, introduced by different organizations in recent years, aim to enhance the visibility and safety of pedestrians. However, their visual complexity may interfere with surveillance systems that rely on vision-based object detection models. In this study, we investigate the impact of asphalt art on pedestrian detection performance of a pretrained vision-based object detection model. We construct realistic crosswalk scenarios by compositing various street art patterns into a fixed surveillance scene and evaluate the model's performance in detecting pedestrians on asphalt-arted crosswalks under both benign and adversarial conditions. A benign case refers to pedestrian crosswalks painted with existing normal asphalt art, whereas an adversarial case involves digitally crafted or altered asphalt art perpetrated by an attacker. Our results show that while simple, color-based designs have minimal effect, complex artistic patterns, particularly those with high visual salience, can significantly degrade pedestrian detection performance. Furthermore, we demonstrate that adversarially crafted asphalt art can be exploited to deliberately obscure real pedestrians or generate non-existent pedestrian detections. These findings highlight a potential vulnerability in urban vision-based pedestrian surveillance systems and underscore the importance of accounting for environmental visual variations when designing robust pedestrian perception models.

http://arxiv.org/abs/2508.02312
A Survey on Data Security in Large Language Models. (5%)
Kang Chen; Xiuze Zhou; Yuanguo Lin; Jinhe Su; Yuanhui Yu; Li Shen; Fan Lin
Large Language Models (LLMs), now a foundation in advancing natural language processing, power applications such as text generation, machine translation, and conversational systems. Despite their transformative potential, these models inherently rely on massive amounts of training data, often collected from diverse and uncurated sources, which exposes them to serious data security risks. Harmful or malicious data can compromise model behavior, leading to issues such as toxic output, hallucinations, and vulnerabilities to threats such as prompt injection or data poisoning. As LLMs continue to be integrated into critical real-world systems, understanding and addressing these data-centric security risks is imperative to safeguard user trust and system reliability. This survey offers a comprehensive overview of the main data security risks facing LLMs and reviews current defense strategies, including adversarial training, RLHF, and data augmentation. Additionally, we categorize and analyze relevant datasets used for assessing robustness and security across different domains, providing guidance for future research. Finally, we highlight key research directions that focus on secure model updates, explainability-driven defenses, and effective governance frameworks, aiming to promote the safe and responsible development of LLM technology. This work aims to inform researchers, practitioners, and policymakers, driving progress toward data security in LLMs.

http://arxiv.org/abs/2508.02092
FPEdit: Robust LLM Fingerprinting through Localized Knowledge Editing. (2%)
Shida Wang; Chaohu Liu; Yubo Wang; Linli Xu
Large language models represent significant investments in computation, data, and engineering expertise, making them extraordinarily valuable intellectual assets. Nevertheless, these AI assets remain vulnerable to unauthorized redistribution and commercial exploitation through fine-tuning or black-box deployment. Current fingerprinting approaches face a fundamental trade-off: intrinsic methods require full parameter access, while backdoor-based techniques employ statistically anomalous triggers easily detected and filtered by adversaries. To address these limitations, we introduce FPEdit, a novel knowledge-editing framework that injects semantically coherent natural language fingerprints by modifying a sparse subset of model weights. This ensures stealthy and precise ownership encoding without degrading the core functionality. Extensive experiments show that FPEdit achieves $95$-$100\%$ fingerprint retention under both full-parameter fine-tuning and parameter-efficient adaptation, while preserving performance on 24 downstream benchmarks. Moreover, FPEdit remains robust under quantization, pruning, and stochastic decoding, and can embed 10 fingerprint pairs into LLaMA2-7B in under 10 minutes using less than 32 GB of GPU memory, a $70\%$ reduction in resource requirements compared to existing techniques. These advances establish FPEdit as the first fingerprinting approach to simultaneously achieve robustness against adaptation, resistance to detection, and preservation of model utility, providing a minimally invasive solution for reliable provenance verification of large language models in adversarial deployment scenarios.

http://arxiv.org/abs/2508.06534
MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving. (2%)
Aishan Liu; Jiakai Wang; Tianyuan Zhang; Hainan Li; Jiangfan Liu; Siyuan Liang; Yilong Ren; Xianglong Liu; Dacheng Tao
Evaluating and ensuring the adversarial robustness of autonomous driving (AD) systems is a critical and unresolved challenge. This paper introduces MetAdv, a novel adversarial testing platform that enables realistic, dynamic, and interactive evaluation by tightly integrating virtual simulation with physical vehicle feedback. At its core, MetAdv establishes a hybrid virtual-physical sandbox, within which we design a three-layer closed-loop testing environment with dynamic adversarial test evolution. This architecture facilitates end-to-end adversarial evaluation, ranging from high-level unified adversarial generation, through mid-level simulation-based interaction, to low-level execution on physical vehicles. Additionally, MetAdv supports a broad spectrum of AD tasks, algorithmic paradigms (e.g., modular deep learning pipelines, end-to-end learning, vision-language models). It supports flexible 3D vehicle modeling and seamless transitions between simulated and physical environments, with built-in compatibility for commercial platforms such as Apollo and Tesla. A key feature of MetAdv is its human-in-the-loop capability: besides flexible environmental configuration for more customized evaluation, it enables real-time capture of physiological signals and behavioral feedback from drivers, offering new insights into human-machine trust under adversarial conditions. We believe MetAdv can offer a scalable and unified framework for adversarial assessment, paving the way for safer AD.

http://arxiv.org/abs/2508.02063
TRACEALIGN -- Tracing the Drift: Attributing Alignment Failures to Training-Time Belief Sources in LLMs. (1%)
Amitava Das; Vinija Jain; Aman Chadha
Large Language Models (LLMs) fine-tuned to align with human values often exhibit alignment drift, producing unsafe or policy-violating completions when exposed to adversarial prompts, decoding perturbations, or paraphrased jailbreaks. While prior work has behaviorally characterized alignment failure, little is known about the training-time belief sources underlying these failures. We introduce TraceAlign, a unified framework for tracing unsafe completions back to their root causes in the model's training corpus. Central to our approach is the Belief Conflict Index (BCI), which quantifies semantic inconsistency between generated spans and aligned policies, based on retrieved training documents using suffix-array matching. We propose three complementary interventions: (i) TraceShield, an inference-time safety filter that refuses completions with high-BCI spans, (ii) Contrastive Belief Deconfliction Loss, a contrastive fine-tuning objective penalizing high-BCI continuations during DPO, and (iii) Prov-Decode, a provenance-aware decoding strategy that vetoes beam expansions predicted to yield high-BCI spans. Together, these defenses reduce alignment drift by up to 85% on our curated Alignment Drift Benchmark (ADB) while preserving utility on standard tasks, with delta less than 0.2 and improved refusal quality. We further derive a theoretical upper bound on drift likelihood via suffix-array span statistics, linking memorization frequency and length to adversarial reactivation risk. TraceAlign thus provides the first scalable, traceable, and grounded toolkit for understanding and mitigating alignment failures at source. To encourage further exploration and development, we open-source our implementation at: https://anonymous.4open.science/r/tracealign-2DA7

http://arxiv.org/abs/2508.01845
Beyond Vulnerabilities: A Survey of Adversarial Attacks as Both Threats and Defenses in Computer Vision Systems. (99%)
Zhongliang Guo; Yifei Qian; Yanli Li; Weiye Li; Chun Tong Lei; Shuai Zhao; Lei Fang; Ognjen Arandjelović; Chun Pong Lau
Adversarial attacks against computer vision systems have emerged as a critical research area that challenges the fundamental assumptions about neural network robustness and security. This comprehensive survey examines the evolving landscape of adversarial techniques, revealing their dual nature as both sophisticated security threats and valuable defensive tools. We provide a systematic analysis of adversarial attack methodologies across three primary domains: pixel-space attacks, physically realizable attacks, and latent-space attacks. Our investigation traces the technical evolution from early gradient-based methods such as FGSM and PGD to sophisticated optimization techniques incorporating momentum, adaptive step sizes, and advanced transferability mechanisms. We examine how physically realizable attacks have successfully bridged the gap between digital vulnerabilities and real-world threats through adversarial patches, 3D textures, and dynamic optical perturbations. Additionally, we explore the emergence of latent-space attacks that leverage semantic structure in internal representations to create more transferable and meaningful adversarial examples. Beyond traditional offensive applications, we investigate the constructive use of adversarial techniques for vulnerability assessment in biometric authentication systems and protection against malicious generative models. Our analysis reveals critical research gaps, particularly in neural style transfer protection and computational efficiency requirements. This survey contributes a comprehensive taxonomy, evolution analysis, and identification of future research directions, aiming to advance understanding of adversarial vulnerabilities and inform the development of more robust and trustworthy computer vision systems.

http://arxiv.org/abs/2508.01605
Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Models. (98%)
Haoran Dai; Jiawen Wang; Ruo Yang; Manali Sharma; Zhonghao Liao; Yuan Hong; Binghui Wang
Text-to-image diffusion models (T2I DMs) have achieved remarkable success in generating high-quality and diverse images from text prompts, yet recent studies have revealed their vulnerability to backdoor attacks. Existing attack methods suffer from critical limitations: 1) they rely on unnatural adversarial prompts that lack human readability and require massive poisoned data; 2) their effectiveness is typically restricted to specific models, lacking generalizability; and 3) they can be mitigated by recent backdoor defenses.
  To overcome these challenges, we propose a novel backdoor attack framework that achieves three key properties: 1) \emph{Practicality}: Our attack requires only a few stealthy backdoor samples to generate arbitrary attacker-chosen target images, as well as ensuring high-quality image generation in benign scenarios. 2) \emph{Generalizability:} The attack is applicable across multiple T2I DMs without requiring model-specific redesign. 3) \emph{Robustness:} The attack remains effective against existing backdoor defenses and adaptive defenses. Our extensive experimental results on multiple T2I DMs demonstrate that with only 10 carefully crafted backdoored samples, our attack method achieves $>$90\% attack success rate with negligible degradation in benign image generation quality. We also conduct human evaluation to validate our attack effectiveness. Furthermore, recent backdoor detection and mitigation methods, as well as adaptive defense tailored to our attack are not sufficiently effective, highlighting the pressing need for more robust defense mechanisms against the proposed attack.

http://arxiv.org/abs/2508.01554
Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Models. (98%)
Yujia Zheng; Tianhao Li; Haotian Huang; Tianyu Zeng; Jingyu Lu; Chuangxin Chu; Yuekai Huang; Ziyou Jiang; Qian Xiong; Yuyao Ge; Mingyang Li
Prompt-based adversarial attacks have become an effective means to assess the robustness of large language models (LLMs). However, existing approaches often treat prompts as monolithic text, overlooking their structural heterogeneity-different prompt components contribute unequally to adversarial robustness. Prior works like PromptRobust assume prompts are value-neutral, but our analysis reveals that complex, domain-specific prompts with rich structures have components with differing vulnerabilities. To address this gap, we introduce PromptAnatomy, an automated framework that dissects prompts into functional components and generates diverse, interpretable adversarial examples by selectively perturbing each component using our proposed method, ComPerturb. To ensure linguistic plausibility and mitigate distribution shifts, we further incorporate a perplexity (PPL)-based filtering mechanism. As a complementary resource, we annotate four public instruction-tuning datasets using the PromptAnatomy framework, verified through human review. Extensive experiments across these datasets and five advanced LLMs demonstrate that ComPerturb achieves state-of-the-art attack success rates. Ablation studies validate the complementary benefits of prompt dissection and PPL filtering. Our results underscore the importance of prompt structure awareness and controlled perturbation for reliable adversarial robustness evaluation in LLMs. Code and data are available at https://github.com/Yujiaaaaa/PACP.

http://arxiv.org/abs/2508.01932
Proactive Disentangled Modeling of Trigger-Object Pairings for Backdoor Defense. (97%)
Kyle Stein; Andrew A. Mahyari; Guillermo III Francia; Eman El-Sheikh
Deep neural networks (DNNs) and generative AI (GenAI) are increasingly vulnerable to backdoor attacks, where adversaries embed triggers into inputs to cause models to misclassify or misinterpret target labels. Beyond traditional single-trigger scenarios, attackers may inject multiple triggers across various object classes, forming unseen backdoor-object configurations that evade standard detection pipelines. In this paper, we introduce DBOM (Disentangled Backdoor-Object Modeling), a proactive framework that leverages structured disentanglement to identify and neutralize both seen and unseen backdoor threats at the dataset level. Specifically, DBOM factorizes input image representations by modeling triggers and objects as independent primitives in the embedding space through the use of Vision-Language Models (VLMs). By leveraging the frozen, pre-trained encoders of VLMs, our approach decomposes the latent representations into distinct components through a learnable visual prompt repository and prompt prefix tuning, ensuring that the relationships between triggers and objects are explicitly captured. To separate trigger and object representations in the visual prompt repository, we introduce the trigger-object separation and diversity losses that aids in disentangling trigger and object visual features. Next, by aligning image features with feature decomposition and fusion, as well as learned contextual prompt tokens in a shared multimodal space, DBOM enables zero-shot generalization to novel trigger-object pairings that were unseen during training, thereby offering deeper insights into adversarial attack patterns. Experimental results on CIFAR-10 and GTSRB demonstrate that DBOM robustly detects poisoned images prior to downstream training, significantly enhancing the security of DNN training pipelines.

http://arxiv.org/abs/2508.01741
Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models. (81%)
Ruofan Wang; Xin Wang; Yang Yao; Xuan Tong; Xingjun Ma
Fine-tuning open-source Vision-Language Models (VLMs) creates a critical yet underexplored attack surface: vulnerabilities in the base VLM could be retained in fine-tuned variants, rendering them susceptible to transferable jailbreak attacks. To demonstrate this risk, we introduce the Simulated Ensemble Attack (SEA), a novel grey-box jailbreak method in which the adversary has full access to the base VLM but no knowledge of the fine-tuned target's weights or training configuration. To improve jailbreak transferability across fine-tuned VLMs, SEA combines two key techniques: Fine-tuning Trajectory Simulation (FTS) and Targeted Prompt Guidance (TPG). FTS generates transferable adversarial images by simulating the vision encoder's parameter shifts, while TPG is a textual strategy that steers the language decoder toward adversarially optimized outputs. Experiments on the Qwen2-VL family (2B and 7B) demonstrate that SEA achieves high transfer attack success rates exceeding 86.5% and toxicity rates near 49.5% across diverse fine-tuned variants, even those specifically fine-tuned to improve safety behaviors. Notably, while direct PGD-based image jailbreaks rarely transfer across fine-tuned VLMs, SEA reliably exploits inherited vulnerabilities from the base model, significantly enhancing transferability. These findings highlight an urgent need to safeguard fine-tuned proprietary VLMs against transferable vulnerabilities inherited from open-source foundations, motivating the development of holistic defenses across the entire model lifecycle.

http://arxiv.org/abs/2508.01595
BeDKD: Backdoor Defense based on Dynamic Knowledge Distillation and Directional Mapping Modulator. (68%)
Zhengxian Wu; Juan Wen; Wanli Peng; Yinghan Zhou; Changtong dou; Yiming Xue
Although existing backdoor defenses have gained success in mitigating backdoor attacks, they still face substantial challenges. In particular, most of them rely on large amounts of clean data to weaken the backdoor mapping but generally struggle with residual trigger effects, resulting in persistently high attack success rates (ASR). Therefore, in this paper, we propose a novel Backdoor defense method based on Directional mapping module and adversarial Knowledge Distillation (BeDKD), which balances the trade-off between defense effectiveness and model performance using a small amount of clean and poisoned data. We first introduce a directional mapping module to identify poisoned data, which destroys clean mapping while keeping backdoor mapping on a small set of flipped clean data. Then, the adversarial knowledge distillation is designed to reinforce clean mapping and suppress backdoor mapping through a cycle iteration mechanism between trust and punish distillations using clean and identified poisoned data. We conduct experiments to mitigate mainstream attacks on three datasets, and experimental results demonstrate that BeDKD surpasses the state-of-the-art defenses and reduces the ASR by 98% without significantly reducing the CACC. Our code are available in https://github.com/CAU-ISS-Lab/Backdoor-Attack-Defense-LLMs/tree/main/BeDKD.

http://arxiv.org/abs/2508.01676
Benchmarking Adversarial Patch Selection and Location. (56%)
Shai Kimhi; Avi Mendlson; Moshe Kimhi
Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap, the first spatially exhaustive benchmark of patch placement, built by evaluating over 1.5e8 forward passes on ImageNet validation images. PatchMap reveals systematic hot-spots where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation guided placement heuristic that leverages off the shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet50, our method boosts attack success rates by 8 to 13 percentage points compared to random or fixed placements. We publicly release PatchMap and the code implementation. The full PatchMap bench (6.5B predictions, multiple backbones) will be released soon to further accelerate research on location-aware defenses and adaptive attacks.

http://arxiv.org/abs/2508.01887
Complete Evasion, Zero Modification: PDF Attacks on AI Text Detection. (10%)
Aldan Creo
AI-generated text detectors have become essential tools for maintaining content authenticity, yet their robustness against evasion attacks remains questionable. We present PDFuzz, a novel attack that exploits the discrepancy between visual text layout and extraction order in PDF documents. Our method preserves exact textual content while manipulating character positioning to scramble extraction sequences. We evaluate this approach against the ArguGPT detector using a dataset of human and AI-generated text. Our results demonstrate complete evasion: detector performance drops from (93.6 $\pm$ 1.4) % accuracy and 0.938 $\pm$ 0.014 F1 score to random-level performance ((50.4 $\pm$ 3.2) % accuracy, 0.0 F1 score) while maintaining perfect visual fidelity. Our work reveals a vulnerability in current detection systems that is inherent to PDF document structures and underscores the need for implementing sturdy safeguards against such attacks. We make our code publicly available at https://github.com/ACMCMC/PDFuzz.

http://arxiv.org/abs/2508.01768
"Energon": Unveiling Transformers from GPU Power and Thermal Side-Channels. (2%)
Arunava Chaudhuri; Shubhi Shukla; Sarani Bhattacharya; Debdeep Mukhopadhyay
Transformers have become the backbone of many Machine Learning (ML) applications, including language translation, summarization, and computer vision. As these models are increasingly deployed in shared Graphics Processing Unit (GPU) environments via Machine Learning as a Service (MLaaS), concerns around their security grow. In particular, the risk of side-channel attacks that reveal architectural details without physical access remains underexplored, despite the high value of the proprietary models they target. This work to the best of our knowledge is the first to investigate GPU power and thermal fluctuations as side-channels and further exploit them to extract information from pre-trained transformer models. The proposed analysis shows how these side channels can be exploited at user-privilege to reveal critical architectural details such as encoder/decoder layer and attention head for both language and vision transformers. We demonstrate the practical impact by evaluating multiple language and vision pre-trained transformers which are publicly available. Through extensive experimental evaluations, we demonstrate that the attack model achieves a high accuracy of over 89% on average for model family identification and 100% for hyperparameter classification, in both single-process as well as noisy multi-process scenarios. Moreover, by leveraging the extracted architectural information, we demonstrate highly effective black-box transfer adversarial attacks with an average success rate exceeding 93%, underscoring the security risks posed by GPU side-channel leakage in deployed transformer models.

http://arxiv.org/abs/2508.01276
Defending Against Beta Poisoning Attacks in Machine Learning Models. (70%)
Nilufer Gulciftci; M. Emre Gursoy
Poisoning attacks, in which an attacker adversarially manipulates the training dataset of a machine learning (ML) model, pose a significant threat to ML security. Beta Poisoning is a recently proposed poisoning attack that disrupts model accuracy by making the training dataset linearly nonseparable. In this paper, we propose four defense strategies against Beta Poisoning attacks: kNN Proximity-Based Defense (KPB), Neighborhood Class Comparison (NCC), Clustering-Based Defense (CBD), and Mean Distance Threshold (MDT). The defenses are based on our observations regarding the characteristics of poisoning samples generated by Beta Poisoning, e.g., poisoning samples have close proximity to one another, and they are centered near the mean of the target class. Experimental evaluations using MNIST and CIFAR-10 datasets demonstrate that KPB and MDT can achieve perfect accuracy and F1 scores, while CBD and NCC also provide strong defensive capabilities. Furthermore, by analyzing performance across varying parameters, we offer practical insights regarding defenses' behaviors under varying conditions.

http://arxiv.org/abs/2503.13652
Web Artifact Attacks Disrupt Vision Language Models. (78%)
Maan Qraitem; Piotr Teterwak; Kate Saenko; Bryan A. Plummer
Vision-language models (VLMs) (e.g. CLIP, LLaVA) are trained on large-scale, lightly curated web datasets, leading them to learn unintended correlations between semantic concepts and unrelated visual signals. These associations degrade model accuracy by causing predictions to rely on incidental patterns rather than genuine visual understanding. Prior work has weaponized these correlations as an attack vector to manipulate model predictions, such as inserting a deceiving class text onto the image in a "typographic" attack. These attacks succeed due to VLMs' text-heavy bias-a result of captions that echo visible words rather than describing content. However, this attack has focused solely on text that matches the target class exactly, overlooking a broader range of correlations, including non-matching text and graphical symbols, which arise from the abundance of branding content in web-scale data. To address this gap, we introduce "artifact-based" attacks: a novel class of manipulations that mislead models using both non-matching text and graphical elements. Unlike typographic attacks, these artifacts are not predefined, making them simultaneously harder to defend against and more challenging to find. We address this by framing artifact attacks as a search problem and demonstrate their effectiveness across five datasets, with some artifacts reinforcing each other to reach 100% attack success rates. These attacks transfer across models with up to 90% effectiveness, making it possible to attack unseen models. To defend against these attacks, we extend prior work's artifact aware prompting to the graphical setting. We see a moderate reduction of success rates of up to 15% relative to standard prompts, suggesting a promising direction for enhancing model robustness. Code: https://github.com/mqraitem/Web-Artifact-Attacks

http://arxiv.org/abs/2508.00965
VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI. (76%)
Roie Kazoom; Ofir Cohen; Rami Puzis; Asaf Shabtai; Ofer Hadar
We introduce VAULT, a fully automated adversarial RAG pipeline that systematically uncovers and remedies weaknesses in NLI models through three stages: retrieval, adversarial generation, and iterative retraining. First, we perform balanced few-shot retrieval by embedding premises with both semantic (BGE) and lexical (BM25) similarity. Next, we assemble these contexts into LLM prompts to generate adversarial hypotheses, which are then validated by an LLM ensemble for label fidelity. Finally, the validated adversarial examples are injected back into the training set at increasing mixing ratios, progressively fortifying a zero-shot RoBERTa-base model.On standard benchmarks, VAULT elevates RoBERTa-base accuracy from 88.48% to 92.60% on SNLI +4.12%, from 75.04% to 80.95% on ANLI +5.91%, and from 54.67% to 71.99% on MultiNLI +17.32%. It also consistently outperforms prior in-context adversarial methods by up to 2.0% across datasets. By automating high-quality adversarial data curation at scale, VAULT enables rapid, human-independent robustness improvements in NLI inference tasks.

http://arxiv.org/abs/2508.00555
Activation-Guided Local Editing for Jailbreaking Attacks. (45%)
Jiecong Wang; Haoran Li; Hao Peng; Ziqian Zeng; Zihao Wang; Haohua Du; Zhengtao Yu
Jailbreaking is an essential adversarial technique for red-teaming these models to uncover and patch security flaws. However, existing jailbreak methods face significant drawbacks. Token-level jailbreak attacks often produce incoherent or unreadable inputs and exhibit poor transferability, while prompt-level attacks lack scalability and rely heavily on manual effort and human ingenuity. We propose a concise and effective two-stage framework that combines the advantages of these approaches. The first stage performs a scenario-based generation of context and rephrases the original malicious query to obscure its harmful intent. The second stage then utilizes information from the model's hidden states to guide fine-grained edits, effectively steering the model's internal representation of the input from a malicious toward a benign one. Extensive experiments demonstrate that this method achieves state-of-the-art Attack Success Rate, with gains of up to 37.74% over the strongest baseline, and exhibits excellent transferability to black-box models. Our analysis further demonstrates that AGILE maintains substantial effectiveness against prominent defense mechanisms, highlighting the limitations of current safeguards and providing valuable insights for future defense development. Our code is available at https://github.com/yunsaijc/AGILE.

http://arxiv.org/abs/2508.01062
CP-FREEZER: Latency Attacks against Vehicular Cooperative Perception. (26%)
Chenyi Wang; Ruoyu Song; Raymond Muller; Jean-Philippe Monteuuis; Z. Berkay Celik; Jonathan Petit; Ryan Gerdes; Ming Li
Cooperative perception (CP) enhances situational awareness of connected and autonomous vehicles by exchanging and combining messages from multiple agents. While prior work has explored adversarial integrity attacks that degrade perceptual accuracy, little is known about CP's robustness against attacks on timeliness (or availability), a safety-critical requirement for autonomous driving. In this paper, we present CP-FREEZER, the first latency attack that maximizes the computation delay of CP algorithms by injecting adversarial perturbation via V2V messages. Our attack resolves several unique challenges, including the non-differentiability of point cloud preprocessing, asynchronous knowledge of the victim's input due to transmission delays, and uses a novel loss function that effectively maximizes the execution time of the CP pipeline. Extensive experiments show that CP-FREEZER increases end-to-end CP latency by over $90\times$, pushing per-frame processing time beyond 3 seconds with a 100% success rate on our real-world vehicle testbed. Our findings reveal a critical threat to the availability of CP systems, highlighting the urgent need for robust defenses.

http://arxiv.org/abs/2508.01084
Provably Secure Retrieval-Augmented Generation. (9%)
Pengcheng Zhou; Yinglun Feng; Zhongliang Yang
Although Retrieval-Augmented Generation (RAG) systems have been widely applied, the privacy and security risks they face, such as data leakage and data poisoning, have not been systematically addressed yet. Existing defense strategies primarily rely on heuristic filtering or enhancing retriever robustness, which suffer from limited interpretability, lack of formal security guarantees, and vulnerability to adaptive attacks. To address these challenges, this paper proposes the first provably secure framework for RAG systems(SAG). Our framework employs a pre-storage full-encryption scheme to ensure dual protection of both retrieved content and vector embeddings, guaranteeing that only authorized entities can access the data. Through formal security proofs, we rigorously verify the scheme's confidentiality and integrity under a computational security model. Extensive experiments across multiple benchmark datasets demonstrate that our framework effectively resists a range of state-of-the-art attacks. This work establishes a theoretical foundation and practical paradigm for verifiably secure RAG systems, advancing AI-powered services toward formally guaranteed security.

http://arxiv.org/abs/2508.00368
Preliminary Investigation into Uncertainty-Aware Attack Stage Classification. (9%)
Alessandro Gaudenzi; Lorenzo Nodari; Lance Kaplan; Alessandra Russo; Murat Sensoy; Federico Cerutti
Advanced Persistent Threats (APTs) represent a significant challenge in cybersecurity due to their prolonged, multi-stage nature and the sophistication of their operators. Traditional detection systems typically focus on identifying malicious activity in binary terms (benign or malicious) without accounting for the progression of an attack. However, effective response strategies depend on accurate inference of the attack's current stage, as countermeasures must be tailored to whether an adversary is in the early reconnaissance phase or actively conducting exploitation or exfiltration. This work addresses the problem of attack stage inference under uncertainty, with a focus on robustness to out-of-distribution (OOD) inputs. We propose a classification approach based on Evidential Deep Learning (EDL), which models predictive uncertainty by outputting parameters of a Dirichlet distribution over possible stages. This allows the system not only to predict the most likely stage of an attack but also to indicate when it is uncertain or the input lies outside the training distribution. Preliminary experiments in a simulated environment demonstrate that the proposed model can accurately infer the stage of an attack with calibrated confidence while effectively detecting OOD inputs, which may indicate changes in the attackers' tactics. These results support the feasibility of deploying uncertainty-aware models for staged threat detection in dynamic and adversarial environments.

http://arxiv.org/abs/2508.01107
AdVAR-DNN: Adversarial Misclassification Attack on Collaborative DNN Inference. (5%)
Shima Yousefi; Motahare Mounesan; Saptarshi Debroy
In recent years, Deep Neural Networks (DNNs) have become increasingly integral to IoT-based environments, enabling realtime visual computing. However, the limited computational capacity of these devices has motivated the adoption of collaborative DNN inference, where the IoT device offloads part of the inference-related computation to a remote server. Such offloading often requires dynamic DNN partitioning information to be exchanged among the participants over an unsecured network or via relays/hops, leading to novel privacy vulnerabilities. In this paper, we propose AdVAR-DNN, an adversarial variational autoencoder (VAE)-based misclassification attack, leveraging classifiers to detect model information and a VAE to generate untraceable manipulated samples, specifically designed to compromise the collaborative inference process. AdVAR-DNN attack uses the sensitive information exchange vulnerability of collaborative DNN inference and is black-box in nature in terms of having no prior knowledge about the DNN model and how it is partitioned. Our evaluation using the most popular object classification DNNs on the CIFAR-100 dataset demonstrates the effectiveness of AdVAR-DNN in terms of high attack success rate with little to no probability of detection.

http://arxiv.org/abs/2508.00636
FedGuard: A Diverse-Byzantine-Robust Mechanism for Federated Learning with Major Malicious Clients. (4%)
Haocheng Jiang; Hua Shen; Jixin Zhang; Willy Susilo; Mingwu Zhang
Federated learning is a distributed training framework vulnerable to Byzantine attacks, particularly when over 50% of clients are malicious or when datasets are highly non-independent and identically distributed (non-IID). Additionally, most existing defense mechanisms are designed for specific attack types (e.g., gradient similarity-based schemes can only defend against outlier model poisoning), limiting their effectiveness. In response, we propose FedGuard, a novel federated learning mechanism. FedGuard cleverly addresses the aforementioned issues by leveraging the high sensitivity of membership inference to model bias. By requiring clients to include an additional mini-batch of server-specified data in their training, FedGuard can identify and exclude poisoned models, as their confidence in the mini-batch will drop significantly. Our comprehensive evaluation unequivocally shows that, under three highly non-IID datasets, with 90% of clients being Byzantine and seven different types of Byzantine attacks occurring in each round, FedGuard significantly outperforms existing robust federated learning schemes in mitigating various types of Byzantine attacks.

http://arxiv.org/abs/2508.01074
Evading Data Provenance in Deep Neural Networks. (2%)
Hongyu Zhu; Sichu Liang; Wenwen Wang; Zhuomeng Zhang; Fangqi Li; Shi-Lin Wang
Modern over-parameterized deep models are highly data-dependent, with large scale general-purpose and domain-specific datasets serving as the bedrock for rapid advancements. However, many datasets are proprietary or contain sensitive information, making unrestricted model training problematic. In the open world where data thefts cannot be fully prevented, Dataset Ownership Verification (DOV) has emerged as a promising method to protect copyright by detecting unauthorized model training and tracing illicit activities. Due to its diversity and superior stealth, evading DOV is considered extremely challenging. However, this paper identifies that previous studies have relied on oversimplistic evasion attacks for evaluation, leading to a false sense of security. We introduce a unified evasion framework, in which a teacher model first learns from the copyright dataset and then transfers task-relevant yet identifier-independent domain knowledge to a surrogate student using an out-of-distribution (OOD) dataset as the intermediary. Leveraging Vision-Language Models and Large Language Models, we curate the most informative and reliable subsets from the OOD gallery set as the final transfer set, and propose selectively transferring task-oriented knowledge to achieve a better trade-off between generalization and evasion effectiveness. Experiments across diverse datasets covering eleven DOV methods demonstrate our approach simultaneously eliminates all copyright identifiers and significantly outperforms nine state-of-the-art evasion attacks in both generalization and effectiveness, with moderate computational overhead. As a proof of concept, we reveal key vulnerabilities in current DOV methods, highlighting the need for long-term development to enhance practicality.

http://arxiv.org/abs/2412.10537
ExclaveFL: Providing Transparency to Federated Learning using Exclaves. (1%)
Jinnan Guo; Kapil Vaswani; Andrew Paverd; Peter Pietzuch
In federated learning (FL), data providers jointly train a model without disclosing their training data. Despite its inherent privacy benefits, a malicious data provider can simply deviate from the correct training protocol without being detected, potentially compromising the trained model. While current solutions have explored the use of trusted execution environments (TEEs) to combat such attacks, they usually assume side-channel attacks against the TEEs are out of scope. However, such side-channel attacks can undermine the security properties of TEE-based FL frameworks, not by extracting the FL data, but by leaking keys that allow the adversary to impersonate as the TEE whilst deviating arbitrarily from the correct training protocol.
  We describe ExclaveFL, an FL platform that provides end-to-end integrity and transparency, even in the presence of side-channel attacks on TEEs. We propose a new paradigm in which existing TEEs are used as exclaves -- integrity-protected execution environments that do not contain any secrets, making them immune to side-channel attacks. Whereas previous approaches attest the TEE itself and bind this attestation to a key held by the TEE, ExclaveFL attests individual data transformations at runtime. These runtime attestations form an attested dataflow graph, which can be checked to ensure the FL training job satisfies claims, such as deviations from the correct computation. We implement ExclaveFL by extending the popular NVFlare FL framework to use exclaves, and show experimentally that ExclaveFL introduces less than 10% overhead compared to the same FL framework without TEEs, whilst providing stronger security guarantees.

http://arxiv.org/abs/2508.00602
LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks. (1%)
Francesco Panebianco; Stefano Bonfanti; Francesco Trovò; Michele Carminati
The generalization capabilities of Large Language Models (LLMs) have led to their widespread deployment across various applications. However, this increased adoption has introduced several security threats, notably in the forms of jailbreaking and data leakage attacks. Additionally, Retrieval Augmented Generation (RAG), while enhancing context-awareness in LLM responses, has inadvertently introduced vulnerabilities that can result in the leakage of sensitive information. Our contributions are twofold. First, we introduce a methodology to analyze historical interaction data from an LLM system, enabling the generation of usage maps categorized by topics (including adversarial interactions). This approach further provides forensic insights for tracking the evolution of jailbreaking attack patterns. Second, we propose LeakSealer, a model-agnostic framework that combines static analysis for forensic insights with dynamic defenses in a Human-In-The-Loop (HITL) pipeline. This technique identifies topic groups and detects anomalous patterns, allowing for proactive defense mechanisms. We empirically evaluate LeakSealer under two scenarios: (1) jailbreak attempts, employing a public benchmark dataset, and (2) PII leakage, supported by a curated dataset of labeled LLM interactions. In the static setting, LeakSealer achieves the highest precision and recall on the ToxicChat dataset when identifying prompt injection. In the dynamic setting, PII leakage detection achieves an AUPRC of $0.97$, significantly outperforming baselines such as Llama Guard.

http://arxiv.org/abs/2507.23202
Adversarial-Guided Diffusion for Multimodal LLM Attacks. (99%)
Chengwei Xia; Fan Ma; Ruijie Quan; Kun Zhan; Yi Yang
This paper addresses the challenge of generating adversarial image using a diffusion model to deceive multimodal large language models (MLLMs) into generating the targeted responses, while avoiding significant distortion of the clean image. To address the above challenges, we propose an adversarial-guided diffusion (AGD) approach for adversarial attack MLLMs. We introduce adversarial-guided noise to ensure attack efficacy. A key observation in our design is that, unlike most traditional adversarial attacks which embed high-frequency perturbations directly into the clean image, AGD injects target semantics into the noise component of the reverse diffusion. Since the added noise in a diffusion model spans the entire frequency spectrum, the adversarial signal embedded within it also inherits this full-spectrum property. Importantly, during reverse diffusion, the adversarial image is formed as a linear combination of the clean image and the noise. Thus, when applying defenses such as a simple low-pass filtering, which act independently on each component, the adversarial image within the noise component is less likely to be suppressed, as it is not confined to the high-frequency band. This makes AGD inherently robust to variety defenses. Extensive experiments demonstrate that our AGD outperforms state-of-the-art methods in attack performance as well as in model robustness to some defenses.

http://arxiv.org/abs/2507.23577
T-Detect: Tail-Aware Statistical Normalization for Robust Detection of Adversarial Machine-Generated Text. (87%)
Alva West; Luodan Zhang; Liuliu Zhang; Minjun Zhu; Yixuan Weng; Yue Zhang
The proliferation of sophisticated text generation models necessitates the development of robust detection methods capable of identifying machine-generated content, particularly text designed to evade detection through adversarial perturbations. Existing zero-shot detectors often rely on statistical measures that implicitly assume Gaussian distributions, a premise that falters when confronted with the heavy-tailed statistical artifacts characteristic of adversarial or non-native English texts. This paper introduces T-Detect, a novel detection method that fundamentally redesigns the statistical core of curvature-based detectors. Our primary innovation is the replacement of standard Gaussian normalization with a heavy-tailed discrepancy score derived from the Student's t-distribution. This approach is theoretically grounded in the empirical observation that adversarial texts exhibit significant leptokurtosis, rendering traditional statistical assumptions inadequate. T-Detect computes a detection score by normalizing the log-likelihood of a passage against the expected moments of a t-distribution, providing superior resilience to statistical outliers. We validate our approach on the challenging RAID benchmark for adversarial text and the comprehensive HART dataset. Experiments show that T-Detect provides a consistent performance uplift over strong baselines, improving AUROC by up to 3.9\% in targeted domains. When integrated into a two-dimensional detection framework (CT), our method achieves state-of-the-art performance, with an AUROC of 0.926 on the Books domain of RAID. Our contributions are a new, theoretically-justified statistical foundation for text detection, an ablation-validated method that demonstrates superior robustness, and a comprehensive analysis of its performance under adversarial conditions. Ours code are released at https://github.com/ResearAI/t-detect.

http://arxiv.org/abs/2507.23335
Scalable and Precise Patch Robustness Certification for Deep Learning Models with Top-k Predictions. (38%)
Qilin Zhou; Haipeng Wang; Zhengyuan Wei; W. K. Chan
Patch robustness certification is an emerging verification approach for defending against adversarial patch attacks with provable guarantees for deep learning systems. Certified recovery techniques guarantee the prediction of the sole true label of a certified sample. However, existing techniques, if applicable to top-k predictions, commonly conduct pairwise comparisons on those votes between labels, failing to certify the sole true label within the top k prediction labels precisely due to the inflation on the number of votes controlled by the attacker (i.e., attack budget); yet enumerating all combinations of vote allocation suffers from the combinatorial explosion problem. We propose CostCert, a novel, scalable, and precise voting-based certified recovery defender. CostCert verifies the true label of a sample within the top k predictions without pairwise comparisons and combinatorial explosion through a novel design: whether the attack budget on the sample is infeasible to cover the smallest total additional votes on top of the votes uncontrollable by the attacker to exclude the true labels from the top k prediction labels. Experiments show that CostCert significantly outperforms the current state-of-the-art defender PatchGuard, such as retaining up to 57.3% in certified accuracy when the patch size is 96, whereas PatchGuard has already dropped to zero.

http://arxiv.org/abs/2507.23229
Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitation. (11%)
Yufei Chen; Yao Wang; Haibin Zhang; Tao Gu
Retrieval-augmented generation (RAG) systems enhance large language models (LLMs) by integrating external knowledge bases, but this advancement introduces significant privacy risks. Existing privacy attacks on RAG systems can trigger data leakage but often fail to accurately isolate knowledge-base-derived sentences within mixed responses. They also lack robustness when applied across multiple domains. This paper addresses these challenges by presenting a novel black-box attack framework that exploits knowledge asymmetry between RAG and standard LLMs to achieve fine-grained privacy extraction across heterogeneous knowledge landscapes. We propose a chain-of-thought reasoning strategy that creates adaptive prompts to steer RAG systems away from sensitive content. Specifically, we first decompose adversarial queries to maximize information disparity and then apply a semantic relationship scoring to resolve lexical and syntactic ambiguities. We finally train a neural network on these feature scores to precisely identify sentences containing private information. Unlike prior work, our framework generalizes to unseen domains through iterative refinement without pre-defined knowledge. Experimental results show that we achieve over 91% privacy extraction rate in single-domain and 83% in multi-domain scenarios, reducing sensitive sentence exposure by over 65% in case studies. This work bridges the gap between attack and defense in RAG systems, enabling precise extraction of private information while providing a foundation for adaptive mitigation.

http://arxiv.org/abs/2402.01124
TransFR: Transferable Federated Recommendation with Adapter Tuning on Pre-trained Language Models. (1%)
Honglei Zhang; Zhiwei Li; Haoxuan Li; Xin Zhou; Jie Zhang; Yidong Li
Federated recommendations (FRs), facilitating multiple local clients to collectively learn a global model without disclosing user private data, have emerged as a prevalent on-device service. In conventional FRs, a dominant paradigm is to utilize discrete identities to represent clients and items, which are then mapped to domain-specific embeddings to participate in model training. Despite considerable performance, we reveal three inherent limitations that can not be ignored in federated settings, i.e., non-transferability across domains, ineffectiveness in cold-start settings, and potential privacy violations during federated training. To this end, we propose a transferable federated recommendation model, TransFR, which delicately incorporates the general capabilities empowered by pre-trained models and the personalized abilities by fine-tuning local private data. Specifically, it first learns domain-agnostic representations of items by exploiting pre-trained models with public textual corpora. To tailor for FR tasks, we further introduce efficient federated adapter-tuning and test-time adaptation mechanisms, which facilitate personalized local adapters for each client by fitting their private data distributions. We theoretically prove the advantages of incorporating adapter tuning in FRs regarding both effectiveness and privacy. Through extensive experiments, we show that our TransFR model surpasses several state-of-the-art FRs on transferability.

http://arxiv.org/abs/2507.22880
AUV-Fusion: Cross-Modal Adversarial Fusion of User Interactions and Visual Perturbations Against VARS. (99%)
Hai Ling; Tianchi Wang; Xiaohao Liu; Zhulin Tao; Lifang Yang; Xianglin Huang
Modern Visual-Aware Recommender Systems (VARS) exploit the integration of user interaction data and visual features to deliver personalized recommendations with high precision. However, their robustness against adversarial attacks remains largely underexplored, posing significant risks to system reliability and security. Existing attack strategies suffer from notable limitations: shilling attacks are costly and detectable, and visual-only perturbations often fail to align with user preferences. To address these challenges, we propose AUV-Fusion, a cross-modal adversarial attack framework that adopts high-order user preference modeling and cross-modal adversary generation. Specifically, we obtain robust user embeddings through multi-hop user-item interactions and transform them via an MLP into semantically aligned perturbations. These perturbations are injected onto the latent space of a pre-trained VAE within the diffusion model. By synergistically integrating genuine user interaction data with visually plausible perturbations, AUV-Fusion eliminates the need for injecting fake user profiles and effectively mitigates the challenge of insufficient user preference extraction inherent in traditional visual-only attacks. Comprehensive evaluations on diverse VARS architectures and real-world datasets demonstrate that AUV-Fusion significantly enhances the exposure of target (cold-start) items compared to conventional baseline methods. Moreover, AUV-Fusion maintains exceptional stealth under rigorous scrutiny.

http://arxiv.org/abs/2507.22428
Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss. (92%)
Yunrui Yu; Hang Su; Cheng-zhong Xu; Zhizhong Su; Jun Zhu
Gradient-based adversarial attacks using the Cross-Entropy (CE) loss often suffer from overestimation due to relative errors in gradient computation induced by floating-point arithmetic. This paper provides a rigorous theoretical analysis of these errors, conducting the first comprehensive study of floating-point computation errors in gradient-based attacks across four distinct scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. We establish theoretical foundations characterizing the behavior of relative numerical errors under different attack conditions, revealing previously unknown patterns in gradient computation instability, and identify floating-point underflow and rounding as key contributors. Building on this insight, we propose the Theoretical MIFPE (T-MIFPE) loss function, which incorporates an optimal scaling factor $T = t^*$ to minimize the impact of floating-point errors, thereby enhancing the accuracy of gradient computation in adversarial attacks. Extensive experiments on the MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate that T-MIFPE outperforms existing loss functions, including CE, C\&W, DLR, and MIFPE, in terms of attack potency and robustness evaluation accuracy.

http://arxiv.org/abs/2507.22446
RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function. (70%)
Yunrui Yu; Kafeng Wang; Hang Su; Jun Zhu
Despite their widespread success, deep neural networks remain critically vulnerable to adversarial attacks, posing significant risks in safety-sensitive applications. This paper investigates activation functions as a crucial yet underexplored component for enhancing model robustness. We propose a Rademacher Complexity Reduction Activation Function (RCR-AF), a novel activation function designed to improve both generalization and adversarial resilience. RCR-AF uniquely combines the advantages of GELU (including smoothness, gradient stability, and negative information retention) with ReLU's desirable monotonicity, while simultaneously controlling both model sparsity and capacity through built-in clipping mechanisms governed by two hyperparameters, $α$ and $γ$. Our theoretical analysis, grounded in Rademacher complexity, demonstrates that these parameters directly modulate the model's Rademacher complexity, offering a principled approach to enhance robustness. Comprehensive empirical evaluations show that RCR-AF consistently outperforms widely-used alternatives (ReLU, GELU, and Swish) in both clean accuracy under standard training and in adversarial robustness within adversarial training paradigms.

http://arxiv.org/abs/2508.05658
Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguards. (64%)
Song Yan; Hui Wei; Jinlong Fei; Guoliang Yang; Zhengyu Zhao; Zheng Wamg
Various (text) prompt filters and (image) safety checkers have been implemented to mitigate the misuse of Text-to-Image (T2I) models in creating Not-Safe-For-Work (NSFW) content.In order to expose potential security vulnerabilities of such safeguards, multimodal jailbreaks have been studied.However, existing jailbreaks are limited to prompt-specific and image-specific perturbations, which suffer from poor scalability and time-consuming optimization.To address these limitations, we propose Universally Unfiltered and Unseen (U3)-Attack, a multimodal jailbreak attack method against T2I safeguards.Specifically, U3-Attack optimizes an adversarial patch on the image background to universally bypass safety checkers and optimizes a safe paraphrase set from a sensitive word to universally bypass prompt filters while eliminating redundant computations.Extensive experimental results demonstrate the superiority of our U3-Attack on both open-source and commercial T2I models.For example, on the commercial Runway-inpainting model with both prompt filter and safety checker, our U3-Attack achieves $~4\times$ higher success rates than the state-of-the-art multimodal jailbreak attack, MMA-Diffusion.Content Warning: This paper includes examples of NSFW content.

http://arxiv.org/abs/2507.22813
DISTIL: Data-Free Inversion of Suspicious Trojan Inputs via Latent Diffusion. (61%)
Hossein Mirzaei; Zeinab Taghavi; Sepehr Rezaee; Masoud Hadi; Moein Madadi; Mackenzie W. Mathis
Deep neural networks have demonstrated remarkable success across numerous tasks, yet they remain vulnerable to Trojan (backdoor) attacks, raising serious concerns about their safety in real-world mission-critical applications. A common countermeasure is trigger inversion -- reconstructing malicious "shortcut" patterns (triggers) inserted by an adversary during training. Current trigger-inversion methods typically search the full pixel space under specific assumptions but offer no assurances that the estimated trigger is more than an adversarial perturbation that flips the model output. Here, we propose a data-free, zero-shot trigger-inversion strategy that restricts the search space while avoiding strong assumptions on trigger appearance. Specifically, we incorporate a diffusion-based generator guided by the target classifier; through iterative generation, we produce candidate triggers that align with the internal representations the model relies on for malicious behavior. Empirical evaluations, both quantitative and qualitative, show that our approach reconstructs triggers that effectively distinguish clean versus Trojaned models. DISTIL surpasses alternative methods by high margins, achieving up to 7.1% higher accuracy on the BackdoorBench dataset and a 9.4% improvement on trojaned object detection model scanning, offering a promising new direction for reliable backdoor defense without reliance on extensive data or strong prior assumptions about triggers. The code is available at https://github.com/AdaptiveMotorControlLab/DISTIL.

http://arxiv.org/abs/2507.22564
Exploiting Synergistic Cognitive Biases to Bypass Safety in LLMs. (41%)
Xikang Yang; Biyu Zhou; Xuehai Tang; Jizhong Han; Songlin Hu
Large Language Models (LLMs) demonstrate impressive capabilities across a wide range of tasks, yet their safety mechanisms remain susceptible to adversarial attacks that exploit cognitive biases -- systematic deviations from rational judgment. Unlike prior jailbreaking approaches focused on prompt engineering or algorithmic manipulation, this work highlights the overlooked power of multi-bias interactions in undermining LLM safeguards. We propose CognitiveAttack, a novel red-teaming framework that systematically leverages both individual and combined cognitive biases. By integrating supervised fine-tuning and reinforcement learning, CognitiveAttack generates prompts that embed optimized bias combinations, effectively bypassing safety protocols while maintaining high attack success rates. Experimental results reveal significant vulnerabilities across 30 diverse LLMs, particularly in open-source models. CognitiveAttack achieves a substantially higher attack success rate compared to the SOTA black-box method PAP (60.1% vs. 31.6%), exposing critical limitations in current defense mechanisms. These findings highlight multi-bias interactions as a powerful yet underexplored attack vector. This work introduces a novel interdisciplinary perspective by bridging cognitive science and LLM safety, paving the way for more robust and human-aligned AI systems.

http://arxiv.org/abs/2403.16591
Bridging Privacy and Robustness for Trustworthy Machine Learning. (2%)
Xiaojin Zhang; Wei Chen
The widespread adoption of machine learning necessitates robust privacy protection alongside algorithmic resilience. While Local Differential Privacy (LDP) provides foundational guarantees, sophisticated adversaries with prior knowledge demand more nuanced Bayesian privacy notions, such as Maximum Bayesian Privacy (MBP) and Average Bayesian Privacy (ABP), first introduced by \cite{zhang2022no}. Concurrently, machine learning systems require inherent robustness against data perturbations and adversarial manipulations. This paper systematically investigates the intricate theoretical relationships among LDP, MBP, and ABP. Crucially, we bridge these privacy concepts with algorithmic robustness, particularly within the Probably Approximately Correct (PAC) learning framework. Our work demonstrates that privacy-preserving mechanisms inherently confer PAC robustness. We present key theoretical results, including the formalization of the established LDP-MBP relationship, novel bounds between MBP and ABP, and a proof demonstrating PAC robustness from MBP. Furthermore, we establish a novel theoretical relationship quantifying how privacy leakage directly influences an algorithm's input robustness. These results provide a unified theoretical framework for understanding and optimizing the privacy-robustness trade-off, paving the way for the development of more secure, trustworthy, and resilient machine learning systems.

http://arxiv.org/abs/2507.23128
Evaluating and Improving the Robustness of Speech Command Recognition Models to Noise and Distribution Shifts. (1%)
Anaïs Baranger; Lucas Maison
Although prior work in computer vision has shown strong correlations between in-distribution (ID) and out-of-distribution (OOD) accuracies, such relationships remain underexplored in audio-based models. In this study, we investigate how training conditions and input features affect the robustness and generalization abilities of spoken keyword classifiers under OOD conditions. We benchmark several neural architectures across a variety of evaluation sets. To quantify the impact of noise on generalization, we make use of two metrics: Fairness (F), which measures overall accuracy gains compared to a baseline model, and Robustness (R), which assesses the convergence between ID and OOD performance. Our results suggest that noise-aware training improves robustness in some configurations. These findings shed new light on the benefits and limitations of noise-based augmentation for generalization in speech models.

http://arxiv.org/abs/2508.00923
Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models. (1%)
Jiazhen Pan; Bailiang Jian; Paul Hager; Yundi Zhang; Che Liu; Friedrike Jungmann; Hongwei Bran Li; Chenyu You; Junde Wu; Jiayuan Zhu; Fenglin Liu; Yuyuan Liu; Niklas Bubeck; Christian Wachinger; Chen; Chen; Zhenyu Gong; Cheng Ouyang; Georgios Kaissis; Benedikt Wiestler; Daniel Rueckert
Ensuring the safety and reliability of large language models (LLMs) in clinical practice is critical to prevent patient harm and promote trustworthy healthcare applications of AI. However, LLMs are advancing so rapidly that static safety benchmarks often become obsolete upon publication, yielding only an incomplete and sometimes misleading picture of model trustworthiness. We demonstrate that a Dynamic, Automatic, and Systematic (DAS) red-teaming framework that continuously stress-tests LLMs can reveal significant weaknesses of current LLMs across four safety-critical domains: robustness, privacy, bias/fairness, and hallucination. A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses, uncovering vulnerabilities in real time without human intervention. Applying DAS to 15 proprietary and open-source LLMs revealed a stark contrast between static benchmark performance and vulnerability under adversarial pressure. Despite a median MedQA accuracy exceeding 80\%, 94\% of previously correct answers failed our dynamic robustness tests. We observed similarly high failure rates across other domains: privacy leaks were elicited in 86\% of scenarios, cognitive-bias priming altered clinical recommendations in 81\% of fairness tests, and we identified hallucination rates exceeding 66\% in widely used models. Such profound residual risks are incompatible with routine clinical practice. By converting red-teaming from a static checklist into a dynamic stress-test audit, DAS red-teaming offers the surveillance that hospitals/regulators/technology vendors require as LLMs become embedded in patient chatbots, decision-support dashboards, and broader healthcare workflows. Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.

http://arxiv.org/abs/2507.22576
COOkeD: Ensemble-based OOD detection in the era of zero-shot CLIP. (1%)
Galadrielle Humblot-Renaux; Gianni Franchi; Sergio Escalera; Thomas B. Moeslund
Out-of-distribution (OOD) detection is an important building block in trustworthy image recognition systems as unknown classes may arise at test-time. OOD detection methods typically revolve around a single classifier, leading to a split in the research field between the classical supervised setting (e.g. ResNet18 classifier trained on CIFAR100) vs. the zero-shot setting (class names fed as prompts to CLIP). In both cases, an overarching challenge is that the OOD detection performance is implicitly constrained by the classifier's capabilities on in-distribution (ID) data. In this work, we show that given a little open-mindedness from both ends, remarkable OOD detection can be achieved by instead creating a heterogeneous ensemble - COOkeD combines the predictions of a closed-world classifier trained end-to-end on a specific dataset, a zero-shot CLIP classifier, and a linear probe classifier trained on CLIP image features. While bulky at first sight, this approach is modular, post-hoc and leverages the availability of pre-trained VLMs, thus introduces little overhead compared to training a single standard classifier. We evaluate COOkeD on popular CIFAR100 and ImageNet benchmarks, but also consider more challenging, realistic settings ranging from training-time label noise, to test-time covariate shift, to zero-shot shift which has been previously overlooked. Despite its simplicity, COOkeD achieves state-of-the-art performance and greater robustness compared to both classical and CLIP-based OOD detection methods. Code is available at https://github.com/glhr/COOkeD

http://arxiv.org/abs/2507.22304
Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embedding. (1%)
Chetan Pathade
Vision-language models (VLMs) have revolutionized multimodal AI applications but introduce novel security vulnerabilities that remain largely unexplored. We present the first comprehensive study of steganographic prompt injection attacks against VLMs, where malicious instructions are invisibly embedded within images using advanced steganographic techniques. Our approach demonstrates that current VLM architectures can inadvertently extract and execute hidden prompts during normal image processing, leading to covert behavioral manipulation. We develop a multi-domain embedding framework combining spatial, frequency, and neural steganographic methods, achieving an overall attack success rate of 24.3% (plus or minus 3.2%, 95% CI) across leading VLMs including GPT-4V, Claude, and LLaVA, with neural steganography methods reaching up to 31.8%, while maintaining reasonable visual imperceptibility (PSNR greater than 38 dB, SSIM greater than 0.94). Through systematic evaluation on 12 diverse datasets and 8 state-of-the-art models, we reveal moderate but meaningful vulnerabilities in current VLM architectures and propose effective countermeasures. Our findings have significant implications for VLM deployment in security-critical applications and highlight the need for proportionate multimodal AI security frameworks.

http://arxiv.org/abs/2507.21750
Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal. (99%)
Yang Wang; Chenghao Xiao; Yizhi Li; Stuart E. Middleton; Noura Al Moubayed; Chenghua Lin
Pre-trained language models (PLMs) have driven substantial progress in natural language processing but remain vulnerable to adversarial attacks, raising concerns about their robustness in real-world applications. Previous studies have sought to mitigate the impact of adversarial attacks by introducing adversarial perturbations into the training process, either implicitly or explicitly. While both strategies enhance robustness, they often incur high computational costs. In this work, we propose a simple yet effective add-on module that enhances the adversarial robustness of PLMs by removing instance-level principal components, without relying on conventional adversarial defences or perturbing the original training data. Our approach transforms the embedding space to approximate Gaussian properties, thereby reducing its susceptibility to adversarial perturbations while preserving semantic relationships. This transformation aligns embedding distributions in a way that minimises the impact of adversarial noise on decision boundaries, enhancing robustness without requiring adversarial examples or costly training-time augmentation. Evaluations on eight benchmark datasets show that our approach improves adversarial robustness while maintaining comparable before-attack accuracy to baselines, achieving a balanced trade-off between robustness and generalisation.

http://arxiv.org/abs/2507.21992
Teach Me to Trick: Exploring Adversarial Transferability via Knowledge Distillation. (99%)
Siddhartha Pradhan; Shikshya Shiwakoti; Neha Bathuri
We investigate whether knowledge distillation (KD) from multiple heterogeneous teacher models can enhance the generation of transferable adversarial examples. A lightweight student model is trained using two KD strategies: curriculum-based switching and joint optimization, with ResNet50 and DenseNet-161 as teachers. The trained student is then used to generate adversarial examples using FG, FGS, and PGD attacks, which are evaluated against a black-box target model (GoogLeNet). Our results show that student models distilled from multiple teachers achieve attack success rates comparable to ensemble-based baselines, while reducing adversarial example generation time by up to a factor of six. An ablation study further reveals that lower temperature settings and the inclusion of hard-label supervision significantly enhance transferability. These findings suggest that KD can serve not only as a model compression technique but also as a powerful tool for improving the efficiency and effectiveness of black-box adversarial attacks.

http://arxiv.org/abs/2507.21985
ZIUM: Zero-Shot Intent-Aware Adversarial Attack on Unlearned Models. (92%)
Hyun Jun Yook; Ga San Jhun; Jae Hyun Cho; Min Jeon; Donghyun Kim; Tae Hyung Kim; Youn Kyu Lee
Machine unlearning (MU) removes specific data points or concepts from deep learning models to enhance privacy and prevent sensitive content generation. Adversarial prompts can exploit unlearned models to generate content containing removed concepts, posing a significant security risk. However, existing adversarial attack methods still face challenges in generating content that aligns with an attacker's intent while incurring high computational costs to identify successful prompts. To address these challenges, we propose ZIUM, a Zero-shot Intent-aware adversarial attack on Unlearned Models, which enables the flexible customization of target attack images to reflect an attacker's intent. Additionally, ZIUM supports zero-shot adversarial attacks without requiring further optimization for previously attacked unlearned concepts. The evaluation across various MU scenarios demonstrated ZIUM's effectiveness in successfully customizing content based on user-intent prompts while achieving a superior attack success rate compared to existing methods. Moreover, its zero-shot adversarial attack significantly reduces the attack time for previously attacked unlearned concepts.

http://arxiv.org/abs/2507.21820
Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is. (86%)
Ahmed B Mustafa; Zihan Ye; Yang Lu; Michael P Pound; Shreyank N Gowda
Despite significant advancements in alignment and content moderation, large language models (LLMs) and text-to-image (T2I) systems remain vulnerable to prompt-based attacks known as jailbreaks. Unlike traditional adversarial examples requiring expert knowledge, many of today's jailbreaks are low-effort, high-impact crafted by everyday users with nothing more than cleverly worded prompts. This paper presents a systems-style investigation into how non-experts reliably circumvent safety mechanisms through techniques such as multi-turn narrative escalation, lexical camouflage, implication chaining, fictional impersonation, and subtle semantic edits. We propose a unified taxonomy of prompt-level jailbreak strategies spanning both text-output and T2I models, grounded in empirical case studies across popular APIs. Our analysis reveals that every stage of the moderation pipeline, from input filtering to output validation, can be bypassed with accessible strategies. We conclude by highlighting the urgent need for context-aware defenses that reflect the ease with which these jailbreaks can be reproduced in real-world settings.

http://arxiv.org/abs/2507.21540
PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking. (64%)
Quanchen Zou; Zonghao Ying; Moyang Chen; Wenzhuo Xu; Yisong Xiao; Yakai Li; Deyue Zhang; Dongdong Yang; Zhao Liu; Xiangzheng Zhang
The increasing sophistication of large vision-language models (LVLMs) has been accompanied by advances in safety alignment mechanisms designed to prevent harmful content generation. However, these defenses remain vulnerable to sophisticated adversarial attacks. Existing jailbreak methods typically rely on direct and semantically explicit prompts, overlooking subtle vulnerabilities in how LVLMs compose information over multiple reasoning steps. In this paper, we propose a novel and effective jailbreak framework inspired by Return-Oriented Programming (ROP) techniques from software security. Our approach decomposes a harmful instruction into a sequence of individually benign visual gadgets. A carefully engineered textual prompt directs the sequence of inputs, prompting the model to integrate the benign visual gadgets through its reasoning process to produce a coherent and harmful output. This makes the malicious intent emergent and difficult to detect from any single component. We validate our method through extensive experiments on established benchmarks including SafeBench and MM-SafetyBench, targeting popular LVLMs. Results show that our approach consistently and substantially outperforms existing baselines on state-of-the-art models, achieving near-perfect attack success rates (over 0.90 on SafeBench) and improving ASR by up to 0.39. Our findings reveal a critical and underexplored vulnerability that exploits the compositional reasoning abilities of LVLMs, highlighting the urgent need for defenses that secure the entire reasoning process.

http://arxiv.org/abs/2407.15549
Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs. (56%)
Abhay Sheshadri; Aidan Ewart; Phillip Guo; Aengus Lynch; Cindy Wu; Vivek Hebbar; Henry Sleight; Asa Cooper Stickland; Ethan Perez; Dylan Hadfield-Menell; Stephen Casper
Large language models (LLMs) can often be made to behave in undesirable ways that they are explicitly fine-tuned not to. For example, the LLM red-teaming literature has produced a wide variety of 'jailbreaking' techniques to elicit harmful text from models that were fine-tuned to be harmless. Recent work on red-teaming, model editing, and interpretability suggests that this challenge stems from how (adversarial) fine-tuning largely serves to suppress rather than remove undesirable capabilities from LLMs. Prior work has introduced latent adversarial training (LAT) as a way to improve robustness to broad classes of failures. These prior works have considered untargeted latent space attacks where the adversary perturbs latent activations to maximize loss on examples of desirable behavior. Untargeted LAT can provide a generic type of robustness but does not leverage information about specific failure modes. Here, we experiment with targeted LAT where the adversary seeks to minimize loss on a specific competing task. We find that it can augment a wide variety of state-of-the-art methods. First, we use targeted LAT to improve robustness to jailbreaks, outperforming a strong R2D2 baseline with orders of magnitude less compute. Second, we use it to more effectively remove backdoors with no knowledge of the trigger. Finally, we use it to more effectively unlearn knowledge for specific undesirable tasks in a way that is also more robust to re-learning. Overall, our results suggest that targeted LAT can be an effective tool for defending against harmful behaviors from LLMs.

http://arxiv.org/abs/2507.22160
Strategic Deflection: Defending LLMs from Logit Manipulation. (26%)
Yassine Rachidy; Jihad Rbaiti; Youssef Hmamouche; Faissal Sehbaoui; Amal El Fallah Seghrouchni
With the growing adoption of Large Language Models (LLMs) in critical areas, ensuring their security against jailbreaking attacks is paramount. While traditional defenses primarily rely on refusing malicious prompts, recent logit-level attacks have demonstrated the ability to bypass these safeguards by directly manipulating the token-selection process during generation. We introduce Strategic Deflection (SDeflection), a defense that redefines the LLM's response to such advanced attacks. Instead of outright refusal, the model produces an answer that is semantically adjacent to the user's request yet strips away the harmful intent, thereby neutralizing the attacker's harmful intent. Our experiments demonstrate that SDeflection significantly lowers Attack Success Rate (ASR) while maintaining model performance on benign queries. This work presents a critical shift in defensive strategies, moving from simple refusal to strategic content redirection to neutralize advanced threats.

http://arxiv.org/abs/2507.22020
XAI for Point Cloud Data using Perturbations based on Meaningful Segmentation. (8%)
Raju Ningappa Mulawade; Christoph Garth; Alexander Wiebel
We propose a novel segmentation-based explainable artificial intelligence (XAI) method for neural networks working on point cloud classification. As one building block of this method, we propose a novel point-shifting mechanism to introduce perturbations in point cloud data. Recently, AI has seen an exponential growth. Hence, it is important to understand the decision-making process of AI algorithms when they are applied in critical areas. Our work focuses on explaining AI algorithms that classify point cloud data. An important aspect of the methods used for explaining AI algorithms is their ability to produce explanations that are easy for humans to understand. This allows them to analyze the AI algorithms better and make appropriate decisions based on that analysis. Therefore, in this work, we intend to generate meaningful explanations that can be easily interpreted by humans. The point cloud data we consider represents 3D objects such as cars, guitars, and laptops. We make use of point cloud segmentation models to generate explanations for the working of classification models. The segments are used to introduce perturbations into the input point cloud data and generate saliency maps. The perturbations are introduced using the novel point-shifting mechanism proposed in this work which ensures that the shifted points no longer influence the output of the classification algorithm. In contrast to previous methods, the segments used by our method are meaningful, i.e. humans can easily interpret the meaning of the segments. Thus, the benefit of our method over other methods is its ability to produce more meaningful saliency maps. We compare our method with the use of classical clustering algorithms to generate explanations. We also analyze the saliency maps generated for example inputs using our method to demonstrate the usefulness of the method in generating meaningful explanations.

http://arxiv.org/abs/2507.22037
Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security. (3%)
Muzhi Dai; Shixuan Liu; Zhiyuan Zhao; Junyu Gao; Hao Sun; Xuelong Li
The rapid advancement of multimodal large language models (MLLMs) has led to breakthroughs in various applications, yet their security remains a critical challenge. One pressing issue involves unsafe image-query pairs--jailbreak inputs specifically designed to bypass security constraints and elicit unintended responses from MLLMs. Compared to general multimodal data, such unsafe inputs are relatively sparse, which limits the diversity and richness of training samples available for developing robust defense models. Meanwhile, existing guardrail-type methods rely on external modules to enforce security constraints but fail to address intrinsic vulnerabilities within MLLMs. Traditional supervised fine-tuning (SFT), on the other hand, often over-refuses harmless inputs, compromising general performance. Given these challenges, we propose Secure Tug-of-War (SecTOW), an innovative iterative defense-attack training method to enhance the security of MLLMs. SecTOW consists of two modules: a defender and an auxiliary attacker, both trained iteratively using reinforcement learning (GRPO). During the iterative process, the attacker identifies security vulnerabilities in the defense model and expands jailbreak data. The expanded data are then used to train the defender, enabling it to address identified security vulnerabilities. We also design reward mechanisms used for GRPO to simplify the use of response labels, reducing dependence on complex generative labels and enabling the efficient use of synthetic data. Additionally, a quality monitoring mechanism is used to mitigate the defender's over-refusal of harmless inputs and ensure the diversity of the jailbreak data generated by the attacker. Experimental results on safety-specific and general benchmarks demonstrate that SecTOW significantly improves security while preserving general performance.

http://arxiv.org/abs/2507.21387
Radio Adversarial Attacks on EMG-based Gesture Recognition Networks. (99%)
Hongyi Xie
Surface electromyography (EMG) enables non-invasive human-computer interaction in rehabilitation, prosthetics, and virtual reality. While deep learning models achieve over 97% classification accuracy, their vulnerability to adversarial attacks remains largely unexplored in the physical domain. We present ERa Attack, the first radio frequency (RF) adversarial method targeting EMG devices through intentional electromagnetic interference (IEMI). Using low-power software-defined radio transmitters, attackers inject optimized RF perturbations to mislead downstream models. Our approach bridges digital and physical domains: we generate adversarial perturbations using Projected Gradient Descent, extract 50-150 Hz components via inverse STFT, and employ synchronization-free strategies (constant spectrum noise or narrowband modulation). Perturbations, constrained to 1-10% of signal amplitude, are amplitude-modulated onto 433 MHz carriers. Experiments on the Myo Dataset (7 gestures, 350 samples) demonstrate significant impact: at 1 meter and 0 dBm transmission power, classification accuracy drops from 97.8% to 58.3%, with 41.7% misclassification rate and 25.6% targeted attack success rate. Attack effectiveness decreases exponentially with distance, recovering to 85% accuracy at 3 meters. Increasing power to 10 dBm reduces accuracy by an additional 15% at 1 meter. This work pioneers RF-based adversarial attacks on EMG recognition systems, revealing critical vulnerabilities in safety-critical applications. We quantify attack effectiveness across different perturbation modes and distances, and propose defenses including hardware shielding, spectrum monitoring, and adversarial training. Our findings inform the design of robust EMG systems against electromagnetic threats.

http://arxiv.org/abs/2507.20996
Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation. (99%)
Hayat Ullah; Syed Muhammad Talha Zaidi; Arslan Munir
Convolutional neural networks (CNNs) excel in computer vision but are susceptible to adversarial attacks, crafted perturbations designed to mislead predictions. Despite advances in adversarial training, a gap persists between model accuracy and robustness. To mitigate this issue, in this paper, we present a multi-teacher adversarial robustness distillation using an adaptive learning strategy. Specifically, our proposed method first trained multiple clones of a baseline CNN model using an adversarial training strategy on a pool of perturbed data acquired through different adversarial attacks. Once trained, these adversarially trained models are used as teacher models to supervise the learning of a student model on clean data using multi-teacher knowledge distillation. To ensure an effective robustness distillation, we design an adaptive learning strategy that controls the knowledge contribution of each model by assigning weights as per their prediction precision. Distilling knowledge from adversarially pre-trained teacher models not only enhances the learning capabilities of the student model but also empowers it with the capacity to withstand different adversarial attacks, despite having no exposure to adversarial data. To verify our claims, we extensively evaluated our proposed method on MNIST-Digits and Fashion-MNIST datasets across diverse experimental settings. The obtained results exhibit the efficacy of our multi-teacher adversarial distillation and adaptive learning strategy, enhancing CNNs' adversarial robustness against various adversarial attacks.

http://arxiv.org/abs/2507.20526
Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition. (83%)
Andy Zou; Maxwell Lin; Eliot Jones; Micha Nowak; Mateusz Dziemian; Nick Winter; Alexander Grattan; Valent Nathanael; Ayla Croft; Xander Davies; Jai Patel; Robert Kirk; Nate Burnikell; Yarin Gal; Dan Hendrycks; J. Zico Kolter; Matt Fredrikson
Recent advances have enabled LLM-powered AI agents to autonomously execute complex tasks by combining language model reasoning with tools, memory, and web access. But can these systems be trusted to follow deployment policies in realistic environments, especially under attack? To investigate, we ran the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios. Participants submitted 1.8 million prompt-injection attacks, with over 60,000 successfully eliciting policy violations such as unauthorized data access, illicit financial actions, and regulatory noncompliance. We use these results to build the Agent Red Teaming (ART) benchmark - a curated set of high-impact attacks - and evaluate it across 19 state-of-the-art models. Nearly all agents exhibit policy violations for most behaviors within 10-100 queries, with high attack transferability across models and tasks. Importantly, we find limited correlation between agent robustness and model size, capability, or inference-time compute, suggesting that additional defenses are needed against adversarial misuse. Our findings highlight critical and persistent vulnerabilities in today's AI agents. By releasing the ART benchmark and accompanying evaluation framework, we aim to support more rigorous security assessment and drive progress toward safer agent deployment.

http://arxiv.org/abs/2409.13864
Persistent Backdoor Attacks in Continual Learning. (73%)
Zhen Guo; Abhinav Kumar; Reza Tourani
Backdoor attacks pose a significant threat to neural networks, enabling adversaries to manipulate model outputs on specific inputs, often with devastating consequences, especially in critical applications. While backdoor attacks have been studied in various contexts, little attention has been given to their practicality and persistence in continual learning, particularly in understanding how the continual updates to model parameters, as new data distributions are learned and integrated, impact the effectiveness of these attacks over time. To address this gap, we introduce two persistent backdoor attacks-Blind Task Backdoor and Latent Task Backdoor-each leveraging minimal adversarial influence. Our blind task backdoor subtly alters the loss computation without direct control over the training process, while the latent task backdoor influences only a single task's training, with all other tasks trained benignly. We evaluate these attacks under various configurations, demonstrating their efficacy with static, dynamic, physical, and semantic triggers. Our results show that both attacks consistently achieve high success rates across different continual learning algorithms, while effectively evading state-of-the-art defenses, such as SentiNet and I-BAU.

http://arxiv.org/abs/2505.16789
Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards. (69%)
Punya Syon Pandey; Samuel Simko; Kellin Pelrine; Zhijing Jin
As large language models (LLMs) gain popularity, their vulnerability to adversarial attacks emerges as a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can inadvertently introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Vulnerability, unexpected vulnerabilities arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity across multiple experimental datasets. We then evaluate the adversarial robustness of these fine-tuned models, analyzing persona shifts and interpretability traits to understand how dataset factors contribute to attack success rates. Lastly, we explore causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment. Our code is available at https://github.com/psyonp/accidental_vulnerability.

http://arxiv.org/abs/2402.08290
The Effect of Data Poisoning on Counterfactual Explanations. (22%)
André Artelt; Shubham Sharma; Freddy Lecué; Barbara Hammer
Counterfactual explanations are a widely used approach for examining the predictions of black-box systems. They can offer the opportunity for computational recourse by suggesting actionable changes on how to alter the input to obtain a different (i.e., more favorable) system output. However, recent studies have pointed out their susceptibility to various forms of manipulation.
  This work studies the vulnerability of counterfactual explanations to data poisoning. We formally introduce and investigate data poisoning in the context of counterfactual explanations for increasing the cost of recourse on three different levels: locally for a single instance, a sub-group of instances, or globally for all instances. In this context, we formally introduce and characterize data poisonings, from which we derive and investigate a general data poisoning mechanism. We demonstrate the impact of such data poisoning in the critical real-world application of explaining event detections in water distribution networks. Additionally, we conduct an extensive empirical evaluation, demonstrating that state-of-the-art counterfactual generation methods and toolboxes are vulnerable to such data poisoning. Furthermore, we find that existing defense methods fail to detect those poisonous samples.

http://arxiv.org/abs/2507.20434
Is Crunching Public Data the Right Approach to Detect BGP Hijacks? (15%)
Alessandro Giaconia; Muoi Tran; Laurent Vanbever; Stefano Vissicchio
The Border Gateway Protocol (BGP) remains a fragile pillar of Internet routing. BGP hijacks still occurr daily. While full deployment of Route Origin Validation (ROV) is ongoing, attackers have already adapted, launching post-ROV attacks such as forged-origin hijacks. To detect these, recent approaches like DFOH [Holterbach et al., USENIX NSDI '24] and BEAM [Chen et al., USENIX Security '24] apply machine learning (ML) to analyze data from globally distributed BGP monitors, assuming anomalies will stand out against historical patterns. However, this assumption overlooks a key threat: BGP monitors themselves can be misled by adversaries injecting bogus routes. This paper shows that state-of-the-art hijack detection systems like DFOH and BEAM are vulnerable to data poisoning. Using large-scale BGP simulations, we show that attackers can evade detection with just a handful of crafted announcements beyond the actual hijack. These announcements are indeed sufficient to corrupt the knowledge base used by ML-based defenses and distort the metrics they rely on. Our results highlight a worrying weakness of relying solely on public BGP data.

http://arxiv.org/abs/2408.07703
Knowledge Distillation with Refined Logits. (1%)
Wujie Sun; Defang Chen; Siwei Lyu; Genlang Chen; Chun Chen; Can Wang
Recent research on knowledge distillation has increasingly focused on logit distillation because of its simplicity, effectiveness, and versatility in model compression. In this paper, we introduce Refined Logit Distillation (RLD) to address the limitations of current logit distillation methods. Our approach is motivated by the observation that even high-performing teacher models can make incorrect predictions, creating an exacerbated divergence between the standard distillation loss and the cross-entropy loss, which can undermine the consistency of the student model's learning objectives. Previous attempts to use labels to empirically correct teacher predictions may undermine the class correlations. In contrast, our RLD employs labeling information to dynamically refine teacher logits. In this way, our method can effectively eliminate misleading information from the teacher while preserving crucial class correlations, thus enhancing the value and efficiency of distilled knowledge. Experimental results on CIFAR-100 and ImageNet demonstrate its superiority over existing methods. Our code is available at https://github.com/zju-SWJ/RLD.

http://arxiv.org/abs/2507.21177
FedBAP: Backdoor Defense via Benign Adversarial Perturbation in Federated Learning. (92%)
Xinhai Yan; Libing Wu; Zhuangzhuang Zhang; Bingyi Liu; Lijuan Huo; Jing Wang
Federated Learning (FL) enables collaborative model training while preserving data privacy, but it is highly vulnerable to backdoor attacks. Most existing defense methods in FL have limited effectiveness due to their neglect of the model's over-reliance on backdoor triggers, particularly as the proportion of malicious clients increases. In this paper, we propose FedBAP, a novel defense framework for mitigating backdoor attacks in FL by reducing the model's reliance on backdoor triggers. Specifically, first, we propose a perturbed trigger generation mechanism that creates perturbation triggers precisely matching backdoor triggers in location and size, ensuring strong influence on model outputs. Second, we utilize these perturbation triggers to generate benign adversarial perturbations that disrupt the model's dependence on backdoor triggers while forcing it to learn more robust decision boundaries. Finally, we design an adaptive scaling mechanism to dynamically adjust perturbation intensity, effectively balancing defense strength and model performance. The experimental results demonstrate that FedBAP reduces the attack success rates by 0.22%-5.34%, 0.48%-6.34%, and 97.22%-97.6% under three types of backdoor attacks, respectively. In particular, FedBAP demonstrates outstanding performance against novel backdoor attacks.

http://arxiv.org/abs/2507.19739
Enhancing IoT Intrusion Detection Systems through Adversarial Training. (84%)
Karma Gurung; Ashutosh Ghimire; Fathi Amsaad
The augmentation of Internet of Things (IoT) devices transformed both automation and connectivity but revealed major security vulnerabilities in networks. We address these challenges by designing a robust intrusion detection system (IDS) to detect complex attacks by learning patterns from the NF-ToN-IoT v2 dataset. Intrusion detection has a realistic testbed through the dataset's rich and high-dimensional features. We combine distributed preprocessing to manage the dataset size with Fast Gradient Sign Method (FGSM) adversarial attacks to mimic actual attack scenarios and XGBoost model adversarial training for improved system robustness. Our system achieves 95.3% accuracy on clean data and 94.5% accuracy on adversarial data to show its effectiveness against complex threats. Adversarial training demonstrates its potential to strengthen IDS against evolving cyber threats and sets the foundation for future studies. Real-time IoT environments represent a future deployment opportunity for these systems, while extensions to detect emerging threats and zero-day vulnerabilities would enhance their utility.

http://arxiv.org/abs/2507.19905
ConSeg: Contextual Backdoor Attack Against Semantic Segmentation. (50%)
Bilal Hussain Abbasi; Zirui Gong; Yanjun Zhang; Shang Gao; Antonio Robles-Kelly; Leo Zhang
Despite significant advancements in computer vision, semantic segmentation models may be susceptible to backdoor attacks. These attacks, involving hidden triggers, aim to cause the models to misclassify instances of the victim class as the target class when triggers are present, posing serious threats to the reliability of these models. To further explore the field of backdoor attacks against semantic segmentation, in this paper, we propose a simple yet effective backdoor attack called Contextual Segmentation Backdoor Attack (ConSeg). ConSeg leverages the contextual information inherent in semantic segmentation models to enhance backdoor performance. Our method is motivated by an intriguing observation, i.e., when the target class is set as the `co-occurring' class of the victim class, the victim class can be more easily `mis-segmented'. Building upon this insight, ConSeg mimics the contextual information of the target class and rebuilds it in the victim region to establish the contextual relationship between the target class and the victim class, making the attack easier. Our experiments reveal that ConSeg achieves improvements in Attack Success Rate (ASR) with increases of 15.55\%, compared to existing methods, while exhibiting resilience against state-of-the-art backdoor defenses.

http://arxiv.org/abs/2507.19880
Trivial Trojans: How Minimal MCP Servers Enable Cross-Tool Exfiltration of Sensitive Data. (2%)
Nicola Croce; Tobin South
The Model Context Protocol (MCP) represents a significant advancement in AI-tool integration, enabling seamless communication between AI agents and external services. However, this connectivity introduces novel attack vectors that remain largely unexplored. This paper demonstrates how unsophisticated threat actors, requiring only basic programming skills and free web tools, can exploit MCP's trust model to exfiltrate sensitive financial data. We present a proof-of-concept attack where a malicious weather MCP server, disguised as benign functionality, discovers and exploits legitimate banking tools to steal user account balances. The attack chain requires no advanced technical knowledge, server infrastructure, or monetary investment. The findings reveal a critical security gap in the emerging MCP ecosystem: while individual servers may appear trustworthy, their combination creates unexpected cross-server attack surfaces. Unlike traditional cybersecurity threats that assume sophisticated adversaries, our research shows that the barrier to entry for MCP-based attacks is alarmingly low. A threat actor with undergraduate-level Python knowledge can craft convincing social engineering attacks that exploit the implicit trust relationships MCP establishes between AI agents and tool providers. This work contributes to the nascent field of MCP security by demonstrating that current MCP implementations allow trivial cross-server attacks and proposing both immediate mitigations and protocol improvements to secure this emerging ecosystem.

http://arxiv.org/abs/2507.21163
Generating Adversarial Point Clouds Using Diffusion Model. (99%)
Ruiyang Zhao; Bingbing Zhu; Chuxuan Tong; Xiaoyi Zhou; Xi Zheng
Adversarial attack methods for 3D point cloud classification reveal the vulnerabilities of point cloud recognition models. This vulnerability could lead to safety risks in critical applications that use deep learning models, such as autonomous vehicles. To uncover the deficiencies of these models, researchers can evaluate their security through adversarial attacks. However, most existing adversarial attack methods are based on white-box attacks. While these methods achieve high attack success rates and imperceptibility, their applicability in real-world scenarios is limited. Black-box attacks, which are more meaningful in real-world scenarios, often yield poor results. This paper proposes a novel black-box adversarial example generation method that utilizes a diffusion model to improve the attack success rate and imperceptibility in the black-box setting, without relying on the internal information of the point cloud classification model to generate adversarial samples. We use a 3D diffusion model to use the compressed features of the point cloud as prior knowledge to guide the reverse diffusion process to add adversarial points to clean examples. Subsequently, its reverse process is employed to transform the distribution of other categories into adversarial points, which are then added to the point cloud.

http://arxiv.org/abs/2507.18870
Transferable and Undefendable Point Cloud Attacks via Medial Axis Transform. (99%)
Keke Tang; Yuze Gao; Weilong Peng; Xiaofei Wang; Meie Fang; Peican Zhu
Studying adversarial attacks on point clouds is essential for evaluating and improving the robustness of 3D deep learning models. However, most existing attack methods are developed under ideal white-box settings and often suffer from limited transferability to unseen models and insufficient robustness against common defense mechanisms. In this paper, we propose MAT-Adv, a novel adversarial attack framework that enhances both transferability and undefendability by explicitly perturbing the medial axis transform (MAT) representations, in order to induce inherent adversarialness in the resulting point clouds. Specifically, we employ an autoencoder to project input point clouds into compact MAT representations that capture the intrinsic geometric structure of point clouds. By perturbing these intrinsic representations, MAT-Adv introduces structural-level adversarial characteristics that remain effective across diverse models and defense strategies. To mitigate overfitting and prevent perturbation collapse, we incorporate a dropout strategy into the optimization of MAT perturbations, further improving transferability and undefendability. Extensive experiments demonstrate that MAT-Adv significantly outperforms existing state-of-the-art methods in both transferability and undefendability. Codes will be made public upon paper acceptance.

http://arxiv.org/abs/2507.05113
CLIP-Guided Backdoor Defense through Entropy-Based Poisoned Dataset Separation. (76%)
Binyan Xu; Fan Yang; Xilin Dai; Di Tang; Kehuan Zhang
Deep Neural Networks (DNNs) are susceptible to backdoor attacks, where adversaries poison training data to implant backdoor into the victim model. Current backdoor defenses on poisoned data often suffer from high computational costs or low effectiveness against advanced attacks like clean-label and clean-image backdoors. To address them, we introduce CLIP-Guided backdoor Defense (CGD), an efficient and effective method that mitigates various backdoor attacks. CGD utilizes a publicly accessible CLIP model to identify inputs that are likely to be clean or poisoned. It then retrains the model with these inputs, using CLIP's logits as a guidance to effectively neutralize the backdoor. Experiments on 4 datasets and 11 attack types demonstrate that CGD reduces attack success rates (ASRs) to below 1% while maintaining clean accuracy (CA) with a maximum drop of only 0.3%, outperforming existing defenses. Additionally, we show that clean-data-based defenses can be adapted to poisoned data using CGD. Also, CGD exhibits strong robustness, maintaining low ASRs even when employing a weaker CLIP model or when CLIP itself is compromised by a backdoor. These findings underscore CGD's exceptional efficiency, effectiveness, and applicability for real-world backdoor defense scenarios. Code: https://github.com/binyxu/CGD.

http://arxiv.org/abs/2505.15265
Blind Spot Navigation: Evolutionary Discovery of Sensitive Semantic Concepts for LVLMs. (38%)
Zihao Pan; Yu Tong; Weibin Wu; Jingyi Wang; Lifeng Chen; Zhe Zhao; Jiajia Wei; Yitong Qiao; Zibin Zheng
Adversarial attacks aim to generate malicious inputs that mislead deep models, but beyond causing model failure, they cannot provide certain interpretable information such as ``\textit{What content in inputs make models more likely to fail?}'' However, this information is crucial for researchers to specifically improve model robustness. Recent research suggests that models may be particularly sensitive to certain semantics in visual inputs (such as ``wet,'' ``foggy''), making them prone to errors. Inspired by this, in this paper we conducted the first exploration on large vision-language models (LVLMs) and found that LVLMs indeed are susceptible to hallucinations and various errors when facing specific semantic concepts in images. To efficiently search for these sensitive concepts, we integrated large language models (LLMs) and text-to-image (T2I) models to propose a novel semantic evolution framework. Randomly initialized semantic concepts undergo LLM-based crossover and mutation operations to form image descriptions, which are then converted by T2I models into visual inputs for LVLMs. The task-specific performance of LVLMs on each input is quantified as fitness scores for the involved semantics and serves as reward signals to further guide LLMs in exploring concepts that induce LVLMs. Extensive experiments on seven mainstream LVLMs and two multimodal tasks demonstrate the effectiveness of our method. Additionally, we provide interesting findings about the sensitive semantics of LVLMs, aiming to inspire further in-depth research.

http://arxiv.org/abs/2506.15606
LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning. (22%)
Gabriel J. Perin; Runjin Chen; Xuxi Chen; Nina S. T. Hirata; Zhangyang Wang; Junyuan Hong
Large Language Models (LLMs) have become indispensable in real-world applications. However, their widespread adoption raises significant safety concerns, particularly in responding to socially harmful questions. Despite substantial efforts to improve model safety through alignment, aligned models can still have their safety protections undermined by subsequent fine-tuning - even when the additional training data appears benign. In this paper, we empirically demonstrate that this vulnerability stems from the sensitivity of safety-critical low-rank subspaces in LLM parameters to fine-tuning. Building on this insight, we propose a novel training-free method, termed Low-Rank Extrapolation (LoX), to enhance safety robustness by extrapolating the safety subspace of an aligned LLM. Our experimental results confirm the effectiveness of LoX, demonstrating significant improvements in robustness against both benign and malicious fine-tuning attacks while preserving the model's adaptability to new tasks. For instance, LoX leads to 11% to 54% absolute reductions in attack success rates (ASR) facing benign or malicious fine-tuning attacks. By investigating the ASR landscape of parameters, we attribute the success of LoX to that the extrapolation moves LLM parameters to a flatter zone, thereby less sensitive to perturbations. The code is available at github.com/VITA-Group/LoX.

http://arxiv.org/abs/2504.21019
Kill two birds with one stone: generalized and robust AI-generated text detection via dynamic perturbations. (4%)
Yinghan Zhou; Juan Wen; Wanli Peng; Yiming Xue; Ziwei Zhang; Zhengxian Wu
The growing popularity of large language models has raised concerns regarding the potential to misuse AI-generated text (AIGT). It becomes increasingly critical to establish an excellent AIGT detection method with high generalization and robustness. However, existing methods either focus on model generalization or concentrate on robustness. The unified mechanism, to simultaneously address the challenges of generalization and robustness, is less explored. In this paper, we argue that robustness can be view as a specific form of domain shift, and empirically reveal an intrinsic mechanism for model generalization of AIGT detection task. Then, we proposed a novel AIGT detection method (DP-Net) via dynamic perturbations introduced by a reinforcement learning with elaborated reward and action. Experimentally, extensive results show that the proposed DP-Net significantly outperforms some state-of-the-art AIGT detection methods for generalization capacity in three cross-domain scenarios. Meanwhile, the DP-Net achieves best robustness under two text adversarial attacks. The code is publicly available at https://github.com/CAU-ISS-Lab/AIGT-Detection-Evade-Detection/tree/main/DP-Net.

http://arxiv.org/abs/2507.19609
Securing the Internet of Medical Things (IoMT): Real-World Attack Taxonomy and Practical Security Measures. (1%)
Suman Deb; Emil Lupu; Emm Mic Drakakis; Anil Anthony Bharath; Zhen Kit Leung; Guang Rui Ma; Anupam Chattopadhyay
The Internet of Medical Things (IoMT) has the potential to radically improve healthcare by enabling real-time monitoring, remote diagnostics, and AI-driven decision making. However, the connectivity, embedded intelligence, and inclusion of a wide variety of novel sensors expose medical devices to severe cybersecurity threats, compromising patient safety and data privacy. In addition, many devices also have direct capacity - individually or in conjunction with other IoMT devices - to perform actions on the patient, such as delivering an electrical stimulus, administering a drug, or activating a motor, which can potentially be life-threatening. We provide a taxonomy of potential attacks targeting IoMT, presenting attack surfaces, vulnerabilities, and mitigation strategies across all layers of the IoMT architecture. It answers key questions such as: What makes IoMT security different from traditional IT security? What are the cybersecurity threats to medical devices? How can engineers design secure IoMT systems and protect hospital networks from cyberattacks? By analyzing historical cyber incidents, we highlight critical security gaps and propose practical security guidelines for medical device engineers and security professionals. This work bridges the gap between research and implementation, equipping healthcare stakeholders with actionable insights to build resilient and privacy-preserving IoMT ecosystems. Finally, we present the latest standardization and compliance frameworks, that IoMT security designers should be aware of.

http://arxiv.org/abs/2507.19368
Counterfactual Explanations in Medical Imaging: Exploring SPN-Guided Latent Space Manipulation. (1%)
Julia Siekiera; Stefan Kramer
Artificial intelligence is increasingly leveraged across various domains to automate decision-making processes that significantly impact human lives. In medical image analysis, deep learning models have demonstrated remarkable performance. However, their inherent complexity makes them black box systems, raising concerns about reliability and interpretability. Counterfactual explanations provide comprehensible insights into decision processes by presenting hypothetical "what-if" scenarios that alter model classifications. By examining input alterations, counterfactual explanations provide patterns that influence the decision-making process. Despite their potential, generating plausible counterfactuals that adhere to similarity constraints providing human-interpretable explanations remains a challenge. In this paper, we investigate this challenge by a model-specific optimization approach. While deep generative models such as variational autoencoders (VAEs) exhibit significant generative power, probabilistic models like sum-product networks (SPNs) efficiently represent complex joint probability distributions. By modeling the likelihood of a semi-supervised VAE's latent space with an SPN, we leverage its dual role as both a latent space descriptor and a classifier for a given discrimination task. This formulation enables the optimization of latent space counterfactuals that are both close to the original data distribution and aligned with the target class distribution. We conduct experimental evaluation on the cheXpert dataset. To evaluate the effectiveness of the integration of SPNs, our SPN-guided latent space manipulation is compared against a neural network baseline. Additionally, the trade-off between latent variable regularization and counterfactual quality is analyzed.

http://arxiv.org/abs/2502.20268
Large Language Models as Attribution Regularizers for Efficient Model Training. (1%)
Davor Vukadin; Marin Šilić; Goran Delač
Large Language Models (LLMs) have demonstrated remarkable performance across diverse domains. However, effectively leveraging their vast knowledge for training smaller downstream models remains an open challenge, especially in domains like tabular data learning, where simpler models are often preferred due to interpretability and efficiency.
  In this paper, we introduce a novel yet straightforward method for incorporating LLM-generated global task feature attributions into the training process of smaller networks. Specifically, we propose an attribution-matching regularization term that aligns the training dynamics of the smaller model with the insights provided by the LLM. By doing so, our approach yields superior performance in few-shot learning scenarios. Notably, our method requires only black-box API access to the LLM, making it easy to integrate into existing training pipelines with minimal computational overhead.
  Furthermore, we demonstrate how this method can be used to address common issues in real-world datasets, such as skewness and bias. By integrating high-level knowledge from LLMs, our approach improves generalization, even when training data is limited or imbalanced. We validate its effectiveness through extensive experiments across multiple tasks, demonstrating improved learning efficiency and model robustness.

http://arxiv.org/abs/2507.18484
Reinforced Embodied Active Defense: Exploiting Adaptive Interaction for Robust Visual Perception in Adversarial 3D Environments. (99%)
Xiao Yang; Lingxuan Wu; Lizhong Wang; Chengyang Ying; Hang Su; Jun Zhu
Adversarial attacks in 3D environments have emerged as a critical threat to the reliability of visual perception systems, particularly in safety-sensitive applications such as identity verification and autonomous driving. These attacks employ adversarial patches and 3D objects to manipulate deep neural network (DNN) predictions by exploiting vulnerabilities within complex scenes. Existing defense mechanisms, such as adversarial training and purification, primarily employ passive strategies to enhance robustness. However, these approaches often rely on pre-defined assumptions about adversarial tactics, limiting their adaptability in dynamic 3D settings. To address these challenges, we introduce Reinforced Embodied Active Defense (Rein-EAD), a proactive defense framework that leverages adaptive exploration and interaction with the environment to improve perception robustness in 3D adversarial contexts. By implementing a multi-step objective that balances immediate prediction accuracy with predictive entropy minimization, Rein-EAD optimizes defense strategies over a multi-step horizon. Additionally, Rein-EAD involves an uncertainty-oriented reward-shaping mechanism that facilitates efficient policy updates, thereby reducing computational overhead and supporting real-world applicability without the need for differentiable environments. Comprehensive experiments validate the effectiveness of Rein-EAD, demonstrating a substantial reduction in attack success rates while preserving standard accuracy across diverse tasks. Notably, Rein-EAD exhibits robust generalization to unseen and adaptive attacks, making it suitable for real-world complex tasks, including 3D object classification, face recognition and autonomous driving.

http://arxiv.org/abs/2309.03791
Optimal Transport Regularized Divergences: Application to Adversarial Robustness. (95%)
Jeremiah Birrell; Reza Ebrahimi
We introduce a new class of optimal-transport-regularized divergences, $D^c$, constructed via an infimal convolution between an information divergence, $D$, and an optimal-transport (OT) cost, $C$, and study their use in distributionally robust optimization (DRO). In particular, we propose the $ARMOR_D$ methods as novel approaches to enhancing the adversarial robustness of deep learning models. These DRO-based methods are defined by minimizing the maximum expected loss over a $D^c$-neighborhood of the empirical distribution of the training data. Viewed as a tool for constructing adversarial samples, our method allows samples to be both transported, according to the OT cost, and re-weighted, according to the information divergence; the addition of a principled and dynamical adversarial re-weighting on top of adversarial sample transport is a key innovation of $ARMOR_D$. $ARMOR_D$ can be viewed as a generalization of the best-performing loss functions and OT costs in the adversarial training literature; we demonstrate this flexibility by using $ARMOR_D$ to augment the UDR, TRADES, and MART methods and obtain improved performance on CIFAR-10 and CIFAR-100 image recognition. Specifically, augmenting with $ARMOR_D$ leads to 1.9\% and 2.1\% improvement against AutoAttack, a powerful ensemble of adversarial attacks, on CIFAR-10 and CIFAR-100 respectively. To foster reproducibility, we made the code accessible at https://github.com/star-ailab/ARMOR.

http://arxiv.org/abs/2507.18113
Policy Disruption in Reinforcement Learning:Adversarial Attack with Large Language Models and Critical State Identification. (93%)
Junyong Jiang; Buwei Tian; Chenxing Xu; Songze Li; Lu Dong
Reinforcement learning (RL) has achieved remarkable success in fields like robotics and autonomous driving, but adversarial attacks designed to mislead RL systems remain challenging. Existing approaches often rely on modifying the environment or policy, limiting their practicality. This paper proposes an adversarial attack method in which existing agents in the environment guide the target policy to output suboptimal actions without altering the environment. We propose a reward iteration optimization framework that leverages large language models (LLMs) to generate adversarial rewards explicitly tailored to the vulnerabilities of the target agent, thereby enhancing the effectiveness of inducing the target agent toward suboptimal decision-making. Additionally, a critical state identification algorithm is designed to pinpoint the target agent's most vulnerable states, where suboptimal behavior from the victim leads to significant degradation in overall performance. Experimental results in diverse environments demonstrate the superiority of our method over existing approaches.

http://arxiv.org/abs/2507.21157
Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems. (74%)
Naseem Khan; Tuan Nguyen; Amine Bermak; Issa Khalil
The rapid advancement of Generative Artificial Intelligence has fueled deepfake proliferation-synthetic media encompassing fully generated content and subtly edited authentic material-posing challenges to digital security, misinformation mitigation, and identity preservation. This systematic review evaluates state-of-the-art deepfake detection methodologies, emphasizing reproducible implementations for transparency and validation. We delineate two core paradigms: (1) detection of fully synthetic media leveraging statistical anomalies and hierarchical feature extraction, and (2) localization of manipulated regions within authentic content employing multi-modal cues such as visual artifacts and temporal inconsistencies. These approaches, spanning uni-modal and multi-modal frameworks, demonstrate notable precision and adaptability in controlled settings, effectively identifying manipulations through advanced learning techniques and cross-modal fusion. However, comprehensive assessment reveals insufficient evaluation of adversarial robustness across both paradigms. Current methods exhibit vulnerability to adversarial perturbations-subtle alterations designed to evade detection-undermining reliability in real-world adversarial contexts. This gap highlights critical disconnect between methodological development and evolving threat landscapes. To address this, we contribute a curated GitHub repository aggregating open-source implementations, enabling replication and testing. Our findings emphasize urgent need for future work prioritizing adversarial resilience, advocating scalable, modality-agnostic architectures capable of withstanding sophisticated manipulations. This review synthesizes strengths and shortcomings of contemporary deepfake detection while charting paths toward robust trustworthy systems.

http://arxiv.org/abs/2507.18313
Regression-aware Continual Learning for Android Malware Detection. (61%)
Daniele Ghiani; Daniele Angioni; Giorgio Piras; Angelo Sotgiu; Luca Minnei; Srishti Gupta; Maura Pintor; Fabio Roli; Battista Biggio
Malware evolves rapidly, forcing machine learning (ML)-based detectors to adapt continuously. With antivirus vendors processing hundreds of thousands of new samples daily, datasets can grow to billions of examples, making full retraining impractical. Continual learning (CL) has emerged as a scalable alternative, enabling incremental updates without full data access while mitigating catastrophic forgetting. In this work, we analyze a critical yet overlooked issue in this context: security regression. Unlike forgetting, which manifests as a general performance drop on previously seen data, security regression captures harmful prediction changes at the sample level, such as a malware sample that was once correctly detected but evades detection after a model update. Although often overlooked, regressions pose serious risks in security-critical applications, as the silent reintroduction of previously detected threats in the system may undermine users' trust in the whole updating process. To address this issue, we formalize and quantify security regression in CL-based malware detectors and propose a regression-aware penalty to mitigate it. Specifically, we adapt Positive Congruent Training (PCT) to the CL setting, preserving prior predictive behavior in a model-agnostic manner. Experiments on the ELSA, Tesseract, and AZ-Class datasets show that our method effectively reduces regression across different CL scenarios while maintaining strong detection performance over time.

http://arxiv.org/abs/2507.18036
NWaaS: Nonintrusive Watermarking as a Service for X-to-Image DNN. (10%)
Haonan An; Guang Hua; Yu Guo; Hangcheng Cao; Susanto Rahardja; Yuguang Fang
The intellectual property of deep neural network (DNN) models can be protected with DNN watermarking, which embeds copyright watermarks into model parameters (white-box), model behavior (black-box), or model outputs (box-free), and the watermarks can be subsequently extracted to verify model ownership or detect model theft. Despite recent advances, these existing methods are inherently intrusive, as they either modify the model parameters or alter the structure. This natural intrusiveness raises concerns about watermarking-induced shifts in model behavior and the additional cost of fine-tuning, further exacerbated by the rapidly growing model size. As a result, model owners are often reluctant to adopt DNN watermarking in practice, which limits the development of practical Watermarking as a Service (WaaS) systems. To address this issue, we introduce Nonintrusive Watermarking as a Service (NWaaS), a novel trustless paradigm designed for X-to-Image models, in which we hypothesize that with the model untouched, an owner-defined watermark can still be extracted from model outputs. Building on this concept, we propose ShadowMark, a concrete implementation of NWaaS which addresses critical deployment challenges by establishing a robust and nonintrusive side channel in the protected model's black-box API, leveraging a key encoder and a watermark decoder. It is significantly distinctive from existing solutions by attaining the so-called absolute fidelity and being applicable to different DNN architectures, while being also robust against existing attacks, eliminating the fidelity-robustness trade-off. Extensive experiments on image-to-image, noise-to-image, noise-and-text-to-image, and text-to-image models, demonstrate the efficacy and practicality of ShadowMark for real-world deployment of nonintrusive DNN watermarking.

http://arxiv.org/abs/2507.18457
Revisiting Physically Realizable Adversarial Object Attack against LiDAR-based Detection: Clarifying Problem Formulation and Experimental Protocols. (8%)
Luo Cheng; Hanwei Zhang; Lijun Zhang; Holger Hermanns
Adversarial robustness in LiDAR-based 3D object detection is a critical research area due to its widespread application in real-world scenarios. While many digital attacks manipulate point clouds or meshes, they often lack physical realizability, limiting their practical impact. Physical adversarial object attacks remain underexplored and suffer from poor reproducibility due to inconsistent setups and hardware differences. To address this, we propose a device-agnostic, standardized framework that abstracts key elements of physical adversarial object attacks, supports diverse methods, and provides open-source code with benchmarking protocols in simulation and real-world settings. Our framework enables fair comparison, accelerates research, and is validated by successfully transferring simulated attacks to a physical LiDAR system. Beyond the framework, we offer insights into factors influencing attack success and advance understanding of adversarial robustness in real-world LiDAR perception.

http://arxiv.org/abs/2503.17724
Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models. (5%)
Jie Zhang; Zhongqi Wang; Shiguang Shan; Xilin Chen
Backdoor attacks targeting text-to-image diffusion models have advanced rapidly. However, current backdoor samples often exhibit two key abnormalities compared to benign samples: 1) Semantic Consistency, where backdoor prompts tend to generate images with similar semantic content even with significant textual variations to the prompts; 2) Attention Consistency, where the trigger induces consistent structural responses in the cross-attention maps. These consistencies leave detectable traces for defenders, making backdoors easier to identify. In this paper, toward stealthy backdoor samples, we propose Trigger without Trace (TwT) by explicitly mitigating these consistencies. Specifically, our approach leverages syntactic structures as backdoor triggers to amplify the sensitivity to textual variations, effectively breaking down the semantic consistency. Besides, a regularization method based on Kernel Maximum Mean Discrepancy (KMMD) is proposed to align the distribution of cross-attention responses between backdoor and benign samples, thereby disrupting attention consistency. Extensive experiments demonstrate that our method achieves a 97.5% attack success rate while exhibiting stronger resistance to defenses. It achieves an average of over 98% backdoor samples bypassing three state-of-the-art detection mechanisms, revealing the vulnerabilities of current backdoor defense methods. The code is available at https://github.com/Robin-WZQ/TwT.

http://arxiv.org/abs/2507.18202
Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection. (2%)
San Kim; Jonghwi Kim; Yejin Jeon; Gary Geunbae Lee
Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by providing external knowledge for accurate and up-to-date responses. However, this reliance on external sources exposes a security risk, attackers can inject poisoned documents into the knowledge base to steer the generation process toward harmful or misleading outputs. In this paper, we propose Gradient-based Masked Token Probability (GMTP), a novel defense method to detect and filter out adversarially crafted documents. Specifically, GMTP identifies high-impact tokens by examining gradients of the retriever's similarity function. These key tokens are then masked, and their probabilities are checked via a Masked Language Model (MLM). Since injected tokens typically exhibit markedly low masked-token probabilities, this enables GMTP to easily detect malicious documents and achieve high-precision filtering. Experiments demonstrate that GMTP is able to eliminate over 90% of poisoned content while retaining relevant documents, thus maintaining robust retrieval and generation performance across diverse datasets and adversarial settings.

http://arxiv.org/abs/2507.18631
Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment. (1%)
Hao Li; Lijun Li; Zhenghao Lu; Xianyi Wei; Rui Li; Jing Shao; Lei Sha
With rapid advancement and increasing accessibility of LLMs, fine-tuning aligned models has become a critical step for adapting them to real-world applications, which makes the safety of this fine-tuning process more important than ever. However, recent studies have highlighted a critical challenge: even when fine-tuning with seemingly benign downstream datasets, the safety of aligned LLMs can be compromised, making them more susceptible to malicious instructions. In this paper, we show that fine-tuning datasets often contain samples with safety-degrading features that are not easily identifiable on the surface. These samples can significantly degrade the safety alignment of LLMs during fine-tuning. To address this issue, we propose LARF, a \textbf{L}ayer-\textbf{A}ware \textbf{R}epresentation \textbf{F}iltering method. This method identifies safety-sensitive layers within the LLM and leverages their representations to detect which data samples in the post-training dataset contain safety-degrading features. Experimental results demonstrate that LARF can effectively identify benign data with safety-degrading features. After removing such data, the safety alignment degradation caused by fine-tuning is mitigated. Please see our code at \href{https://github.com/LLLeoLi/LARF}{https://github.com/LLLeoLi/LARF}.

http://arxiv.org/abs/2507.17554
An h-space Based Adversarial Attack for Protection Against Few-shot Personalization. (98%)
Xide Xu; Sandesh Kamath; Muhammad Atif Butt; Bogdan Raducanu
The versatility of diffusion models in generating customized images from few samples raises significant privacy concerns, particularly regarding unauthorized modifications of private content. This concerning issue has renewed the efforts in developing protection mechanisms based on adversarial attacks, which generate effective perturbations to poison diffusion models. Our work is motivated by the observation that these models exhibit a high degree of abstraction within their semantic latent space (`h-space'), which encodes critical high-level features for generating coherent and meaningful content. In this paper, we propose a novel anti-customization approach, called HAAD (h-space based Adversarial Attack for Diffusion models), that leverages adversarial attacks to craft perturbations based on the h-space that can efficiently degrade the image generation process. Building upon HAAD, we further introduce a more efficient variant, HAAD-KV, that constructs perturbations solely based on the KV parameters of the h-space. This strategy offers a stronger protection, that is computationally less expensive. Despite their simplicity, our methods outperform state-of-the-art adversarial attacks, highlighting their effectiveness.

http://arxiv.org/abs/2507.17725
On the Interaction of Compressibility and Adversarial Robustness. (92%)
Melih Barsbey; Antônio H. Ribeiro; Umut Şimşekli; Tolga Birdal
Modern neural networks are expected to simultaneously satisfy a host of desirable properties: accurate fitting to training data, generalization to unseen inputs, parameter and computational efficiency, and robustness to adversarial perturbations. While compressibility and robustness have each been studied extensively, a unified understanding of their interaction still remains elusive. In this work, we develop a principled framework to analyze how different forms of compressibility - such as neuron-level sparsity and spectral compressibility - affect adversarial robustness. We show that these forms of compression can induce a small number of highly sensitive directions in the representation space, which adversaries can exploit to construct effective perturbations. Our analysis yields a simple yet instructive robustness bound, revealing how neuron and spectral compressibility impact $L_\infty$ and $L_2$ robustness via their effects on the learned representations. Crucially, the vulnerabilities we identify arise irrespective of how compression is achieved - whether via regularization, architectural bias, or implicit learning dynamics. Through empirical evaluations across synthetic and realistic tasks, we confirm our theoretical predictions, and further demonstrate that these vulnerabilities persist under adversarial training and transfer learning, and contribute to the emergence of universal adversarial perturbations. Our findings show a fundamental tension between structured compressibility and robustness, and suggest new pathways for designing models that are both efficient and secure.

http://arxiv.org/abs/2507.17577
Boosting Ray Search Procedure of Hard-label Attacks with Transfer-based Priors. (76%)
Chen Ma; Xinjie Xu; Shuyu Cheng; Qi Xuan
One of the most practical and challenging types of black-box adversarial attacks is the hard-label attack, where only the top-1 predicted label is available. One effective approach is to search for the optimal ray direction from the benign image that minimizes the $\ell_p$-norm distance to the adversarial region. The unique advantage of this approach is that it transforms the hard-label attack into a continuous optimization problem. The objective function value is the ray's radius, which can be obtained via binary search at a high query cost. Existing methods use a "sign trick" in gradient estimation to reduce the number of queries. In this paper, we theoretically analyze the quality of this gradient estimation and propose a novel prior-guided approach to improve ray search efficiency both theoretically and empirically. Specifically, we utilize the transfer-based priors from surrogate models, and our gradient estimators appropriately integrate them by approximating the projection of the true gradient onto the subspace spanned by these priors and random directions, in a query-efficient manner. We theoretically derive the expected cosine similarities between the obtained gradient estimators and the true gradient, and demonstrate the improvement achieved by incorporating priors. Extensive experiments on the ImageNet and CIFAR-10 datasets show that our approach significantly outperforms 11 state-of-the-art methods in terms of query efficiency.

http://arxiv.org/abs/2409.19638
BadHMP: Backdoor Attack against Human Motion Prediction. (61%)
Chaohui Xu; Si Wang; Chip-Hong Chang
Precise future human motion prediction over sub-second horizons from past observations is crucial for various safety-critical applications. To date, only a few studies have examined the vulnerability of skeleton-based neural networks to evasion and backdoor attacks. In this paper, we propose BadHMP, a novel backdoor attack that targets specifically human motion prediction tasks. Our approach involves generating poisoned training samples by embedding a localized backdoor trigger in one limb of the skeleton, causing selected joints to follow predefined motion in historical time steps. Subsequently, the future sequences are globally modified that all the joints move following the target trajectories. Our carefully designed backdoor triggers and targets guarantee the smoothness and naturalness of the poisoned samples, making them stealthy enough to evade detection by the model trainer while keeping the poisoned model unobtrusive in terms of prediction fidelity to untainted sequences. The target sequences can be successfully activated by the designed input sequences even with a low poisoned sample injection ratio. Experimental results on two datasets (Human3.6M and CMU-Mocap) and two network architectures (LTD and HRI) demonstrate the high-fidelity, effectiveness, and stealthiness of BadHMP. Robustness of our attack against fine-tuning defense is also verified.

http://arxiv.org/abs/2507.21145
Leveraging Trustworthy AI for Automotive Security in Multi-Domain Operations: Towards a Responsive Human-AI Multi-Domain Task Force for Cyber Social Security. (61%)
Vita Santa Barletta; Danilo Caivano; Gabriel Cellammare; Vescovo Samuele del; Annita Larissa Sciacovelli
Multi-Domain Operations (MDOs) emphasize cross-domain defense against complex and synergistic threats, with civilian infrastructures like smart cities and Connected Autonomous Vehicles (CAVs) emerging as primary targets. As dual-use assets, CAVs are vulnerable to Multi-Surface Threats (MSTs), particularly from Adversarial Machine Learning (AML) which can simultaneously compromise multiple in-vehicle ML systems (e.g., Intrusion Detection Systems, Traffic Sign Recognition Systems). Therefore, this study investigates how key hyperparameters in Decision Tree-based ensemble models-Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB)-affect the time required for a Black-Box AML attack i.e. Zeroth Order Optimization (ZOO). Findings show that parameters like the number of trees or boosting rounds significantly influence attack execution time, with RF and GB being more sensitive than XGB. Adversarial Training (AT) time is also analyzed to assess the attacker's window of opportunity. By optimizing hyperparameters, this research supports Defensive Trustworthy AI (D-TAI) practices within MST scenarios and contributes to the development of resilient ML systems for civilian and military domains, aligned with Cyber Social Security framework in MDOs and Human-AI Multi-Domain Task Forces.

http://arxiv.org/abs/2507.17922
From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models. (26%)
Jessica Quaye; Charvi Rastogi; Alicia Parrish; Oana Inel; Minsuk Kahng; Lora Aroyo; Vijay Janapa Reddi
Text-to-image (T2I) models have become prevalent across numerous applications, making their robust evaluation against adversarial attacks a critical priority. Continuous access to new and challenging adversarial prompts across diverse domains is essential for stress-testing these models for resilience against novel attacks from multiple vectors. Current techniques for generating such prompts are either entirely authored by humans or synthetically generated. On the one hand, datasets of human-crafted adversarial prompts are often too small in size and imbalanced in their cultural and contextual representation. On the other hand, datasets of synthetically-generated prompts achieve scale, but typically lack the realistic nuances and creative adversarial strategies found in human-crafted prompts. To combine the strengths of both human and machine approaches, we propose Seed2Harvest, a hybrid red-teaming method for guided expansion of culturally diverse, human-crafted adversarial prompt seeds. The resulting prompts preserve the characteristics and attack patterns of human prompts while maintaining comparable average attack success rates (0.31 NudeNet, 0.36 SD NSFW, 0.12 Q16). Our expanded dataset achieves substantially higher diversity with 535 unique geographic locations and a Shannon entropy of 7.48, compared to 58 locations and 5.28 entropy in the original dataset. Our work demonstrates the importance of human-machine collaboration in leveraging human creativity and machine computational capacity to achieve comprehensive, scalable red-teaming for continuous T2I model safety evaluation.

http://arxiv.org/abs/2507.17453
Efficient Neural Network Verification via Order Leading Exploration of Branch-and-Bound Trees. (15%)
Guanqin Zhang; Kota Fukuda; Zhenya Zhang; H. M. N. Dilum Bandara; Shiping Chen; Jianjun Zhao; Yulei Sui
The vulnerability of neural networks to adversarial perturbations has necessitated formal verification techniques that can rigorously certify the quality of neural networks. As the state-of-the-art, branch and bound (BaB) is a "divide-and-conquer" strategy that applies off-the-shelf verifiers to sub-problems for which they perform better. While BaB can identify the sub-problems that are necessary to be split, it explores the space of these sub-problems in a naive "first-come-first-serve" manner, thereby suffering from an issue of inefficiency to reach a verification conclusion. To bridge this gap, we introduce an order over different sub-problems produced by BaB, concerning with their different likelihoods of containing counterexamples. Based on this order, we propose a novel verification framework Oliva that explores the sub-problem space by prioritizing those sub-problems that are more likely to find counterexamples, in order to efficiently reach the conclusion of the verification. Even if no counterexample can be found in any sub-problem, it only changes the order of visiting different sub-problem and so will not lead to a performance degradation. Specifically, Oliva has two variants, including $Oliva^{GR}$, a greedy strategy that always prioritizes the sub-problems that are more likely to find counterexamples, and $Oliva^{SA}$, a balanced strategy inspired by simulated annealing that gradually shifts from exploration to exploitation to locate the globally optimal sub-problems. We experimentally evaluate the performance of Oliva on 690 verification problems spanning over 5 models with datasets MNIST and CIFAR10. Compared to the state-of-the-art approaches, we demonstrate the speedup of Oliva for up to 25X in MNIST, and up to 80X in CIFAR10.

http://arxiv.org/abs/2505.23404
MEF: A Capability-Aware Multi-Encryption Framework for Evaluating Vulnerabilities in Black-Box Large Language Models. (12%)
Mingyu Yu; Wei Wang; Yanjie Wei; Sujuan Qin; Fei Gao; Wenmin Li
Recent advancements in adversarial jailbreak attacks have exposed critical vulnerabilities in Large Language Models (LLMs), enabling the circumvention of alignment safeguards through increasingly sophisticated prompt manipulations. Based on our experiments, we found that the effectiveness of jailbreak strategies is influenced by the comprehension ability of the attacked LLM. Building on this insight, we propose a capability-aware Multi-Encryption Framework (MEF) for evaluating vulnerabilities in black-box LLMs. Specifically, MEF first categorizes the comprehension ability level of the LLM, then applies different strategies accordingly: For models with limited comprehension ability, MEF adopts the Fu+En1 strategy, which integrates layered semantic mutations with an encryption technique, more effectively contributing to evasion of the LLM's defenses at the input and inference stages. For models with strong comprehension ability, MEF uses a more complex Fu+En1+En2 strategy, in which additional dual-ended encryption techniques are applied to the LLM's responses, further contributing to evasion of the LLM's defenses at the output stage. Experimental results demonstrate the effectiveness of our approach, achieving attack success rates of 98.9% on GPT-4o (29 May 2025 release) and 99.8% on GPT-4.1 (8 July 2025 release). Our work contributes to a deeper understanding of the vulnerabilities in current LLM alignment mechanisms.

http://arxiv.org/abs/2404.16417
Constructing Optimal Noise Channels for Enhanced Robustness in Quantum Machine Learning. (2%)
David Winderl; Nicola Franco; Jeanette Miriam Lorenz
With the rapid advancement of Quantum Machine Learning (QML), the critical need to enhance security measures against adversarial attacks and protect QML models becomes increasingly evident. In this work, we outline the connection between quantum noise channels and differential privacy (DP), by constructing a family of noise channels which are inherently $ε$-DP: $(α, γ)$-channels. Through this approach, we successfully replicate the $ε$-DP bounds observed for depolarizing and random rotation channels, thereby affirming the broad generality of our framework. Additionally, we use a semi-definite program to construct an optimally robust channel. In a small-scale experimental evaluation, we demonstrate the benefits of using our optimal noise channel over depolarizing noise, particularly in enhancing adversarial accuracy. Moreover, we assess how the variables $α$ and $γ$ affect the certifiable robustness and investigate how different encoding methods impact the classifier's robustness.

http://arxiv.org/abs/2507.17944
Evaluating the Performance of AI Text Detectors, Few-Shot and Chain-of-Thought Prompting Using DeepSeek Generated Text. (2%)
Hulayyil Alshammari; Praveen Rao
Large language models (LLMs) have rapidly transformed the creation of written materials. LLMs have led to questions about writing integrity, thereby driving the creation of artificial intelligence (AI) detection technologies. Adversarial attacks, such as standard and humanized paraphrasing, inhibit detectors' ability to detect machine-generated text. Previous studies have mainly focused on ChatGPT and other well-known LLMs and have shown varying accuracy across detectors. However, there is a clear gap in the literature about DeepSeek, a recently published LLM. Therefore, in this work, we investigate whether six generally accessible AI detection tools -- AI Text Classifier, Content Detector AI, Copyleaks, QuillBot, GPT-2, and GPTZero -- can consistently recognize text generated by DeepSeek. The detectors were exposed to the aforementioned adversarial attacks. We also considered DeepSeek as a detector by performing few-shot prompting and chain-of-thought reasoning (CoT) for classifying AI and human-written text. We collected 49 human-authored question-answer pairs from before the LLM era and generated matching responses using DeepSeek-v3, producing 49 AI-generated samples. Then, we applied adversarial techniques such as paraphrasing and humanizing to add 196 more samples. These were used to challenge detector robustness and assess accuracy impact. While QuillBot and Copyleaks showed near-perfect performance on original and paraphrased DeepSeek text, others -- particularly AI Text Classifier and GPT-2 -- showed inconsistent results. The most effective attack was humanization, reducing accuracy to 71% for Copyleaks, 58% for QuillBot, and 52% for GPTZero. Few-shot and CoT prompting showed high accuracy, with the best five-shot result misclassifying only one of 49 samples (AI recall 96%, human recall 100%).

http://arxiv.org/abs/2507.10054
Explicit Vulnerability Generation with LLMs: An Investigation Beyond Adversarial Attacks. (1%)
Emir Bosnak; Sahand Moslemi; Mayasah Lami; Anil Koyuncu
Large Language Models (LLMs) are increasingly used as code assistants, yet their behavior when explicitly asked to generate insecure code remains poorly understood. While prior research has focused on unintended vulnerabilities, this study examines a more direct threat: open-source LLMs generating vulnerable code when prompted. We propose a dual experimental design: (1) Dynamic Prompting, which systematically varies vulnerability type, user persona, and prompt phrasing across structured templates; and (2) Reverse Prompting, which derives natural-language prompts from real vulnerable code samples. We evaluate three open-source 7B-parameter models (Qwen2, Mistral, Gemma) using static analysis to assess both the presence and correctness of generated vulnerabilities. Our results show that all models frequently generate the requested vulnerabilities, though with significant performance differences. Gemma achieves the highest correctness for memory vulnerabilities under Dynamic Prompting (e.g., 98.6% for buffer overflows), while Qwen2 demonstrates the most balanced performance across all tasks. We find that professional personas (e.g., "DevOps Engineer") consistently elicit higher success rates than student personas, and that the effectiveness of direct versus indirect phrasing is inverted depending on the prompting strategy. Vulnerability reproduction accuracy follows a non-linear pattern with code complexity, peaking in a moderate range. Our findings expose how LLMs' reliance on pattern recall over semantic reasoning creates significant blind spots in their safety alignments, particularly for requests framed as plausible professional tasks.

http://arxiv.org/abs/2507.17161
Tabular Diffusion based Actionable Counterfactual Explanations for Network Intrusion Detection. (1%)
Vinura Galwaduge; Jagath Samarabandu
Modern network intrusion detection systems (NIDS) frequently utilize the predictive power of complex deep learning models. However, the "black-box" nature of such deep learning methods adds a layer of opaqueness that hinders the proper understanding of detection decisions, trust in the decisions and prevent timely countermeasures against such attacks. Explainable AI (XAI) methods provide a solution to this problem by providing insights into the causes of the predictions. The majority of the existing XAI methods provide explanations which are not convenient to convert into actionable countermeasures. In this work, we propose a novel diffusion-based counterfactual explanation framework that can provide actionable explanations for network intrusion attacks. We evaluated our proposed algorithm against several other publicly available counterfactual explanation algorithms on 3 modern network intrusion datasets. To the best of our knowledge, this work also presents the first comparative analysis of existing counterfactual explanation algorithms within the context of network intrusion detection systems. Our proposed method provide minimal, diverse counterfactual explanations out of the tested counterfactual explanation algorithms in a more efficient manner by reducing the time to generate explanations. We also demonstrate how counterfactual explanations can provide actionable explanations by summarizing them to create a set of global rules. These rules are actionable not only at instance level but also at the global level for intrusion attacks. These global counterfactual rules show the ability to effectively filter out incoming attack queries which is crucial for efficient intrusion detection and defense mechanisms.

http://arxiv.org/abs/2507.17834
Smoothed Analysis of Online Metric Problems. (1%)
Christian Coester; Jack Umenberger
We study three classical online problems -- $k$-server, $k$-taxi, and chasing size $k$ sets -- through a lens of smoothed analysis. Our setting allows request locations to be adversarial up to small perturbations, interpolating between worst-case and average-case models. Specifically, we show that if the metric space is contained in a ball in any normed space and requests are drawn from distributions whose density functions are upper bounded by $1/σ$ times the uniform density over the ball, then all three problems admit polylog$(k/σ)$-competitive algorithms. Our approach is simple: it reduces smoothed instances to fully adversarial instances on finite metrics and leverages existing algorithms in a black-box manner. We also provide a lower bound showing that no algorithm can achieve a competitive ratio sub-polylogarithmic in $k/σ$, matching our upper bounds up to the exponent of the polylogarithm. In contrast, the best known competitive ratios for these problems in the fully adversarial setting are $2k-1$, $\infty$ and $Θ(k^2)$, respectively.

http://arxiv.org/abs/2507.16257
Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models. (99%)
Futa Waseda; Saku Sugawara; Isao Echizen
Defending pre-trained vision-language models (VLMs), such as CLIP, against adversarial attacks is crucial, as these models are widely used in diverse zero-shot tasks, including image classification. However, existing adversarial training (AT) methods for robust fine-tuning largely overlook the role of language in enhancing visual robustness. Specifically, (1) supervised AT methods rely on short texts (e.g., class labels) to generate adversarial perturbations, leading to overfitting to object classes in the training data, and (2) unsupervised AT avoids this overfitting but remains suboptimal against practical text-guided adversarial attacks due to its lack of semantic guidance. To address these limitations, we propose Quality Text-guided Adversarial Fine-Tuning (QT-AFT), which leverages high-quality captions during training to guide adversarial examples away from diverse semantics present in images. This enables the visual encoder to robustly recognize a broader range of image features even under adversarial noise, thereby enhancing robustness across diverse downstream tasks. QT-AFT overcomes the key weaknesses of prior methods -- overfitting in supervised AT and lack of semantic awareness in unsupervised AT -- achieving state-of-the-art zero-shot adversarial robustness and clean accuracy, evaluated across 16 zero-shot datasets. Furthermore, our comprehensive study uncovers several key insights into the role of language in enhancing vision robustness; for example, describing object properties in addition to object names further enhances zero-shot robustness. Our findings point to an urgent direction for future work -- centering high-quality linguistic supervision in robust visual representation learning.

http://arxiv.org/abs/2507.16164
Attacking interpretable NLP systems. (99%)
Eldor Abdukhamidov; Tamer Abuhmed; Joanna C. S. Santos; Mohammed Abuhamad
Studies have shown that machine learning systems are vulnerable to adversarial examples in theory and practice. Where previous attacks have focused mainly on visual models that exploit the difference between human and machine perception, text-based models have also fallen victim to these attacks. However, these attacks often fail to maintain the semantic meaning of the text and similarity. This paper introduces AdvChar, a black-box attack on Interpretable Natural Language Processing Systems, designed to mislead the classifier while keeping the interpretation similar to benign inputs, thus exploiting trust in system transparency. AdvChar achieves this by making less noticeable modifications to text input, forcing the deep learning classifier to make incorrect predictions and preserve the original interpretation. We use an interpretation-focused scoring approach to determine the most critical tokens that, when changed, can cause the classifier to misclassify the input. We apply simple character-level modifications to measure the importance of tokens, minimizing the difference between the original and new text while generating adversarial interpretations similar to benign ones. We thoroughly evaluated AdvChar by testing it against seven NLP models and three interpretation models using benchmark datasets for the classification task. Our experiments show that AdvChar can significantly reduce the prediction accuracy of current deep learning models by altering just two characters on average in input samples.

http://arxiv.org/abs/2507.17070
Advancing Robustness in Deep Reinforcement Learning with an Ensemble Defense Approach. (92%)
Adithya Mohan; Dominik Rößle; Daniel Cremers; Torsten Schön
Recent advancements in Deep Reinforcement Learning (DRL) have demonstrated its applicability across various domains, including robotics, healthcare, energy optimization, and autonomous driving. However, a critical question remains: How robust are DRL models when exposed to adversarial attacks? While existing defense mechanisms such as adversarial training and distillation enhance the resilience of DRL models, there remains a significant research gap regarding the integration of multiple defenses in autonomous driving scenarios specifically. This paper addresses this gap by proposing a novel ensemble-based defense architecture to mitigate adversarial attacks in autonomous driving. Our evaluation demonstrates that the proposed architecture significantly enhances the robustness of DRL models. Compared to the baseline under FGSM attacks, our ensemble method improves the mean reward from 5.87 to 18.38 (over 213% increase) and reduces the mean collision rate from 0.50 to 0.09 (an 82% decrease) in the highway scenario and merge scenario, outperforming all standalone defense strategies.

http://arxiv.org/abs/2507.16291
Talking Like a Phisher: LLM-Based Attacks on Voice Phishing Classifiers. (86%)
Wenhao Li; Selvakumar Manickam; Yung-wey Chong; Shankar Karuppayah
Voice phishing (vishing) remains a persistent threat in cybersecurity, exploiting human trust through persuasive speech. While machine learning (ML)-based classifiers have shown promise in detecting malicious call transcripts, they remain vulnerable to adversarial manipulations that preserve semantic content. In this study, we explore a novel attack vector where large language models (LLMs) are leveraged to generate adversarial vishing transcripts that evade detection while maintaining deceptive intent. We construct a systematic attack pipeline that employs prompt engineering and semantic obfuscation to transform real-world vishing scripts using four commercial LLMs. The generated transcripts are evaluated against multiple ML classifiers trained on a real-world Korean vishing dataset (KorCCViD) with statistical testing. Our experiments reveal that LLM-generated transcripts are both practically and statistically effective against ML-based classifiers. In particular, transcripts crafted by GPT-4o significantly reduce classifier accuracy (by up to 30.96%) while maintaining high semantic similarity, as measured by BERTScore. Moreover, these attacks are both time-efficient and cost-effective, with average generation times under 9 seconds and negligible financial cost per query. The results underscore the pressing need for more resilient vishing detection frameworks and highlight the imperative for LLM providers to enforce stronger safeguards against prompt misuse in adversarial social engineering contexts.

http://arxiv.org/abs/2507.16345
The Cost of Compression: Tight Quadratic Black-Box Attacks on Sketches for $\ell_2$ Norm Estimation. (82%)
Sara Ahmadian; Edith Cohen; Uri Stemmer
Dimensionality reduction via linear sketching is a powerful and widely used technique, but it is known to be vulnerable to adversarial inputs. We study the black-box adversarial setting, where a fixed, hidden sketching matrix A in $R^{k X n}$ maps high-dimensional vectors v $\in R^n$ to lower-dimensional sketches A v in $R^k$, and an adversary can query the system to obtain approximate ell2-norm estimates that are computed from the sketch.
  We present a universal, nonadaptive attack that, using tilde(O)($k^2$) queries, either causes a failure in norm estimation or constructs an adversarial input on which the optimal estimator for the query distribution (used by the attack) fails. The attack is completely agnostic to the sketching matrix and to the estimator: It applies to any linear sketch and any query responder, including those that are randomized, adaptive, or tailored to the query distribution.
  Our lower bound construction tightly matches the known upper bounds of tilde(Omega)($k^2$), achieved by specialized estimators for Johnson Lindenstrauss transforms and AMS sketches. Beyond sketching, our results uncover structural parallels to adversarial attacks in image classification, highlighting fundamental vulnerabilities of compressed representations.

http://arxiv.org/abs/2507.16372
Depth Gives a False Sense of Privacy: LLM Internal States Inversion. (68%)
Tian Dong; Yan Meng; Shaofeng Li; Guoxing Chen; Zhen Liu; Haojin Zhu
Large Language Models (LLMs) are increasingly integrated into daily routines, yet they raise significant privacy and safety concerns. Recent research proposes collaborative inference, which outsources the early-layer inference to ensure data locality, and introduces model safety auditing based on inner neuron patterns. Both techniques expose the LLM's Internal States (ISs), which are traditionally considered irreversible to inputs due to optimization challenges and the highly abstract representations in deep layers. In this work, we challenge this assumption by proposing four inversion attacks that significantly improve the semantic similarity and token matching rate of inverted inputs. Specifically, we first develop two white-box optimization-based attacks tailored for low-depth and high-depth ISs. These attacks avoid local minima convergence, a limitation observed in prior work, through a two-phase inversion process. Then, we extend our optimization attack under more practical black-box weight access by leveraging the transferability between the source and the derived LLMs. Additionally, we introduce a generation-based attack that treats inversion as a translation task, employing an inversion model to reconstruct inputs. Extensive evaluation of short and long prompts from medical consulting and coding assistance datasets and 6 LLMs validates the effectiveness of our inversion attacks. Notably, a 4,112-token long medical consulting prompt can be nearly perfectly inverted with 86.88 F1 token matching from the middle layer of Llama-3 model. Finally, we evaluate four practical defenses that we found cannot perfectly prevent ISs inversion and draw conclusions for future mitigation design.

http://arxiv.org/abs/2507.18656
ShrinkBox: Backdoor Attack on Object Detection to Disrupt Collision Avoidance in Machine Learning-based Advanced Driver Assistance Systems. (45%)
Muhammad Zaeem Shahzad; Muhammad Abdullah Hanif; Bassem Ouni; Muhammad Shafique
Advanced Driver Assistance Systems (ADAS) significantly enhance road safety by detecting potential collisions and alerting drivers. However, their reliance on expensive sensor technologies such as LiDAR and radar limits accessibility, particularly in low- and middle-income countries. Machine learning-based ADAS (ML-ADAS), leveraging deep neural networks (DNNs) with only standard camera input, offers a cost-effective alternative. Critical to ML-ADAS is the collision avoidance feature, which requires the ability to detect objects and estimate their distances accurately. This is achieved with specialized DNNs like YOLO, which provides real-time object detection, and a lightweight, detection-wise distance estimation approach that relies on key features extracted from the detections like bounding box dimensions and size. However, the robustness of these systems is undermined by security vulnerabilities in object detectors. In this paper, we introduce ShrinkBox, a novel backdoor attack targeting object detection in collision avoidance ML-ADAS. Unlike existing attacks that manipulate object class labels or presence, ShrinkBox subtly shrinks ground truth bounding boxes. This attack remains undetected in dataset inspections and standard benchmarks while severely disrupting downstream distance estimation. We demonstrate that ShrinkBox can be realized in the YOLOv9m object detector at an Attack Success Rate (ASR) of 96%, with only a 4% poisoning ratio in the training instances of the KITTI dataset. Furthermore, given the low error targets introduced in our relaxed poisoning strategy, we find that ShrinkBox increases the Mean Absolute Error (MAE) in downstream distance estimation by more than 3x on poisoned samples, potentially resulting in delays or prevention of collision warnings altogether.

http://arxiv.org/abs/2507.16969
LLM4MEA: Data-free Model Extraction Attacks on Sequential Recommenders via Large Language Models. (31%)
Shilong Zhao; Fei Sun; Kaike Zhang; Shaoling Jing; Du Su; Zhichao Shi; Zhiyi Yin; Huawei Shen; Xueqi Cheng
Recent studies have demonstrated the vulnerability of sequential recommender systems to Model Extraction Attacks (MEAs). MEAs collect responses from recommender systems to replicate their functionality, enabling unauthorized deployments and posing critical privacy and security risks. Black-box attacks in prior MEAs are ineffective at exposing recommender system vulnerabilities due to random sampling in data selection, which leads to misaligned synthetic and real-world distributions. To overcome this limitation, we propose LLM4MEA, a novel model extraction method that leverages Large Language Models (LLMs) as human-like rankers to generate data. It generates data through interactions between the LLM ranker and target recommender system. In each interaction, the LLM ranker analyzes historical interactions to understand user behavior, and selects items from recommendations with consistent preferences to extend the interaction history, which serves as training data for MEA. Extensive experiments demonstrate that LLM4MEA significantly outperforms existing approaches in data quality and attack performance, reducing the divergence between synthetic and real-world data by up to 64.98% and improving MEA performance by 44.82% on average. From a defensive perspective, we propose a simple yet effective defense strategy and identify key hyperparameters of recommender systems that can mitigate the risk of MEAs.

http://arxiv.org/abs/2507.16134
DP2Guard: A Lightweight and Byzantine-Robust Privacy-Preserving Federated Learning Scheme for Industrial IoT. (26%)
Baofu Han; Bing Li; Yining Qi; Raja Jurdak; Kaibin Huang; Chau Yuen
Privacy-Preserving Federated Learning (PPFL) has emerged as a secure distributed Machine Learning (ML) paradigm that aggregates locally trained gradients without exposing raw data. To defend against model poisoning threats, several robustness-enhanced PPFL schemes have been proposed by integrating anomaly detection. Nevertheless, they still face two major challenges: (1) the reliance on heavyweight encryption techniques results in substantial communication and computation overhead; and (2) single-strategy defense mechanisms often fail to provide sufficient robustness against adaptive adversaries. To overcome these challenges, we propose DP2Guard, a lightweight PPFL framework that enhances both privacy and robustness. DP2Guard leverages a lightweight gradient masking mechanism to replace costly cryptographic operations while ensuring the privacy of local gradients. A hybrid defense strategy is proposed, which extracts gradient features using singular value decomposition and cosine similarity, and applies a clustering algorithm to effectively identify malicious gradients. Additionally, DP2Guard adopts a trust score-based adaptive aggregation scheme that adjusts client weights according to historical behavior, while blockchain records aggregated results and trust scores to ensure tamper-proof and auditable training. Extensive experiments conducted on two public datasets demonstrate that DP2Guard effectively defends against four advanced poisoning attacks while ensuring privacy with reduced communication and computation costs.

http://arxiv.org/abs/2508.08262
Argument Quality Annotation and Gender Bias Detection in Financial Communication through Large Language Models. (15%)
Alaa Alhamzeh; Mays Al Rebdawi
Financial arguments play a critical role in shaping investment decisions and public trust in financial institutions. Nevertheless, assessing their quality remains poorly studied in the literature. In this paper, we examine the capabilities of three state-of-the-art LLMs GPT-4o, Llama 3.1, and Gemma 2 in annotating argument quality within financial communications, using the FinArgQuality dataset. Our contributions are twofold. First, we evaluate the consistency of LLM-generated annotations across multiple runs and benchmark them against human annotations. Second, we introduce an adversarial attack designed to inject gender bias to analyse models responds and ensure model's fairness and robustness. Both experiments are conducted across three temperature settings to assess their influence on annotation stability and alignment with human labels. Our findings reveal that LLM-based annotations achieve higher inter-annotator agreement than human counterparts, though the models still exhibit varying degrees of gender bias. We provide a multifaceted analysis of these outcomes and offer practical recommendations to guide future research toward more reliable, cost-effective, and bias-aware annotation methodologies.

http://arxiv.org/abs/2507.16773
When LLMs Copy to Think: Uncovering Copy-Guided Attacks in Reasoning LLMs. (3%)
Yue Li; Xiao Li; Hao Wu; Yue Zhang; Fengyuan Xu; Xiuzhen Cheng; Sheng Zhong
Large Language Models (LLMs) have become integral to automated code analysis, enabling tasks such as vulnerability detection and code comprehension. However, their integration introduces novel attack surfaces. In this paper, we identify and investigate a new class of prompt-based attacks, termed Copy-Guided Attacks (CGA), which exploit the inherent copying tendencies of reasoning-capable LLMs. By injecting carefully crafted triggers into external code snippets, adversaries can induce the model to replicate malicious content during inference. This behavior enables two classes of vulnerabilities: inference length manipulation, where the model generates abnormally short or excessively long reasoning traces; and inference result manipulation, where the model produces misleading or incorrect conclusions. We formalize CGA as an optimization problem and propose a gradient-based approach to synthesize effective triggers. Empirical evaluation on state-of-the-art reasoning LLMs shows that CGA reliably induces infinite loops, premature termination, false refusals, and semantic distortions in code analysis tasks. While highly effective in targeted settings, we observe challenges in generalizing CGA across diverse prompts due to computational constraints, posing an open question for future research. Our findings expose a critical yet underexplored vulnerability in LLM-powered development pipelines and call for urgent advances in prompt-level defense mechanisms.

http://arxiv.org/abs/2507.16393
Are Foundation Models All You Need for Zero-shot Face Presentation Attack Detection? (1%)
Lazaro Janier Gonzalez-Sole; Juan E. Tapia; Christoph Busch
Although face recognition systems have undergone an impressive evolution in the last decade, these technologies are vulnerable to attack presentations (AP). These attacks are mostly easy to create and, by executing them against the system's capture device, the malicious actor can impersonate an authorised subject and thus gain access to the latter's information (e.g., financial transactions). To protect facial recognition schemes against presentation attacks, state-of-the-art deep learning presentation attack detection (PAD) approaches require a large amount of data to produce reliable detection performances and even then, they decrease their performance for unknown presentation attack instruments (PAI) or database (information not seen during training), i.e. they lack generalisability. To mitigate the above problems, this paper focuses on zero-shot PAD. To do so, we first assess the effectiveness and generalisability of foundation models in established and challenging experimental scenarios and then propose a simple but effective framework for zero-shot PAD. Experimental results show that these models are able to achieve performance in difficult scenarios with minimal effort of the more advanced PAD mechanisms, whose weights were optimised mainly with training sets that included APs and bona fide presentations. The top-performing foundation model outperforms by a margin the best from the state of the art observed with the leaving-one-out protocol on the SiW-Mv2 database, which contains challenging unknown 2D and 3D attacks

http://arxiv.org/abs/2507.16052
Disrupting Semantic and Abstract Features for Better Adversarial Transferability. (96%)
Yuyang Luo; Xiaosen Wang; Zhijin Ge; Yingzhe He
Adversarial examples pose significant threats to deep neural networks (DNNs), and their property of transferability in the black-box setting has led to the emergence of transfer-based attacks, making it feasible to target real-world applications employing DNNs. Among them, feature-level attacks, where intermediate features are perturbed based on feature importance weight matrix computed from transformed images, have gained popularity. In this work, we find that existing feature-level attacks primarily manipulate the semantic information to derive the weight matrix. Inspired by several works that find CNNs tend to focus more on high-frequency components (a.k.a. abstract features, e.g., texture, edge, etc.), we validate that transforming images in the high-frequency space also improves transferability. Based on this finding, we propose a balanced approach called Semantic and Abstract FEatures disRuption (SAFER). Specifically, SAFER conducts BLOCKMIX on the input image and SELF-MIX on the frequency spectrum when computing the weight matrix to highlight crucial features. By using such a weight matrix, we can direct the attacker to disrupt both semantic and abstract features, leading to improved transferability. Extensive experiments on the ImageNet dataset also demonstrate the effectiveness of our method in boosting adversarial transferability.

http://arxiv.org/abs/2507.15974
Does More Inference-Time Compute Really Help Robustness? (76%)
Tong Wu; Chong Xiang; Jiachen T. Wang; Weichen Yu; Chawin Sitawarin; Vikash Sehwag; Prateek Mittal
Recently, Zaremba et al. demonstrated that increasing inference-time computation improves robustness in large proprietary reasoning LLMs. In this paper, we first show that smaller-scale, open-source models (e.g., DeepSeek R1, Qwen3, Phi-reasoning) can also benefit from inference-time scaling using a simple budget forcing strategy. More importantly, we reveal and critically examine an implicit assumption in prior work: intermediate reasoning steps are hidden from adversaries. By relaxing this assumption, we identify an important security risk, intuitively motivated and empirically verified as an inverse scaling law: if intermediate reasoning steps become explicitly accessible, increased inference-time computation consistently reduces model robustness. Finally, we discuss practical scenarios where models with hidden reasoning chains are still vulnerable to attacks, such as models with tool-integrated reasoning and advanced reasoning extraction attacks. Our findings collectively demonstrate that the robustness benefits of inference-time scaling depend heavily on the adversarial setting and deployment context. We urge practitioners to carefully weigh these subtle trade-offs before applying inference-time scaling in security-sensitive, real-world applications.

http://arxiv.org/abs/2507.15349
Scaling Decentralized Learning with FLock. (76%)
Zehua Cheng; Rui Sun; Jiahao Sun; Yike Guo
Fine-tuning the large language models (LLMs) are prevented by the deficiency of centralized control and the massive computing and communication overhead on the decentralized schemes. While the typical standard federated learning (FL) supports data privacy, the central server requirement creates a single point of attack and vulnerability to poisoning attacks. Generalizing the result in this direction to 70B-parameter models in the heterogeneous, trustless environments has turned out to be a huge, yet unbroken bottleneck. This paper introduces FLock, a decentralized framework for secure and efficient collaborative LLM fine-tuning. Integrating a blockchain-based trust layer with economic incentives, FLock replaces the central aggregator with a secure, auditable protocol for cooperation among untrusted parties. We present the first empirical validation of fine-tuning a 70B LLM in a secure, multi-domain, decentralized setting. Our experiments show the FLock framework defends against backdoor poisoning attacks that compromise standard FL optimizers and fosters synergistic knowledge transfer. The resulting models show a >68% reduction in adversarial attack success rates. The global model also demonstrates superior cross-domain generalization, outperforming models trained in isolation on their own specialized data.

http://arxiv.org/abs/2507.15219
PromptArmor: Simple yet Effective Prompt Injection Defenses. (73%)
Tianneng Shi; Kaijie Zhu; Zhun Wang; Yuqi Jia; Will Cai; Weida Liang; Haonan Wang; Hend Alzahrani; Joshua Lu; Kenji Kawaguchi; Basel Alomair; Xuandong Zhao; William Yang Wang; Neil Gong; Wenbo Guo; Dawn Song
Despite their potential, recent research has demonstrated that LLM agents are vulnerable to prompt injection attacks, where malicious prompts are injected into the agent's input, causing it to perform an attacker-specified task rather than the intended task provided by the user. In this paper, we present PromptArmor, a simple yet effective defense against prompt injection attacks. Specifically, PromptArmor prompts an off-the-shelf LLM to detect and remove potential injected prompts from the input before the agent processes it. Our results show that PromptArmor can accurately identify and remove injected prompts. For example, using GPT-4o, GPT-4.1, or o4-mini, PromptArmor achieves both a false positive rate and a false negative rate below 1% on the AgentDojo benchmark. Moreover, after removing injected prompts with PromptArmor, the attack success rate drops to below 1%. We also demonstrate PromptArmor's effectiveness against adaptive attacks and explore different strategies for prompting an LLM. We recommend that PromptArmor be adopted as a standard baseline for evaluating new defenses against prompt injection attacks.

http://arxiv.org/abs/2507.15613
Multi-Stage Prompt Inference Attacks on Enterprise LLM Systems. (67%)
Andrii Balashov; Olena Ponomarova; Xiaohua Zhai
Large Language Models (LLMs) deployed in enterprise settings (e.g., as Microsoft 365 Copilot) face novel security challenges. One critical threat is prompt inference attacks: adversaries chain together seemingly benign prompts to gradually extract confidential data. In this paper, we present a comprehensive study of multi-stage prompt inference attacks in an enterprise LLM context. We simulate realistic attack scenarios where an attacker uses mild-mannered queries and indirect prompt injections to exploit an LLM integrated with private corporate data. We develop a formal threat model for these multi-turn inference attacks and analyze them using probability theory, optimization frameworks, and information-theoretic leakage bounds. The attacks are shown to reliably exfiltrate sensitive information from the LLM's context (e.g., internal SharePoint documents or emails), even when standard safety measures are in place.
  We propose and evaluate defenses to counter such attacks, including statistical anomaly detection, fine-grained access control, prompt sanitization techniques, and architectural modifications to LLM deployment. Each defense is supported by mathematical analysis or experimental simulation. For example, we derive bounds on information leakage under differential privacy-based training and demonstrate an anomaly detection method that flags multi-turn attacks with high AUC. We also introduce an approach called "spotlighting" that uses input transformations to isolate untrusted prompt content, reducing attack success by an order of magnitude. Finally, we provide a formal proof of concept and empirical validation for a combined defense-in-depth strategy. Our work highlights that securing LLMs in enterprise settings requires moving beyond single-turn prompt filtering toward a holistic, multi-stage perspective on both attacks and defenses.

http://arxiv.org/abs/2411.15265
Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI. (45%)
Won Jun Kim; Hyungjin Chung; Jaemin Kim; Sangmin Lee; Byeongsu Sim; Jong Chul Ye
Gradient-based methods are a prototypical family of explainability techniques, especially for image-based models. Nonetheless, they have several shortcomings in that they (1) require white-box access to models, (2) are vulnerable to adversarial attacks, and (3) produce attributions that lie off the image manifold, leading to explanations that are not actually faithful to the model and do not align well with human perception. To overcome these challenges, we introduce Derivative-Free Diffusion Manifold-Constrainted Gradients (FreeMCG), a novel method that serves as an improved basis for explainability of a given neural network than the traditional gradient. Specifically, by leveraging ensemble Kalman filters and diffusion models, we derive a derivative-free approximation of the model's gradient projected onto the data manifold, requiring access only to the model's outputs. We demonstrate the effectiveness of FreeMCG by applying it to both counterfactual generation and feature attribution, which have traditionally been treated as distinct tasks. Through comprehensive evaluation on both tasks, counterfactual explanation and feature attribution, we show that our method yields state-of-the-art results while preserving the essential properties expected of XAI tools.

http://arxiv.org/abs/2505.01454
Sparsification Under Siege: Defending Against Poisoning Attacks in Communication-Efficient Federated Learning. (15%)
Zhiyong Jin; Runhua Xu; Chao Li; Yizhong Liu; Jianxin Li; James Joshi
Federated Learning (FL) enables collaborative model training across distributed clients while preserving data privacy, yet it faces significant challenges in communication efficiency and vulnerability to poisoning attacks. While sparsification techniques mitigate communication overhead by transmitting only critical model parameters, they inadvertently amplify security risks: adversarial clients can exploit sparse updates to evade detection and degrade model performance. Existing defense mechanisms, designed for standard FL communication scenarios, are ineffective in addressing these vulnerabilities within sparsified FL. To bridge this gap, we propose FLARE, a novel federated learning framework that integrates sparse index mask inspection and model update sign similarity analysis to detect and mitigate poisoning attacks in sparsified FL. Extensive experiments across multiple datasets and adversarial scenarios demonstrate that FLARE significantly outperforms existing defense strategies, effectively securing sparsified FL against poisoning attacks while maintaining communication efficiency.

http://arxiv.org/abs/2503.01781
Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models. (13%)
Meghana Rajeev; Rajkumar Ramamurthy; Prapti Trivedi; Vikas Yadav; Oluwanifemi Bamgbose; Sathwik Tejaswi Madhusudan; James Zou; Nazneen Rajani
We investigate the robustness of reasoning models trained for step-by-step problem solving by introducing query-agnostic adversarial triggers - short, irrelevant text that, when appended to math problems, systematically mislead models to output incorrect answers without altering the problem's semantics. We propose CatAttack, an automated iterative attack pipeline for generating triggers on a weaker, less expensive proxy model (DeepSeek V3) and successfully transfer them to more advanced reasoning target models like DeepSeek R1 and DeepSeek R1-distilled-Qwen-32B, resulting in greater than 300% increase in the likelihood of the target model generating an incorrect answer. For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong. Our findings highlight critical vulnerabilities in reasoning models, revealing that even state-of-the-art models remain susceptible to subtle adversarial inputs, raising security and reliability concerns. The CatAttack triggers dataset with model responses is available at https://huggingface.co/datasets/collinear-ai/cat-attack-adversarial-triggers.

http://arxiv.org/abs/2507.15285
In-context Learning of Vision Language Models for Detection of Physical and Digital Attacks against Face Recognition Systems. (10%)
Lazaro Janier Gonzalez-Soler; Maciej Salwowski; Christoph Busch
Recent advances in biometric systems have significantly improved the detection and prevention of fraudulent activities. However, as detection methods improve, attack techniques become increasingly sophisticated. Attacks on face recognition systems can be broadly divided into physical and digital approaches. Traditionally, deep learning models have been the primary defence against such attacks. While these models perform exceptionally well in scenarios for which they have been trained, they often struggle to adapt to different types of attacks or varying environmental conditions. These subsystems require substantial amounts of training data to achieve reliable performance, yet biometric data collection faces significant challenges, including privacy concerns and the logistical difficulties of capturing diverse attack scenarios under controlled conditions. This work investigates the application of Vision Language Models (VLM) and proposes an in-context learning framework for detecting physical presentation attacks and digital morphing attacks in biometric systems. Focusing on open-source models, the first systematic framework for the quantitative evaluation of VLMs in security-critical scenarios through in-context learning techniques is established. The experimental evaluation conducted on freely available databases demonstrates that the proposed subsystem achieves competitive performance for physical and digital attack detection, outperforming some of the traditional CNNs without resource-intensive training. The experimental results validate the proposed framework as a promising tool for improving generalisation in attack detection.

http://arxiv.org/abs/2502.11798
BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model. (4%)
Weilin Lin; Nanjun Zhou; Yanyun Wang; Jianze Li; Hui Xiong; Li Liu
Backdoor learning is a critical research topic for understanding the vulnerabilities of deep neural networks. While the diffusion model (DM) has been broadly deployed in public over the past few years, the understanding of its backdoor vulnerability is still in its infancy compared to the extensive studies in discriminative models. Recently, many different backdoor attack and defense methods have been proposed for DMs, but a comprehensive benchmark for backdoor learning on DMs is still lacking. This absence makes it difficult to conduct fair comparisons and thorough evaluations of the existing approaches, thus hindering future research progress. To address this issue, we propose \textit{BackdoorDM}, the first comprehensive benchmark designed for backdoor learning on DMs. It comprises nine state-of-the-art (SOTA) attack methods, four SOTA defense strategies, and three useful visualization analysis tools. We first systematically classify and formulate the existing literature in a unified framework, focusing on three different backdoor attack types and five backdoor target types, which are restricted to a single type in discriminative models. Then, we systematically summarize the evaluation metrics for each type and propose a unified backdoor evaluation method based on multimodal large language model (MLLM). Finally, we conduct a comprehensive evaluation and highlight several important conclusions. We believe that BackdoorDM will help overcome current barriers and contribute to building a trustworthy artificial intelligence generated content (AIGC) community. The codes are released in https://github.com/linweiii/BackdoorDM.

http://arxiv.org/abs/1911.08432
Defective Convolutional Networks.
Tiange Luo; Tianle Cai; Mengxiao Zhang; Siyu Chen; Di He; Liwei Wang
Robustness of convolutional neural networks (CNNs) has gained in importance on account of adversarial examples, i.e., inputs added as well-designed perturbations that are imperceptible to humans but can cause the model to predict incorrectly. Recent research suggests that the noises in adversarial examples break the textural structure, which eventually leads to wrong predictions. To mitigate the threat of such adversarial attacks, we propose defective convolutional networks that make predictions relying less on textural information but more on shape information by properly integrating defective convolutional layers into standard CNNs. The defective convolutional layers contain defective neurons whose activations are set to be a constant function. As defective neurons contain no information and are far different from standard neurons in its spatial neighborhood, the textural features cannot be accurately extracted, and so the model has to seek other features for classification, such as the shape. We show extensive evidence to justify our proposal and demonstrate that defective CNNs can defense against black-box attacks better than standard CNNs. In particular, they achieve state-of-the-art performance against transfer-based attacks without any adversarial training being applied.

http://arxiv.org/abs/2507.14863
Adversarial Destabilization Attacks to Direct Data-Driven Control. (99%)
Hampei Sasahara
This study investigates the vulnerability of direct data-driven control methods, specifically for the linear quadratic regulator problem, to adversarial perturbations in collected data used for controller synthesis. We consider stealthy attacks that subtly manipulate offline-collected data to destabilize the resulting closed-loop system while evading detection. To generate such perturbations, we propose the Directed Gradient Sign Method (DGSM) and its iterative variant (I-DGSM), adaptations of the fast gradient sign method originally developed for neural networks, which align perturbations with the gradient of the spectral radius of the closed-loop matrix to reduce stability. A key contribution is an efficient gradient computation technique based on implicit differentiation through the Karush-Kuhn-Tucker conditions of the underlying semidefinite program, enabling scalable and exact gradient evaluation without repeated optimization computations. To defend against these attacks, we propose two defense strategies: a regularization-based approach that enhances robustness by suppressing controller sensitivity to data perturbations and a robust data-driven control approach that guarantees closed-loop stability within bounded perturbation sets. Extensive numerical experiments on benchmark systems show that adversarial perturbations with magnitudes up to ten times smaller than random noise can destabilize controllers trained on corrupted data and that the proposed defense strategies effectively mitigate attack success rates while maintaining control performance. Additionally, we evaluate attack transferability under partial knowledge scenarios, highlighting the practical importance of protecting training data confidentiality.

http://arxiv.org/abs/2507.15067
ROBAD: Robust Adversary-aware Local-Global Attended Bad Actor Detection Sequential Model. (73%)
Bing He; Mustaque Ahamad; Srijan Kumar
Detecting bad actors is critical to ensure the safety and integrity of internet platforms. Several deep learning-based models have been developed to identify such users. These models should not only accurately detect bad actors, but also be robust against adversarial attacks that aim to evade detection. However, past deep learning-based detection models do not meet the robustness requirement because they are sensitive to even minor changes in the input sequence. To address this issue, we focus on (1) improving the model understanding capability and (2) enhancing the model knowledge such that the model can recognize potential input modifications when making predictions. To achieve these goals, we create a novel transformer-based classification model, called ROBAD (RObust adversary-aware local-global attended Bad Actor Detection model), which uses the sequence of user posts to generate user embedding to detect bad actors. Particularly, ROBAD first leverages the transformer encoder block to encode each post bidirectionally, thus building a post embedding to capture the local information at the post level. Next, it adopts the transformer decoder block to model the sequential pattern in the post embeddings by using the attention mechanism, which generates the sequence embedding to obtain the global information at the sequence level. Finally, to enrich the knowledge of the model, embeddings of modified sequences by mimicked attackers are fed into a contrastive-learning-enhanced classification layer for sequence prediction. In essence, by capturing the local and global information (i.e., the post and sequence information) and leveraging the mimicked behaviors of bad actors in training, ROBAD can be robust to adversarial attacks. Extensive experiments on Yelp and Wikipedia datasets show that ROBAD can effectively detect bad actors when under state-of-the-art adversarial attacks.

http://arxiv.org/abs/2507.15042
DeRAG: Black-box Adversarial Attacks on Multiple Retrieval-Augmented Generation Applications via Prompt Injection. (64%)
Jerry Wang; Fang Yu
Adversarial prompt attacks can significantly alter the reliability of Retrieval-Augmented Generation (RAG) systems by re-ranking them to produce incorrect outputs. In this paper, we present a novel method that applies Differential Evolution (DE) to optimize adversarial prompt suffixes for RAG-based question answering. Our approach is gradient-free, treating the RAG pipeline as a black box and evolving a population of candidate suffixes to maximize the retrieval rank of a targeted incorrect document to be closer to real world scenarios. We conducted experiments on the BEIR QA datasets to evaluate attack success at certain retrieval rank thresholds under multiple retrieving applications. Our results demonstrate that DE-based prompt optimization attains competitive (and in some cases higher) success rates compared to GGPP to dense retrievers and PRADA to sparse retrievers, while using only a small number of tokens (<=5 tokens) in the adversarial suffix. Furthermore, we introduce a readability-aware suffix construction strategy, validated by a statistically significant reduction in MLM negative log-likelihood with Welch's t-test. Through evaluations with a BERT-based adversarial suffix detector, we show that DE-generated suffixes evade detection, yielding near-chance detection accuracy.

http://arxiv.org/abs/2507.14799
Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree. (13%)
Sam Johnson; Viet Pham; Thai Le
This work demonstrates that LLM-based web navigation agents offer powerful automation capabilities but are vulnerable to Indirect Prompt Injection (IPI) attacks. We show that adversaries can embed universal adversarial triggers in webpage HTML to hijack agent behavior that utilizes the accessibility tree to parse HTML, causing unintended or malicious actions. Using the Greedy Coordinate Gradient (GCG) algorithm and a Browser Gym agent powered by Llama-3.1, our system demonstrates high success rates across real websites in both targeted and general attacks, including login credential exfiltration and forced ad clicks. Our empirical results highlight critical security risks and the need for stronger defenses as LLM-driven autonomous web agents become more widely adopted. The system software (https://github.com/sej2020/manipulating-web-agents) is released under the MIT License, with an accompanying publicly available demo website (http://lethaiq.github.io/attack-web-llm-agent).

http://arxiv.org/abs/2507.14625
VTarbel: Targeted Label Attack with Minimal Knowledge on Detector-enhanced Vertical Federated Learning. (92%)
Juntao Tan; Anran Li; Quanchao Liu; Peng Ran; Lan Zhang
Vertical federated learning (VFL) enables multiple parties with disjoint features to collaboratively train models without sharing raw data. While privacy vulnerabilities of VFL are extensively-studied, its security threats-particularly targeted label attacks-remain underexplored. In such attacks, a passive party perturbs inputs at inference to force misclassification into adversary-chosen labels. Existing methods rely on unrealistic assumptions (e.g., accessing VFL-model's outputs) and ignore anomaly detectors deployed in real-world systems. To bridge this gap, we introduce VTarbel, a two-stage, minimal-knowledge attack framework explicitly designed to evade detector-enhanced VFL inference. During the preparation stage, the attacker selects a minimal set of high-expressiveness samples (via maximum mean discrepancy), submits them through VFL protocol to collect predicted labels, and uses these pseudo-labels to train estimated detector and surrogate model on local features. In attack stage, these models guide gradient-based perturbations of remaining samples, crafting adversarial instances that induce targeted misclassifications and evade detection. We implement VTarbel and evaluate it against four model architectures, seven multimodal datasets, and two anomaly detectors. Across all settings, VTarbel outperforms four state-of-the-art baselines, evades detection, and retains effective against three representative privacy-preserving defenses. These results reveal critical security blind spots in current VFL deployments and underscore urgent need for robust, attack-aware defenses.

http://arxiv.org/abs/2507.14629
VMask: Tunable Label Privacy Protection for Vertical Federated Learning via Layer Masking. (33%)
Juntao Tan; Lan Zhang; Zhonghao Hu; Kai Yang; Peng Ran; Bo Li
Though vertical federated learning (VFL) is generally considered to be privacy-preserving, recent studies have shown that VFL system is vulnerable to label inference attacks originating from various attack surfaces. Among these attacks, the model completion (MC) attack is currently the most powerful one. Existing defense methods against it either sacrifice model accuracy or incur impractical computational overhead. In this paper, we propose VMask, a novel label privacy protection framework designed to defend against MC attack from the perspective of layer masking. Our key insight is to disrupt the strong correlation between input data and intermediate outputs by applying the secret sharing (SS) technique to mask layer parameters in the attacker's model. We devise a strategy for selecting critical layers to mask, reducing the overhead that would arise from naively applying SS to the entire model. Moreover, VMask is the first framework to offer a tunable privacy budget to defenders, allowing for flexible control over the levels of label privacy according to actual requirements. We built a VFL system, implemented VMask on it, and extensively evaluated it using five model architectures and 13 datasets with different modalities, comparing it to 12 other defense methods. The results demonstrate that VMask achieves the best privacy-utility trade-off, successfully thwarting the MC attack (reducing the label inference accuracy to a random guessing level) while preserving model performance (e.g., in Transformer-based model, the averaged drop of VFL model accuracy is only 0.09%). VMask's runtime is up to 60,846 times faster than cryptography-based methods, and it only marginally exceeds that of standard VFL by 1.8 times in a large Transformer-based model, which is generally acceptable.

http://arxiv.org/abs/2507.14248
Breaking the Illusion of Security via Interpretation: Interpretable Vision Transformer Systems under Attack. (99%)
Eldor Abdukhamidov; Mohammed Abuhamad; Simon S. Woo; Hyoungshick Kim; Tamer Abuhmed
Vision transformer (ViT) models, when coupled with interpretation models, are regarded as secure and challenging to deceive, making them well-suited for security-critical domains such as medical applications, autonomous vehicles, drones, and robotics. However, successful attacks on these systems can lead to severe consequences. Recent research on threats targeting ViT models primarily focuses on generating the smallest adversarial perturbations that can deceive the models with high confidence, without considering their impact on model interpretations. Nevertheless, the use of interpretation models can effectively assist in detecting adversarial examples. This study investigates the vulnerability of transformer models to adversarial attacks, even when combined with interpretation models. We propose an attack called "AdViT" that generates adversarial examples capable of misleading both a given transformer model and its coupled interpretation model. Through extensive experiments on various transformer models and two transformer-based interpreters, we demonstrate that AdViT achieves a 100% attack success rate in both white-box and black-box scenarios. In white-box scenarios, it reaches up to 98% misclassification confidence, while in black-box scenarios, it reaches up to 76% misclassification confidence. Remarkably, AdViT consistently generates accurate interpretations in both scenarios, making the adversarial examples more difficult to detect.

http://arxiv.org/abs/2507.13727
Adversarial Training Improves Generalization Under Distribution Shifts in Bioacoustics. (96%)
René Heinrich; Lukas Rauch; Bernhard Sick; Christoph Scholz
Adversarial training is a promising strategy for enhancing model robustness against adversarial attacks. However, its impact on generalization under substantial data distribution shifts in audio classification remains largely unexplored. To address this gap, this work investigates how different adversarial training strategies improve generalization performance and adversarial robustness in audio classification. The study focuses on two model architectures: a conventional convolutional neural network (ConvNeXt) and an inherently interpretable prototype-based model (AudioProtoPNet). The approach is evaluated using a challenging bird sound classification benchmark. This benchmark is characterized by pronounced distribution shifts between training and test data due to varying environmental conditions and recording methods, a common real-world challenge. The investigation explores two adversarial training strategies: one based on output-space attacks that maximize the classification loss function, and another based on embedding-space attacks designed to maximize embedding dissimilarity. These attack types are also used for robustness evaluation. Additionally, for AudioProtoPNet, the study assesses the stability of its learned prototypes under targeted embedding-space attacks. Results show that adversarial training, particularly using output-space attacks, improves clean test data performance by an average of 10.5% relative and simultaneously strengthens the adversarial robustness of the models. These findings, although derived from the bird sound domain, suggest that adversarial training holds potential to enhance robustness against both strong distribution shifts and adversarial attacks in challenging audio classification settings.

http://arxiv.org/abs/2507.13761
Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Models. (86%)
Palash Nandi; Maithili Joshi; Tanmoy Chakraborty
Language models are highly sensitive to prompt formulations - small changes in input can drastically alter their output. This raises a critical question: To what extent can prompt sensitivity be exploited to generate inapt content? In this paper, we investigate how discrete components of prompt design influence the generation of inappropriate content in Visual Language Models (VLMs). Specifically, we analyze the impact of three key factors on successful jailbreaks: (a) the inclusion of detailed visual information, (b) the presence of adversarial examples, and (c) the use of positively framed beginning phrases. Our findings reveal that while a VLM can reliably distinguish between benign and harmful inputs in unimodal settings (text-only or image-only), this ability significantly degrades in multimodal contexts. Each of the three factors is independently capable of triggering a jailbreak, and we show that even a small number of in-context examples (as few as three) can push the model toward generating inappropriate outputs. Furthermore, we propose a framework that utilizes a skip-connection between two internal layers of the VLM, which substantially increases jailbreak success rates, even when using benign images. Finally, we demonstrate that memes, often perceived as humorous or harmless, can be as effective as toxic visuals in eliciting harmful content, underscoring the subtle and complex vulnerabilities of VLMs.

http://arxiv.org/abs/2504.10888
CDUPatch: Color-Driven Universal Adversarial Patch Attack for Dual-Modal Visible-Infrared Detectors. (76%)
Jiahuan Long; Wen Yao; Tingsong Jiang; Chao Ma
Adversarial patches are widely used to evaluate the robustness of object detection systems in real-world scenarios. These patches were initially designed to deceive single-modal detectors (e.g., visible or infrared) and have recently been extended to target visible-infrared dual-modal detectors. However, existing dual-modal adversarial patch attacks have limited attack effectiveness across diverse physical scenarios. To address this, we propose CDUPatch, a universal cross-modal patch attack against visible-infrared object detectors across scales, views, and scenarios. Specifically, we observe that color variations lead to different levels of thermal absorption, resulting in temperature differences in infrared imaging. Leveraging this property, we propose an RGB-to-infrared adapter that maps RGB patches to infrared patches, enabling unified optimization of cross-modal patches. By learning an optimal color distribution on the adversarial patch, we can manipulate its thermal response and generate an adversarial infrared texture. Additionally, we introduce a multi-scale clipping strategy and construct a new visible-infrared dataset, MSDrone, which contains aerial vehicle images in varying scales and perspectives. These data augmentation strategies enhance the robustness of our patch in real-world conditions. Experiments on four benchmark datasets (e.g., DroneVehicle, LLVIP, VisDrone, MSDrone) show that our method outperforms existing patch attacks in the digital domain. Extensive physical tests further confirm strong transferability across scales, views, and scenarios.

http://arxiv.org/abs/2507.13686
TopicAttack: An Indirect Prompt Injection Attack via Topic Transition. (54%)
Yulin Chen; Haoran Li; Yuexin Li; Yue Liu; Yangqiu Song; Bryan Hooi
Large language models (LLMs) have shown remarkable performance across a range of NLP tasks. However, their strong instruction-following capabilities and inability to distinguish instructions from data content make them vulnerable to indirect prompt injection attacks. In such attacks, instructions with malicious purposes are injected into external data sources, such as web documents. When LLMs retrieve this injected data through tools, such as a search engine and execute the injected instructions, they provide misled responses. Recent attack methods have demonstrated potential, but their abrupt instruction injection often undermines their effectiveness. Motivated by the limitations of existing attack methods, we propose TopicAttack, which prompts the LLM to generate a fabricated conversational transition prompt that gradually shifts the topic toward the injected instruction, making the injection smoother and enhancing the plausibility and success of the attack. Through comprehensive experiments, TopicAttack achieves state-of-the-art performance, with an attack success rate (ASR) over 90\% in most cases, even when various defense methods are applied. We further analyze its effectiveness by examining attention scores. We find that a higher injected-to-original attention ratio leads to a greater success probability, and our method achieves a much higher ratio than the baseline methods.

http://arxiv.org/abs/2506.24068
STACK: Adversarial Attacks on LLM Safeguard Pipelines. (54%)
Ian R. McKenzie; Oskar J. Hollinsworth; Tom Tseng; Xander Davies; Stephen Casper; Aaron D. Tucker; Robert Kirk; Adam Gleave
Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic guards their latest Claude 4 Opus model using one such defense pipeline, and other frontier developers including Google DeepMind and OpenAI pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks.

http://arxiv.org/abs/2507.13598
GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention. (1%)
Amro Abdalla; Ismail Shaheen; Dan DeGenaro; Rupayan Mallick; Bogdan Raita; Sarah Adel Bargal
We present GIFT: a {G}radient-aware {I}mmunization technique to defend diffusion models against malicious {F}ine-{T}uning while preserving their ability to generate safe content. Existing safety mechanisms like safety checkers are easily bypassed, and concept erasure methods fail under adversarial fine-tuning. GIFT addresses this by framing immunization as a bi-level optimization problem: the upper-level objective degrades the model's ability to represent harmful concepts using representation noising and maximization, while the lower-level objective preserves performance on safe data. GIFT achieves robust resistance to malicious fine-tuning while maintaining safe generative quality. Experimental results show that our method significantly impairs the model's ability to re-learn harmful concepts while maintaining performance on safe content, offering a promising direction for creating inherently safer generative models resistant to adversarial fine-tuning attacks.

http://arxiv.org/abs/2507.13123
Detecting LLM-generated Code with Subtle Modification by Adversarial Training. (87%)
Xin Yin; Xinrui Li; Chao Ni; Xiaodan Xu; Xiaohu Yang
With the rapid development of Large Language Models (LLMs), their powerful code-generation capabilities have been widely applied in tasks like code completion and automated development, demonstrating the value of improving coding efficiency. However, the extensive use of LLM-generated code also raises several new challenges. On the one hand, issues such as the regulation of code provenance, copyright disputes, and code quality have become increasingly concerning. How to effectively detect LLM-generated code and ensure its compliant and responsible use has become a critical and urgent issue. On the other hand, in practical applications, LLM-generated code is often subject to manual modifications, such as variable renaming or structural adjustments. Although some recent studies have proposed training-based and zero-shot methods for detecting LLM-generated code, these approaches show insufficient robustness when facing modified LLM-generated code, and there is a lack of an effective solution. To address the real-world scenario where LLM-generated code may undergo minor modifications, we propose CodeGPTSensor+, an enhanced version of CodeGPTSensor, which employs adversarial training to improve robustness against input perturbations. CodeGPTSensor+ integrates an adversarial sample generation module, Multi-objective Identifier and Structure Transformation (MIST), which systematically generates both high-quality and representative adversarial samples. This module effectively enhances the model's resistance against diverse adversarial attacks. Experimental results on the HMCorp dataset demonstrate that CodeGPTSensor+ significantly improves detection accuracy on the adversarial test set while maintaining high accuracy on the original test set, showcasing superior robustness compared to CodeGPTSensor.

http://arxiv.org/abs/2507.13338
Training Transformers with Enforced Lipschitz Constants. (80%)
Laker Newhouse; R. Preston Hess; Franz Cesista; Andrii Zahorodnii; Jeremy Bernstein; Phillip Isola
Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, divergent training, and overfitting. To combat these problems, past research has looked at building neural networks entirely from Lipschitz components. However, these techniques have not matured to the point where researchers have trained a modern architecture such as a transformer with a Lipschitz certificate enforced beyond initialization. To explore this gap, we begin by developing and benchmarking novel, computationally-efficient tools for maintaining norm-constrained weight matrices. Applying these tools, we are able to train transformer models with Lipschitz bounds enforced throughout training. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard methods -- weight decay and spectral normalization -- allowing models to reach equal performance with a lower Lipschitz bound. Inspired by Muon's update having a fixed spectral norm, we co-design a weight constraint method that improves the Lipschitz vs. performance tradeoff on MLPs and 2M parameter transformers. Our 2-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 10-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^264. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and logit tanh softcapping.

http://arxiv.org/abs/2507.13170
SHIELD: A Secure and Highly Enhanced Integrated Learning for Robust Deepfake Detection against Adversarial Attacks. (61%)
Kutub Uddin; Awais Khan; Muhammad Umar Farooq; Khalid Malik
Audio plays a crucial role in applications like speaker verification, voice-enabled smart devices, and audio conferencing. However, audio manipulations, such as deepfakes, pose significant risks by enabling the spread of misinformation. Our empirical analysis reveals that existing methods for detecting deepfake audio are often vulnerable to anti-forensic (AF) attacks, particularly those attacked using generative adversarial networks. In this article, we propose a novel collaborative learning method called SHIELD to defend against generative AF attacks. To expose AF signatures, we integrate an auxiliary generative model, called the defense (DF) generative model, which facilitates collaborative learning by combining input and output. Furthermore, we design a triplet model to capture correlations for real and AF attacked audios with real-generated and attacked-generated audios using auxiliary generative models. The proposed SHIELD strengthens the defense against generative AF attacks and achieves robust performance across various generative models. The proposed AF significantly reduces the average detection accuracy from 95.49% to 59.77% for ASVspoof2019, from 99.44% to 38.45% for In-the-Wild, and from 98.41% to 51.18% for HalfTruth for three different generative models. The proposed SHIELD mechanism is robust against AF attacks and achieves an average accuracy of 98.13%, 98.58%, and 99.57% in match, and 98.78%, 98.62%, and 98.85% in mismatch settings for the ASVspoof2019, In-the-Wild, and HalfTruth datasets, respectively.

http://arxiv.org/abs/2507.13407
IConMark: Robust Interpretable Concept-Based Watermark For AI Images. (54%)
Vinu Sankar Sadasivan; Mehrdad Saberi; Soheil Feizi
With the rapid rise of generative AI and synthetic media, distinguishing AI-generated images from real ones has become crucial in safeguarding against misinformation and ensuring digital authenticity. Traditional watermarking techniques have shown vulnerabilities to adversarial attacks, undermining their effectiveness in the presence of attackers. We propose IConMark, a novel in-generation robust semantic watermarking method that embeds interpretable concepts into AI-generated images, as a first step toward interpretable watermarking. Unlike traditional methods, which rely on adding noise or perturbations to AI-generated images, IConMark incorporates meaningful semantic attributes, making it interpretable to humans and hence, resilient to adversarial manipulation. This method is not only robust against various image augmentations but also human-readable, enabling manual verification of watermarks. We demonstrate a detailed evaluation of IConMark's effectiveness, demonstrating its superiority in terms of detection accuracy and maintaining image quality. Moreover, IConMark can be combined with existing watermarking techniques to further enhance and complement its robustness. We introduce IConMark+SS and IConMark+TM, hybrid approaches combining IConMark with StegaStamp and TrustMark, respectively, to further bolster robustness against multiple types of image manipulations. Our base watermarking technique (IConMark) and its variants (+TM and +SS) achieve 10.8%, 14.5%, and 15.9% higher mean area under the receiver operating characteristic curve (AUROC) scores for watermark detection, respectively, compared to the best baseline on various datasets.

http://arxiv.org/abs/2507.13136
Adversarial attacks to image classification systems using evolutionary algorithms. (22%)
Sergio Nesmachnow; Jamal Toutouh
Image classification currently faces significant security challenges due to adversarial attacks, which consist of intentional alterations designed to deceive classification models based on artificial intelligence. This article explores an approach to generate adversarial attacks against image classifiers using a combination of evolutionary algorithms and generative adversarial networks. The proposed approach explores the latent space of a generative adversarial network with an evolutionary algorithm to find vectors representing adversarial attacks. The approach was evaluated in two case studies corresponding to the classification of handwritten digits and object images. The results showed success rates of up to 35% for handwritten digits, and up to 75% for object images, improving over other search methods and reported results in related works. The applied method proved to be effective in handling data diversity on the target datasets, even in problem instances that presented additional challenges due to the complexity and richness of information.

http://arxiv.org/abs/2507.13575
Apple Intelligence Foundation Language Models: Tech Report 2025. (15%)
Hanzhi Zhou; Erik Hornberger; Pengsheng Guo; Xiyou Zhou; Saiwen Wang; Xin Wang; Yifei He; Xuankai Chang; Rene Rauch; Louis D'hauwe; John Peebles; Alec Doane; Kohen Chia; Jenna Thibodeau; Zi-Yi Dou; Yuanyang Zhang; Ruoming Pang; Reed Li; Zhifeng Chen; Jeremy Warner; Zhaoyang Xu; Sophy Lee; David Mizrahi; Ramsey Tantawi; Chris Chaney; Kelsey Peterson; Jun Qin; Alex Dombrowski; Mira Chiang; Aiswarya Raghavan; Gerard Casamayor; Qibin Chen; Aonan Zhang; Nathalie Tran; Jianyu Wang; Hang Su; Thomas Voice; Alessandro Pappalardo; Brycen Wershing; Prasanth Yadla; Rui Li; Priyal Chhatrapati; Ismael Fernandez; Yusuf Goren; Xin Zheng; Forrest Huang; Tao Lei; Eray Yildiz; Alper Kokmen; Gokul Santhanam; Areeba Kamal; Kaan Elgin; Dian Ang Yap; Jeremy Liu; Peter Gray; Howard Xing; Kieran Liu; Matteo Ronchi; Moritz Schwarzer-Becker; Yun Zhu; Mandana Saebi; Jeremy Snow; David Griffiths; Guillaume Tartavel; Erin Feldman; Simon Lehnerer; Fernando Bermúdez-Medina; Hans Han; Joe Zhou; Xiaoyi Ren; Sujeeth Reddy; Zirui Wang; Tom Gunter; Albert Antony; Yuanzhi Li; John Dennison; Tony Sun; Yena Han; Yi Qin; Sam Davarnia; Jeffrey Bigham; Wayne Shan; Hannah Gillis Coleman; Guillaume Klein; Peng Liu; Muyang Yu; Jack Cackler; Yuan Gao; Crystal Xiao; Binazir Karimzadeh; Zhengdong Zhang; Felix Bai; Albin Madappally Jose; Feng Nan; Nazir Kamaldin; Dong Yin; Hans Hao; Yanchao Sun; Yi Hua; Charles Maalouf; Alex Guillen Garcia; Guoli Yin; Lezhi Li; Mohana Prasad Sathya Moorthy; Hongbin Gao; Jay Tang; Joanna Arreaza-Taylor; Faye Lao; Carina Peng; Josh Shaffer; Dan Masi; Sushma Rao; Tommi Vehvilainen; Senyu Tong; Dongcai Shen; Yang Zhao; Chris Bartels; Peter Fu; Qingqing Cao; Christopher Neubauer; Ethan Li; Mingfei Gao; Rebecca Callahan; Richard Wei; Patrick Dong; Alex Braunstein; Sachin Ravi; Adolfo Lopez Mendez; Kaiwei Huang; Kun Duan; Haoshuo Huang; Rui Qian; Stefano Ligas; Jordan Huffaker; Dongxu Li; Bailin Wang; Nanzhu Wang; Anuva Agarwal; Tait Madsen; Josh Newnham; Abhishek Sharma; Zhile Ren; Deepak Gopinath; Erik Daxberger; Saptarshi Guha; Oron Levy; Jing Lu; Nan Dun; Marc Kirchner; Yinfei Yang; Manjot Bilkhu; Dave Nelson; Anthony Spalvieri-Kruse; Juan Lao Tebar; Yang Xu; Phani Mutyala; Gabriel Jacoby-Cooper; Yingbo Wang; Karla Vega; Vishaal Mahtani; Darren Botten; Eric Wang; Hanli Li; Matthias Paulik; Haoran Yan; Navid Shiee; Yihao Qian; Bugu Wu; Qi Zhu; Ob Adaranijo; Bhuwan Dhingra; Zhe Gan; Nicholas Seidl; Grace Duanmu; Rong Situ; Yiping Ma; Yin Xia; David Riazati; Vasileios Saveris; Anh Nguyen; Michael; Lee; Patrick Sonnenberg; Chinguun Erdenebileg; Yanghao Li; Vivian Ma; James Chou; Isha Garg; Mark Lee; Keen You; Yuhong Li; Ransen Niu; Nandhitha Raghuram; Pulkit Agrawal; Henry Mason; Sumeet Singh; Keyu He; Hong-You Chen; Lucas Guibert; Shiyu Li; Varsha Paidi; Narendran Raghavan; Mingze Xu; Yuli Yang; Sergiu Sima; Irina Belousova; Sprite Chu; Afshin Dehghan; Philipp Dufter; David Haldimann; Zhen Yang; Margit Bowler; Chang Liu; Ying-Chang Cheng; Vivek Rathod; Syd Evans; Wilson Tsao; Dustin Withers; Haitian Sun; Biyao Wang; Peter Grasch; Walker Cheng; Yihao Feng; Vivek Kumar; Frank Chu; Victoria MönchJuan Haladjian; Doug Kang; Jiarui Lu; Ciro Sannino; Max Lam; Floris Weers; Bowen Pan; Kenneth Jung; Dhaval Doshi; Fangping Shi; Olli Saarikivi; Alp Aygar; Josh Elman; Cheng Leong; Eshan Verma; Matthew Lei; Jeff Nichols; Jiulong Shan; Donald Zhang; Lawrence Zhou; Stephen Murphy; Xianzhi Du; Chang Lan; Ankur Jain; Elmira Amirloo; Marcin Eichner; Naomy Sabo; Anupama Mann Anupama; David Qiu; Zhao Meng; Michael FitzMaurice; Peng Zhang; Simon Yeung; Chen Chen; Marco Zuliani; Andrew Hansen; Yang Lu; Brent Ramerth; Ziyi Zhong; Parsa Mazaheri; Matthew Hopkins; Mengyu Li; Simon Wang; David Chen; Farzin Rasteh; Chong Wang; Josh Gardner; Asaf Liberman; Haoxuan You; Andrew Walkingshaw; Xingyu Zhou; Jinhao Lei; Yan Meng; Quentin Keunebroek; Sam Wiseman; Anders Boesen Lindbo Larsen; Yi Zhang; Zaid Ahmed; Haiming Gang; Aaron Franklin; Kelvin Zou; Guillaume Seguin; Jonathan Janke; Rachel Burger; Co Giang; Cheng Shen; Jen Liu; Sanskruti Shah; Xiang Kong; Yiran Fei; TJ Collins; Chen Zhang; Zhiyun Lu; Michael Booker; Qin Ba; Yasutaka Tanaka; Andres Romero Mier Y Teran; Federico Scozzafava; Regan Poston; Jane Li; Eduardo Jimenez; Bas Straathof; Karanjeet Singh; Lindsay Hislop; Rajat Arora; Deepa Seshadri; Boyue Li; Colorado Reed; Zhen Li; TJ Lu; Yi Wang; Kaelen Haag; Nicholas Lusskin; Raunak Sinha; Rahul Nair; Eldon Schoop; Mary Beth Kery; Mehrdad Farajtbar; Brenda Yang; George Horrell; Shiwen Zhao; Dhruti Shah; Cha Chen; Bowen Zhang; Chang Gao; Devi Krishna; Jennifer Mallalieu; Javier Movellan; Di Feng; Emily Zhang; Sam Xu; Junting Pan; Dominik Moritz; Suma Jayaram; Kevin Smith; Dongseong Hwang; Daniel Parilla; Jiaming Hu; You-Cyuan Jhang; Emad Soroush; Fred Hohman; Nan Du; Emma Wang; Sam Dodge; Pragnya Sridhar; Joris Pelemans; Wei Fang; Nina Wenzel; Joseph Yitan Cheng; Hadas Kotek; Chung-Cheng Chiu; Meng Cao; Haijing Fu; Ruixuan Hou; Ke Ye; Diane Zhu; Nikhil Bhendawade; Joseph Astrauskas; Jian Liu; Sai Aitharaju; Wentao Wu; Artsiom Peshko; Hyunjik Kim; Nilesh Shahdadpuri; Wang Andy De; Qi Shan; Piotr Maj; Raul Rea Menacho; Justin Lazarow; Eric Liang Yang; Arsalan Farooq; Donghan Yu; David Güera; Minsik Cho; Kavya Nerella; Yongqiang Wang; Tao Jia; John Park; Jeff Lai; Haotian Zhang; Futang Peng; Daniele Molinari; Aparna Rajamani; Tyler Johnson; Lauren Gardiner; Chao Jia; Violet Yao; Wojciech Kryscinski; Xiujun Li; Shang-Chen Wu
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model optimized for Apple silicon through architectural innovations such as KV-cache sharing and 2-bit quantization-aware training; and ii a scalable server model built on a novel Parallel-Track Mixture-of-Experts PT-MoE transformer that combines track parallelism, mixture-of-experts sparse computation, and interleaved global-local attention to deliver high quality with competitive cost on Apple's Private Cloud Compute platform. Both models are trained on large-scale multilingual and multimodal datasets sourced via responsible web crawling, licensed corpora, and high-quality synthetic data, then further refined with supervised fine-tuning and reinforcement learning on a new asynchronous platform. The resulting models support several additional languages while understanding images and executing tool calls. In public benchmarks and human evaluations, both the server model and the on-device model match or surpass comparably sized open baselines.
  A new Swift-centric Foundation Models framework exposes guided generation, constrained tool calling, and LoRA adapter fine-tuning, allowing developers to integrate these capabilities with a few lines of code. The latest advancements in Apple Intelligence models are grounded in our Responsible AI approach with safeguards like content filtering and locale-specific evaluation, as well as our commitment to protecting our users' privacy with innovations like Private Cloud Compute.

http://arxiv.org/abs/2507.05630
How Not to Detect Prompt Injections with an LLM. (10%)
Sarthak Choudhary; Divyam Anshumaan; Nils Palumbo; Somesh Jha
LLM-integrated applications and agents are vulnerable to prompt injection attacks, in which adversaries embed malicious instructions within seemingly benign user inputs to manipulate the LLM's intended behavior. Recent defenses based on $\textit{known-answer detection}$ (KAD) have achieved near-perfect performance by using an LLM to classify inputs as clean or contaminated. In this work, we formally characterize the KAD framework and uncover a structural vulnerability in its design that invalidates its core security premise. We design a methodical adaptive attack, $\textit{DataFlip}$, to exploit this fundamental weakness. It consistently evades KAD defenses with detection rates as low as $1.5\%$ while reliably inducing malicious behavior with success rates of up to $88\%$, without needing white-box access to the LLM or any optimization procedures.

http://arxiv.org/abs/2507.12127
Self-Adaptive and Robust Federated Spectrum Sensing without Benign Majority for Cellular Networks. (83%)
Ngoc Duy Pham; Thusitha Dayaratne; Viet Vo; Shangqi Lai; Sharif Abuadbba; Hajime Suzuki; Xingliang Yuan; Carsten Rudolph
Advancements in wireless and mobile technologies, including 5G advanced and the envisioned 6G, are driving exponential growth in wireless devices. However, this rapid expansion exacerbates spectrum scarcity, posing a critical challenge. Dynamic spectrum allocation (DSA)--which relies on sensing and dynamically sharing spectrum--has emerged as an essential solution to address this issue. While machine learning (ML) models hold significant potential for improving spectrum sensing, their adoption in centralized ML-based DSA systems is limited by privacy concerns, bandwidth constraints, and regulatory challenges. To overcome these limitations, distributed ML-based approaches such as Federated Learning (FL) offer promising alternatives. This work addresses two key challenges in FL-based spectrum sensing (FLSS). First, the scarcity of labeled data for training FL models in practical spectrum sensing scenarios is tackled with a semi-supervised FL approach, combined with energy detection, enabling model training on unlabeled datasets. Second, we examine the security vulnerabilities of FLSS, focusing on the impact of data poisoning attacks. Our analysis highlights the shortcomings of existing majority-based defenses in countering such attacks. To address these vulnerabilities, we propose a novel defense mechanism inspired by vaccination, which effectively mitigates data poisoning attacks without relying on majority-based assumptions. Extensive experiments on both synthetic and real-world datasets validate our solutions, demonstrating that FLSS can achieve near-perfect accuracy on unlabeled datasets and maintain Byzantine robustness against both targeted and untargeted data poisoning attacks, even when a significant proportion of participants are malicious.

http://arxiv.org/abs/2507.12107
Non-Adaptive Adversarial Face Generation. (67%)
Sunpill Kim; Seunghun Paik; Chanwoo Hwang; Minsu Kim; Jae Hong Seo
Adversarial attacks on face recognition systems (FRSs) pose serious security and privacy threats, especially when these systems are used for identity verification. In this paper, we propose a novel method for generating adversarial faces-synthetic facial images that are visually distinct yet recognized as a target identity by the FRS. Unlike iterative optimization-based approaches (e.g., gradient descent or other iterative solvers), our method leverages the structural characteristics of the FRS feature space. We figure out that individuals sharing the same attribute (e.g., gender or race) form an attributed subsphere. By utilizing such subspheres, our method achieves both non-adaptiveness and a remarkably small number of queries. This eliminates the need for relying on transferability and open-source surrogate models, which have been a typical strategy when repeated adaptive queries to commercial FRSs are impossible. Despite requiring only a single non-adaptive query consisting of 100 face images, our method achieves a high success rate of over 93% against AWS's CompareFaces API at its default threshold. Furthermore, unlike many existing attacks that perturb a given image, our method can deliberately produce adversarial faces that impersonate the target identity while exhibiting high-level attributes chosen by the adversary.

http://arxiv.org/abs/2507.12384
Trustworthy Tree-based Machine Learning by $MoS_2$ Flash-based Analog CAM with Inherent Soft Boundaries. (15%)
Bo Wen; Guoyun Gao; Zhicheng Xu; Ruibin Mao; Xiaojuan Qi; X. Sharon Hu; Xunzhao Yin; Can Li
The rapid advancement of artificial intelligence has raised concerns regarding its trustworthiness, especially in terms of interpretability and robustness. Tree-based models like Random Forest and XGBoost excel in interpretability and accuracy for tabular data, but scaling them remains computationally expensive due to poor data locality and high data dependence. Previous efforts to accelerate these models with analog content addressable memory (CAM) have struggled, due to the fact that the difficult-to-implement sharp decision boundaries are highly susceptible to device variations, which leads to poor hardware performance and vulnerability to adversarial attacks. This work presents a novel hardware-software co-design approach using $MoS_2$ Flash-based analog CAM with inherent soft boundaries, enabling efficient inference with soft tree-based models. Our soft tree model inference experiments on $MoS_2$ analog CAM arrays show this method achieves exceptional robustness against device variation and adversarial attacks while achieving state-of-the-art accuracy. Specifically, our fabricated analog CAM arrays achieve $96\%$ accuracy on Wisconsin Diagnostic Breast Cancer (WDBC) database, while maintaining decision explainability. Our experimentally calibrated model validated only a $0.6\%$ accuracy drop on the MNIST dataset under $10\%$ device threshold variation, compared to a $45.3\%$ drop for traditional decision trees. This work paves the way for specialized hardware that enhances AI's trustworthiness and efficiency.

http://arxiv.org/abs/2507.12439
A Bayesian Incentive Mechanism for Poison-Resilient Federated Learning. (9%)
Daniel Commey; Rebecca A. Sarpong; Griffith S. Klogo; Winful Bagyl-Bac; Garth V. Crosby
Federated learning (FL) enables collaborative model training across decentralized clients while preserving data privacy. However, its open-participation nature exposes it to data-poisoning attacks, in which malicious actors submit corrupted model updates to degrade the global model. Existing defenses are often reactive, relying on statistical aggregation rules that can be computationally expensive and that typically assume an honest majority. This paper introduces a proactive, economic defense: a lightweight Bayesian incentive mechanism that makes malicious behavior economically irrational. Each training round is modeled as a Bayesian game of incomplete information in which the server, acting as the principal, uses a small, private validation dataset to verify update quality before issuing payments. The design satisfies Individual Rationality (IR) for benevolent clients, ensuring their participation is profitable, and Incentive Compatibility (IC), making poisoning an economically dominated strategy. Extensive experiments on non-IID partitions of MNIST and FashionMNIST demonstrate robustness: with 50% label-flipping adversaries on MNIST, the mechanism maintains 96.7% accuracy, only 0.3 percentage points lower than in a scenario with 30% label-flipping adversaries. This outcome is 51.7 percentage points better than standard FedAvg, which collapses under the same 50% attack. The mechanism is computationally light, budget-bounded, and readily integrates into existing FL frameworks, offering a practical route to economically robust and sustainable FL ecosystems.

http://arxiv.org/abs/2507.12314
Thought Purity: Defense Paradigm For Chain-of-Thought Attack. (9%)
Zihao Xue; Zhen Bi; Long Ma; Zhenlin Hu; Yan Wang; Zhenfang Liu; Qing Sheng; Jie Xiao; Jungang Lou
While reinforcement learning-trained Large Reasoning Models (LRMs, e.g., Deepseek-R1) demonstrate advanced reasoning capabilities in the evolving Large Language Models (LLMs) domain, their susceptibility to security threats remains a critical vulnerability. This weakness is particularly evident in Chain-of-Thought (CoT) generation processes, where adversarial methods like backdoor prompt attacks can systematically subvert the model's core reasoning mechanisms. The emerging Chain-of-Thought Attack (CoTA) reveals this vulnerability through exploiting prompt controllability, simultaneously degrading both CoT safety and task performance with low-cost interventions. To address this compounded security-performance vulnerability, we propose Thought Purity (TP): a defense paradigm that systematically strengthens resistance to malicious content while preserving operational efficacy. Our solution achieves this through three synergistic components: (1) a safety-optimized data processing pipeline (2) reinforcement learning-enhanced rule constraints (3) adaptive monitoring metrics. Our approach establishes the first comprehensive defense mechanism against CoTA vulnerabilities in reinforcement learning-aligned reasoning systems, significantly advancing the security-functionality equilibrium for next-generation AI architectures.

http://arxiv.org/abs/2507.11968
Watch, Listen, Understand, Mislead: Tri-modal Adversarial Attacks on Short Videos for Content Appropriateness Evaluation. (2%)
Sahid Hossain Mustakim; S M Jishanul Islam; Ummay Maria Muna; Montasir Chowdhury; Mohammed Jawwadul Islam; Sadia Ahmmed; Tashfia Sikder; Syed Tasdid Azam Dhrubo; Swakkhar Shatabda
Multimodal Large Language Models (MLLMs) are increasingly used for content moderation, yet their robustness in short-form video contexts remains underexplored. Current safety evaluations often rely on unimodal attacks, failing to address combined attack vulnerabilities. In this paper, we introduce a comprehensive framework for evaluating the tri-modal safety of MLLMs. First, we present the Short-Video Multimodal Adversarial (SVMA) dataset, comprising diverse short-form videos with human-guided synthetic adversarial attacks. Second, we propose ChimeraBreak, a novel tri-modal attack strategy that simultaneously challenges visual, auditory, and semantic reasoning pathways. Extensive experiments on state-of-the-art MLLMs reveal significant vulnerabilities with high Attack Success Rates (ASR). Our findings uncover distinct failure modes, showing model biases toward misclassifying benign or policy-violating content. We assess results using LLM-as-a-judge, demonstrating attack reasoning efficacy. Our dataset and findings provide crucial insights for developing more robust and safe MLLMs.

http://arxiv.org/abs/2507.10998
Crafting Imperceptible On-Manifold Adversarial Attacks for Tabular Data. (99%)
Zhipeng He; Alexander Stevens; Chun Ouyang; Smedt Johannes De; Alistair Barros; Catarina Moreira
Adversarial attacks on tabular data present fundamental challenges distinct from image or text domains due to the heterogeneous nature of mixed categorical and numerical features. Unlike images where pixel perturbations maintain visual similarity, tabular data lacks intuitive similarity metrics, making it difficult to define imperceptible modifications. Additionally, traditional gradient-based methods prioritise $\ell_p$-norm constraints, often producing adversarial examples that deviate from the original data distributions, making them detectable. We propose a latent space perturbation framework using a mixed-input Variational Autoencoder (VAE) to generate imperceptible adversarial examples. The proposed VAE integrates categorical embeddings and numerical features into a unified latent manifold, enabling perturbations that preserve statistical consistency. We specify In-Distribution Success Rate (IDSR) to measure the proportion of adversarial examples that remain statistically indistinguishable from the input distribution. Evaluation across six publicly available datasets and three model architectures demonstrates that our method achieves substantially lower outlier rates and more consistent performance compared to traditional input-space attacks and other VAE-based methods adapted from image domain approaches. Our comprehensive analysis includes hyperparameter sensitivity, sparsity control mechanisms, and generative architectural comparisons, revealing that VAE-based attacks depend critically on reconstruction quality but offer superior practical utility when sufficient training data is available. This work highlights the importance of on-manifold perturbations for realistic adversarial attacks on tabular data, offering a robust approach for practical deployment. The source code can be accessed through https://github.com/ZhipengHe/VAE-TabAttack.

http://arxiv.org/abs/2507.04762
Robustifying 3D Perception via Least-Squares Graphs for Multi-Agent Object Tracking. (15%)
Maria Damanaki; Ioulia Kapsali; Nikos Piperigkos; Alexandros Gkillas; Aris S. Lalos
The critical perception capabilities of EdgeAI systems, such as autonomous vehicles, are required to be resilient against adversarial threats, by enabling accurate identification and localization of multiple objects in the scene over time, mitigating their impact. Single-agent tracking offers resilience to adversarial attacks but lacks situational awareness, underscoring the need for multi-agent cooperation to enhance context understanding and robustness. This paper proposes a novel mitigation framework on 3D LiDAR scene against adversarial noise by tracking objects based on least-squares graph on multi-agent adversarial bounding boxes. Specifically, we employ the least-squares graph tool to reduce the induced positional error of each detection's centroid utilizing overlapped bounding boxes on a fully connected graph via differential coordinates and anchor points. Hence, the multi-vehicle detections are fused and refined mitigating the adversarial impact, and associated with existing tracks in two stages performing tracking to further suppress the adversarial threat. An extensive evaluation study on the real-world V2V4Real dataset demonstrates that the proposed method significantly outperforms both state-of-the-art single and multi-agent tracking frameworks by up to 23.3% under challenging adversarial conditions, operating as a resilient approach without relying on additional defense mechanisms.

http://arxiv.org/abs/2507.11092
MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing. (10%)
Gong Chen; Wenjie Liu; Xiaoyuan Xie; Xunzhu Tang; Tegawendé F. Bissyandé; Songqiang Chen
Recently, several studies have indicated that data poisoning attacks pose a severe security threat to deep learning-based (DL-based) code search models. Attackers inject carefully crafted malicious patterns into the training data, misleading the code search model to learn these patterns during training. During the usage of the poisoned code search model for inference, once the malicious pattern is triggered, the model tends to rank the vulnerability code higher. However, existing detection methods for data poisoning attacks on DL-based code search models remain insufficiently effective. To address this critical security issue, we propose MT4DP, a Data Poisoning Attack Detection Framework for DL-based Code Search Models via Metamorphic Testing. MT4DP introduces a novel Semantically Equivalent Metamorphic Relation (SE-MR) designed to detect data poisoning attacks on DL-based code search models. Specifically, MT4DP first identifies the high-frequency words from search queries as potential poisoning targets and takes their corresponding queries as the source queries. For each source query, MT4DP generates two semantically equivalent follow-up queries and retrieves its source ranking list. Then, each source ranking list is re-ranked based on the semantic similarities between its code snippets and the follow-up queries. Finally, variances between the source and re-ranked lists are calculated to reveal violations of the SE-MR and warn the data poisoning attack. Experimental results demonstrate that MT4DP significantly enhances the detection of data poisoning attacks on DL-based code search models, outperforming the best baseline by 191% on average F1 score and 265% on average precision. Our work aims to promote further research into effective techniques for mitigating data poisoning threats on DL-based code search models.

http://arxiv.org/abs/2507.11112
Multi-Trigger Poisoning Amplifies Backdoor Vulnerabilities in LLMs. (9%)
Sanhanat Sivapiromrat; Caiqi Zhang; Marco Basaldella; Nigel Collier
Recent studies have shown that Large Language Models (LLMs) are vulnerable to data poisoning attacks, where malicious training examples embed hidden behaviours triggered by specific input patterns. However, most existing works assume a phrase and focus on the attack's effectiveness, offering limited understanding of trigger mechanisms and how multiple triggers interact within the model. In this paper, we present a framework for studying poisoning in LLMs. We show that multiple distinct backdoor triggers can coexist within a single model without interfering with each other, enabling adversaries to embed several triggers concurrently. Using multiple triggers with high embedding similarity, we demonstrate that poisoned triggers can achieve robust activation even when tokens are substituted or separated by long token spans. Our findings expose a broader and more persistent vulnerability surface in LLMs. To mitigate this threat, we propose a post hoc recovery method that selectively retrains specific model components based on a layer-wise weight difference analysis. Our method effectively removes the trigger behaviour with minimal parameter updates, presenting a practical and efficient defence against multi-trigger poisoning.

http://arxiv.org/abs/2507.11630
Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibility. (1%)
Brendan Murphy; Dillon Bowen; Shahrad Mohammadzadeh; Julius Broomfield; Adam Gleave; Kellin Pelrine
AI systems are rapidly advancing in capability, and frontier model developers broadly acknowledge the need for safeguards against serious misuse. However, this paper demonstrates that fine-tuning, whether via open weights or closed fine-tuning APIs, can produce helpful-only models. In contrast to prior work which is blocked by modern moderation systems or achieved only partial removal of safeguards or degraded output quality, our jailbreak-tuning method teaches models to generate detailed, high-quality responses to arbitrary harmful requests. For example, OpenAI, Google, and Anthropic models will fully comply with requests for CBRN assistance, executing cyberattacks, and other criminal activity. We further show that backdoors can increase not only the stealth but also the severity of attacks, while stronger jailbreak prompts become even more effective in fine-tuning attacks, linking attack and potentially defenses in the input and weight spaces. Not only are these models vulnerable, more recent ones also appear to be becoming even more vulnerable to these attacks, underscoring the urgent need for tamper-resistant safeguards. Until such safeguards are discovered, companies and policymakers should view the release of any fine-tunable model as simultaneously releasing its evil twin: equally capable as the original model, and usable for any malicious purpose within its capabilities.

http://arxiv.org/abs/2507.10491
BURN: Backdoor Unlearning via Adversarial Boundary Analysis. (99%)
Yanghao Su; Jie Zhang; Yiming Li; Tianwei Zhang; Qing Guo; Weiming Zhang; Nenghai Yu; Nils Lukas; Wenbo Zhou
Backdoor unlearning aims to remove backdoor-related information while preserving the model's original functionality. However, existing unlearning methods mainly focus on recovering trigger patterns but fail to restore the correct semantic labels of poison samples. This limitation prevents them from fully eliminating the false correlation between the trigger pattern and the target label. To address this, we leverage boundary adversarial attack techniques, revealing two key observations. First, poison samples exhibit significantly greater distances from decision boundaries compared to clean samples, indicating they require larger adversarial perturbations to change their predictions. Second, while adversarial predicted labels for clean samples are uniformly distributed, those for poison samples tend to revert to their original correct labels. Moreover, the features of poison samples restore to closely resemble those of corresponding clean samples after adding adversarial perturbations. Building upon these insights, we propose Backdoor Unlearning via adversaRial bouNdary analysis (BURN), a novel defense framework that integrates false correlation decoupling, progressive data refinement, and model purification. In the first phase, BURN employs adversarial boundary analysis to detect poisoned samples based on their abnormal adversarial boundary distances, then restores their correct semantic labels for fine-tuning. In the second phase, it employs a feedback mechanism that tracks prediction discrepancies between the original backdoored model and progressively sanitized models, guiding both dataset refinement and model purification. Extensive evaluations across multiple datasets, architectures, and seven diverse backdoor attack types confirm that BURN effectively removes backdoor threats while maintaining the model's original performance.

http://arxiv.org/abs/2408.04124
Investigating Adversarial Attacks in Software Analytics via Machine Learning Explainability. (99%)
MD Abdul Awal; Mrigank Rochan; Chanchal K. Roy
With the recent advancements in machine learning (ML), numerous ML-based approaches have been extensively applied in software analytics tasks to streamline software development and maintenance processes. Nevertheless, studies indicate that despite their potential usefulness, ML models are vulnerable to adversarial attacks, which may result in significant monetary losses in these processes. As a result, the ML models' robustness against adversarial attacks must be assessed before they are deployed in software analytics tasks. Despite several techniques being available for adversarial attacks in software analytics tasks, exploring adversarial attacks using ML explainability is largely unexplored. Therefore, this study aims to investigate the relationship between ML explainability and adversarial attacks to measure the robustness of ML models in software analytics tasks. In addition, unlike most existing attacks that directly perturb input-space, our attack approach focuses on perturbing feature-space. Our extensive experiments, involving six datasets, three ML explainability techniques, and seven ML models, demonstrate that ML explainability can be used to conduct successful adversarial attacks on ML models in software analytics tasks. This is achieved by modifying only the top 1-3 important features identified by ML explainability techniques. Consequently, the ML models under attack fail to accurately predict up to 86.6% of instances that were correctly predicted before adversarial attacks, indicating the models' low robustness against such attacks. Finally, our proposed technique demonstrates promising results compared to four state-of-the-art adversarial attack techniques targeting tabular data.

http://arxiv.org/abs/2507.10836
REAL-IoT: Characterizing GNN Intrusion Detection Robustness under Practical Adversarial Attack. (99%)
Zhonghao Zhan; Huichi Zhou; Hamed Haddadi
Graph Neural Network (GNN)-based network intrusion detection systems (NIDS) are often evaluated on single datasets, limiting their ability to generalize under distribution drift. Furthermore, their adversarial robustness is typically assessed using synthetic perturbations that lack realism. This measurement gap leads to an overestimation of GNN-based NIDS resilience. To address the limitations, we propose \textbf{REAL-IoT}, a comprehensive framework for robustness evaluation of GNN-based NIDS in IoT environments. Our framework presents a methodology that creates a unified dataset from canonical datasets to assess generalization under drift. In addition, it features a novel intrusion dataset collected from a physical IoT testbed, which captures network traffic and attack scenarios under real-world settings. Furthermore, using REAL-IoT, we explore the usage of Large Language Models (LLMs) to analyze network data and mitigate the impact of adversarial examples by filtering suspicious flows. Our evaluations using REAL-IoT reveal performance drops in GNN models compared to results from standard benchmarks, quantifying their susceptibility to drift and realistic attacks. We also demonstrate the potential of LLM-based filtering to enhance robustness. These findings emphasize the necessity of realistic threat modeling and rigorous measurement practices for developing resilient IoT intrusion detection systems.

http://arxiv.org/abs/2502.12377
Alignment and Adversarial Robustness: Are More Human-Like Models More Secure? (98%)
Blaine Hoak; Kunyang Li; Patrick McDaniel
A small but growing body of work has shown that machine learning models which better align with human vision have also exhibited higher robustness to adversarial examples, raising the question: can human-like perception make models more secure? If true generally, such mechanisms would offer new avenues toward robustness. In this work, we conduct a large-scale empirical analysis to systematically investigate the relationship between representational alignment and adversarial robustness. We evaluate 114 models spanning diverse architectures and training paradigms, measuring their neural and behavioral alignment and engineering task performance across 105 benchmarks as well as their adversarial robustness via AutoAttack. Our findings reveal that while average alignment and robustness exhibit a weak overall correlation, specific alignment benchmarks serve as strong predictors of adversarial robustness, particularly those that measure selectivity toward texture or shape. These results suggest that different forms of alignment play distinct roles in model robustness, motivating further investigation into how alignment-driven approaches can be leveraged to build more secure and perceptually-grounded vision models.

http://arxiv.org/abs/2505.12185
EVALOOP: Assessing LLM Robustness in Programming from a Self-consistency Perspective. (95%)
Sen Fang; Weiyuan Ding; Bowen Xu
Assessing the programming capabilities of Large Language Models (LLMs) is crucial for their effective use in software engineering. Current evaluations, however, predominantly measure the accuracy of generated code on static benchmarks, neglecting the critical aspect of model robustness during programming tasks. While adversarial attacks offer insights on model robustness, their effectiveness is limited and evaluation could be constrained. Current adversarial attack methods for robustness evaluation yield inconsistent results, struggling to provide a unified evaluation across different LLMs. We introduce EVALOOP, a novel assessment framework that evaluate the robustness from a self-consistency perspective, i.e., leveraging the natural duality inherent in popular software engineering tasks, e.g., code generation and code summarization. EVALOOP initiates a self-contained feedback loop: an LLM generates output (e.g., code) from an input (e.g., natural language specification), and then use the generated output as the input to produce a new output (e.g., summarizes that code into a new specification). EVALOOP repeats the process to assess the effectiveness of EVALOOP in each loop. This cyclical strategy intrinsically evaluates robustness without rely on any external attack setups, providing a unified metric to evaluate LLMs' robustness in programming. We evaluate 16 prominent LLMs (e.g., GPT-4.1, O4-mini) on EVALOOP and found that EVALOOP typically induces a 5.01%-19.31% absolute drop in pass@1 performance within ten loops. Intriguingly, robustness does not always align with initial performance (i.e., one-time query); for instance, GPT-3.5-Turbo, despite superior initial code generation compared to DeepSeek-V2, demonstrated lower robustness over repeated evaluation loop.

http://arxiv.org/abs/2507.10330
Bridging Robustness and Generalization Against Word Substitution Attacks in NLP via the Growth Bound Matrix Approach. (93%)
Mohammed Bouri; Adnane Saoud
Despite advancements in Natural Language Processing (NLP), models remain vulnerable to adversarial attacks, such as synonym substitutions. While prior work has focused on improving robustness for feed-forward and convolutional architectures, the robustness of recurrent networks and modern state space models (SSMs), such as S4, remains understudied. These architectures pose unique challenges due to their sequential processing and complex parameter dynamics. In this paper, we introduce a novel regularization technique based on Growth Bound Matrices (GBM) to improve NLP model robustness by reducing the impact of input perturbations on model outputs. We focus on computing the GBM for three architectures: Long Short-Term Memory (LSTM), State Space models (S4), and Convolutional Neural Networks (CNN). Our method aims to (1) enhance resilience against word substitution attacks, (2) improve generalization on clean text, and (3) providing the first systematic analysis of SSM (S4) robustness. Extensive experiments across multiple architectures and benchmark datasets demonstrate that our method improves adversarial robustness by up to 8.8% over existing baselines. These results highlight the effectiveness of our approach, outperforming several state-of-the-art methods in adversarial defense. Codes are available at https://github.com/BouriMohammed/GBM

http://arxiv.org/abs/2507.10733
3S-Attack: Spatial, Spectral and Semantic Invisible Backdoor Attack Against DNN Models. (92%)
Jianyao Yin; Luca Arnaboldi; Honglong Chen; Pascal Berrang
Backdoor attacks involve either poisoning the training data or directly modifying the model in order to implant a hidden behavior, that causes the model to misclassify inputs when a specific trigger is present. During inference, the model maintains high accuracy on benign samples but misclassifies poisoned samples into an attacker-specified target class. Existing research on backdoor attacks has explored developing triggers in the spatial, spectral (frequency), and semantic (feature) domains, aiming to make them stealthy. While some approaches have considered designing triggers that are imperceptible in both spatial and spectral domains, few have incorporated the semantic domain. In this paper, we propose a novel backdoor attack, termed 3S-attack, which is stealthy across the spatial, spectral, and semantic domains. The key idea is to exploit the semantic features of benign samples as triggers, using Gradient-weighted Class Activation Mapping (Grad-CAM) and a preliminary model for extraction. The trigger is then embedded in the spectral domain, followed by pixel-level restrictions after converting the samples back to the spatial domain. This process minimizes the distance between poisoned and benign samples, making the attack harder to detect by existing defenses and human inspection. Extensive experiments on various datasets, along with theoretical analysis, demonstrate the stealthiness of 3S-attack and highlight the need for stronger defenses to ensure AI security. Our code is available at: https://anonymous.4open.science/r/anon-project-3776/

http://arxiv.org/abs/2504.11168
Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems. (87%)
William Hackett; Lewis Birch; Stefan Trawicki; Neeraj Suri; Peter Garraghan
Large Language Models (LLMs) guardrail systems are designed to protect against prompt injection and jailbreak attacks. However, they remain vulnerable to evasion techniques. We demonstrate two approaches for bypassing LLM prompt injection and jailbreak detection systems via traditional character injection methods and algorithmic Adversarial Machine Learning (AML) evasion techniques. Through testing against six prominent protection systems, including Microsoft's Azure Prompt Shield and Meta's Prompt Guard, we show that both methods can be used to evade detection while maintaining adversarial utility achieving in some instances up to 100% evasion success. Furthermore, we demonstrate that adversaries can enhance Attack Success Rates (ASR) against black-box targets by leveraging word importance ranking computed by offline white-box models. Our findings reveal vulnerabilities within current LLM protection mechanisms and highlight the need for more robust guardrail systems.

http://arxiv.org/abs/2508.03696
PLA: Prompt Learning Attack against Text-to-Image Generative Models. (81%)
Xinqi Lyu; Yihao Liu; Yanjie Li; Bin Xiao
Text-to-Image (T2I) models have gained widespread adoption across various applications. Despite the success, the potential misuse of T2I models poses significant risks of generating Not-Safe-For-Work (NSFW) content. To investigate the vulnerability of T2I models, this paper delves into adversarial attacks to bypass the safety mechanisms under black-box settings. Most previous methods rely on word substitution to search adversarial prompts. Due to limited search space, this leads to suboptimal performance compared to gradient-based training. However, black-box settings present unique challenges to training gradient-driven attack methods, since there is no access to the internal architecture and parameters of T2I models. To facilitate the learning of adversarial prompts in black-box settings, we propose a novel prompt learning attack framework (PLA), where insightful gradient-based training tailored to black-box T2I models is designed by utilizing multimodal similarities. Experiments show that our new method can effectively attack the safety mechanisms of black-box T2I models including prompt filters and post-hoc safety checkers with a high success rate compared to state-of-the-art methods. Warning: This paper may contain offensive model-generated content.

http://arxiv.org/abs/2409.01062
Random Erasing vs. Model Inversion: A Promising Defense or a False Hope? (70%)
Viet-Hung Tran; Ngoc-Bao Nguyen; Son T. Mai; Hans Vandierendonck; Ira Assent; Alex Kot; Ngai-Man Cheung
Model Inversion (MI) attacks pose a significant privacy threat by reconstructing private training data from machine learning models. While existing defenses primarily concentrate on model-centric approaches, the impact of data on MI robustness remains largely unexplored. In this work, we explore Random Erasing (RE), a technique traditionally used for improving model generalization under occlusion, and uncover its surprising effectiveness as a defense against MI attacks. Specifically, our novel feature space analysis shows that models trained with RE-images introduce a significant discrepancy between the features of MI-reconstructed images and those of the private data. At the same time, features of private images remain distinct from other classes and well-separated from different classification regions. These effects collectively degrade MI reconstruction quality and attack accuracy while maintaining reasonable natural accuracy. Furthermore, we explore two critical properties of RE including Partial Erasure and Random Location. Partial Erasure prevents the model from observing entire objects during training. We find this has a significant impact on MI, which aims to reconstruct the entire objects. Random Location of erasure plays a crucial role in achieving a strong privacy-utility trade-off. Our findings highlight RE as a simple yet effective defense mechanism that can be easily integrated with existing privacy-preserving techniques. Extensive experiments across 37 setups demonstrate that our method achieves state-of-the-art (SOTA) performance in the privacy-utility trade-off. The results consistently demonstrate the superiority of our defense over existing methods across different MI attacks, network architectures, and attack configurations. For the first time, we achieve a significant degradation in attack accuracy without a decrease in utility for some configurations.

http://arxiv.org/abs/2507.11500
ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning. (54%)
Zhengyue Zhao; Yingzi Ma; Somesh Jha; Marco Pavone; Chaowei Xiao
Large Language Models (LLMs) have demonstrated remarkable generative capabilities. However, their susceptibility to misuse has raised significant safety concerns. While post-training safety alignment methods have been widely adopted, LLMs remain vulnerable to malicious instructions that can bypass safety constraints. Recent efforts have introduced inference-time safety reasoning (system-2 alignment), where LLMs conduct a reasoning process to perform safety verification before final response. We show, however, that these checks are driven by ad-hoc reasoning that diverges from the structured human process, where they first discern a user's true intent, then evaluate the associated risk based on the true intent. Consequently, these defenses remain vulnerable to sophisticated jailbreak prompts that cloak harmful goals in seemingly benign language. To build secure and safe LLMs, we propose a reasoning-based safety alignment framework, ARMOR, that replaces the ad-hoc chains of thought reasoning process with human-aligned, structured one. At inference, ARMOR (1) detects likely jailbreak strategies, (2) extracts the user's core intent while discarding deceptive instructions, and (3) applies a policy-grounded safety analysis to the purified request. ARMOR is evaluated on adaptive jailbreak attacks and multiple safety benchmarks, and a test-time scaling is conducted to further improve its performance. Results demonstrate that ARMOR significantly enhances the robustness against state-of-the-art adaptive jailbreak attacks and outperforms recent reasoning-based aligned models across various safety benchmarks.

http://arxiv.org/abs/2507.10635
Formal Verification of Variational Quantum Circuits. (38%)
Nicola Assolini; Luca Marzari; Isabella Mastroeni; Pierro Alessandra di
Variational quantum circuits (VQCs) are a central component of many quantum machine learning algorithms, offering a hybrid quantum-classical framework that, under certain aspects, can be considered similar to classical deep neural networks. A shared aspect is, for instance, their vulnerability to adversarial inputs, small perturbations that can lead to incorrect predictions. While formal verification techniques have been extensively developed for classical models, no comparable framework exists for certifying the robustness of VQCs. Here, we present the first in-depth theoretical and practical study of the formal verification problem for VQCs. Inspired by abstract interpretation methods used in deep learning, we analyze the applicability and limitations of interval-based reachability techniques in the quantum setting. We show that quantum-specific aspects, such as state normalization, introduce inter-variable dependencies that challenge existing approaches. We investigate these issues by introducing a novel semantic framework based on abstract interpretation, where the verification problem for VQCs can be formally defined, and its complexity analyzed. Finally, we demonstrate our approach on standard verification benchmarks.

http://arxiv.org/abs/2503.14836
On the Robustness Tradeoff in Fine-Tuning. (22%)
Kunyang Li; Jean-Charles Noirot Ferrand; Ryan Sheatsley; Blaine Hoak; Yohan Beugin; Eric Pauley; Patrick McDaniel
Fine-tuning has become the standard practice for adapting pre-trained models to downstream tasks. However, the impact on model robustness is not well understood. In this work, we characterize the robustness-accuracy trade-off in fine-tuning. We evaluate the robustness and accuracy of fine-tuned models over 6 benchmark datasets and 7 different fine-tuning strategies. We observe a consistent trade-off between adversarial robustness and accuracy. Peripheral updates such as BitFit are more effective for simple tasks -- over 75% above the average measured by the area under the Pareto frontiers on CIFAR-10 and CIFAR-100. In contrast, fine-tuning information-heavy layers, such as attention layers via Compacter, achieves a better Pareto frontier on more complex tasks -- 57.5% and 34.6% above the average on Caltech-256 and CUB-200, respectively. Lastly, we observe that the robustness of fine-tuning against out-of-distribution data closely tracks accuracy. These insights emphasize the need for robustness-aware fine-tuning to ensure reliable real-world deployments.

http://arxiv.org/abs/2507.10048
On the Efficiency of Training Robust Decision Trees. (22%)
Benedict Gerlach; Marie Anastacio; Holger H. Hoos
As machine learning gets adopted into the industry quickly, trustworthiness is increasingly in focus. Yet, efficiency and sustainability of robust training pipelines still have to be established. In this work, we consider a simple pipeline for training adversarially robust decision trees and investigate the efficiency of each step. Our pipeline consists of three stages. Firstly, we choose the perturbation size automatically for each dataset. For that, we introduce a simple algorithm, instead of relying on intuition or prior work. Moreover, we show that the perturbation size can be estimated from smaller models than the one intended for full training, and thus significant gains in efficiency can be achieved. Secondly, we train state-of-the-art adversarial training methods and evaluate them regarding both their training time and adversarial accuracy. Thirdly, we certify the robustness of each of the models thus obtained and investigate the time required for this. We find that verification time, which is critical to the efficiency of the full pipeline, is not correlated with training time.

http://arxiv.org/abs/2401.11406
Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts. (11%)
Kiyoon Kim; Shreyank N Gowda; Panagiotis Eustratiadis; Antreas Antoniou; Robert B Fisher
Despite recent advances in video action recognition achieving strong performance on existing benchmarks, these models often lack robustness when faced with natural distribution shifts between training and test data. We propose two novel evaluation methods to assess model resilience to such distribution disparity. One method uses two different datasets collected from different sources and uses one for training and validation, and the other for testing. More precisely, we created dataset splits of HMDB-51 or UCF-101 for training, and Kinetics-400 for testing, using the subset of the classes that are overlapping in both train and test datasets. The other proposed method extracts the feature mean of each class from the target evaluation dataset's training data (i.e. class prototype) and estimates test video prediction as a cosine similarity score between each sample to the class prototypes of each target class. This procedure does not alter model weights using the target dataset and it does not require aligning overlapping classes of two different datasets, thus is a very efficient method to test the model robustness to distribution shifts without prior knowledge of the target distribution. We address the robustness problem by adversarial augmentation training - generating augmented views of videos that are "hard" for the classification model by applying gradient ascent on the augmentation parameters - as well as "curriculum" scheduling the strength of the video augmentations. We experimentally demonstrate the superior performance of the proposed adversarial augmentation approach over baselines across three state-of-the-art action recognition models - TSM, Video Swin Transformer, and Uniformer. The presented work provides critical insight into model robustness to distribution shifts and presents effective techniques to enhance video action recognition performance in a real-world deployment.

http://arxiv.org/abs/2507.10239
Transferring Styles for Reduced Texture Bias and Improved Robustness in Semantic Segmentation Networks. (8%)
Ben Hamscher; Edgar Heinert; Annika Mütze; Kira Maag; Matthias Rottmann
Recent research has investigated the shape and texture biases of deep neural networks (DNNs) in image classification which influence their generalization capabilities and robustness. It has been shown that, in comparison to regular DNN training, training with stylized images reduces texture biases in image classification and improves robustness with respect to image corruptions. In an effort to advance this line of research, we examine whether style transfer can likewise deliver these two effects in semantic segmentation. To this end, we perform style transfer with style varying across artificial image areas. Those random areas are formed by a chosen number of Voronoi cells. The resulting style-transferred data is then used to train semantic segmentation DNNs with the objective of reducing their dependence on texture cues while enhancing their reliance on shape-based features. In our experiments, it turns out that in semantic segmentation, style transfer augmentation reduces texture bias and strongly increases robustness with respect to common image corruptions as well as adversarial attacks. These observations hold for convolutional neural networks and transformer architectures on the Cityscapes dataset as well as on PASCAL Context, showing the generality of the proposed method.

http://arxiv.org/abs/2410.04916
Defense-as-a-Service: Black-box Shielding against Backdoored Graph Models. (8%)
Xiao Yang; Kai Zhou; Yuni Lai; Gaolei Li
With the trend of large graph learning models, business owners tend to employ a model provided by a third party to deliver business services to users. However, these models might be backdoored, and malicious users can submit trigger-embedded inputs to manipulate the model predictions. Current graph backdoor defenses have several limitations: 1) depending on model-related details, 2) requiring additional model fine-tuning, and 3) relying upon extra explainability tools, all of which are infeasible under stringent privacy policies. To address those limitations, we propose GraphProt, which allows resource-constrained business owners to rely on third parties to avoid backdoor attacks on GNN-based graph classifiers. Our GraphProt is model-agnostic and only relies on the input graph. The key insight is to leverage subgraph information for prediction, thereby mitigating backdoor effects induced by triggers. GraphProt comprises two components: clustering-based trigger elimination and robust subgraph ensemble. Specifically, we first propose feature-topology clustering that aims to remove most of the anomalous subgraphs (triggers). Moreover, we design subgraph sampling strategies based on feature-topology clustering to build a robust classifier via majority vote. Experimental results across three backdoor attacks and six benchmark datasets demonstrate that GraphProt significantly reduces the backdoor attack success rate while preserving the model accuracy on regular graph classification tasks.

http://arxiv.org/abs/2507.10162
HASSLE: A Self-Supervised Learning Enhanced Hijacking Attack on Vertical Federated Learning. (8%)
Weiyang He; Chip-Hong Chang
Vertical Federated Learning (VFL) enables an orchestrating active party to perform a machine learning task by cooperating with passive parties that provide additional task-related features for the same training data entities. While prior research has leveraged the privacy vulnerability of VFL to compromise its integrity through a combination of label inference and backdoor attacks, their effectiveness is constrained by the low label inference precision and suboptimal backdoor injection conditions. To facilitate a more rigorous security evaluation on VFL without these limitations, we propose HASSLE, a hijacking attack framework composed of a gradient-direction-based label inference module and an adversarial embedding generation algorithm enhanced by self-supervised learning. HASSLE accurately identifies private samples associated with a targeted label using only a single known instance of that label. In the two-party scenario, it demonstrates strong performance with an attack success rate (ASR) of over 99% across four datasets, including both image and tabular modalities, and achieves 85% ASR on the more complex CIFAR-100 dataset. Evaluation of HASSLE against 8 potential defenses further highlights its significant threat while providing new insights into building a trustworthy VFL system.

http://arxiv.org/abs/2501.03940
Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection. (3%)
Pablo Miralles-González; Javier Huertas-Tato; Alejandro Martín; David Camacho
The rapid advancement in large language models (LLMs) has significantly enhanced their ability to generate coherent and contextually relevant text, raising concerns about the misuse of AI-generated content and making it critical to detect it. However, the task remains challenging, particularly in unseen domains or with unfamiliar LLMs. Leveraging LLM next-token distribution outputs offers a theoretically appealing approach for detection, as they encapsulate insights from the models' extensive pre-training on diverse corpora. Despite its promise, zero-shot methods that attempt to operationalize these outputs have met with limited success. We hypothesize that one of the problems is that they use the mean to aggregate next-token distribution metrics across tokens, when some tokens are naturally easier or harder to predict and should be weighted differently. Based on this idea, we propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length. Although not zero-shot, our method allows us to cache the last hidden states and next-token distribution metrics on disk, greatly reducing the training resource requirements. PAWN shows competitive and even better performance in-distribution than the strongest baselines (fine-tuned LMs) with a fraction of their trainable parameters. Our model also generalizes better to unseen domains and source models, with smaller variability in the decision boundary across distribution shifts. It is also more robust to adversarial attacks, and if the backbone has multilingual capabilities, it presents decent generalization to languages not seen during supervised training, with LLaMA3-1B reaching a mean macro-averaged F1 score of 81.46% in cross-validation with nine languages.

http://arxiv.org/abs/2507.14202
PRM-Free Security Alignment of Large Models via Red Teaming and Adversarial Training. (2%)
Pengfei Du
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet they pose significant security risks that threaten their safe deployment in critical domains. Current security alignment methodologies predominantly rely on Process Reward Models (PRMs) to evaluate intermediate reasoning steps, introducing substantial computational overhead and scalability constraints. This paper presents a novel PRM-free security alignment framework that leverages automated red teaming and adversarial training to achieve robust security guarantees while maintaining computational efficiency. Our approach systematically identifies vulnerabilities through sophisticated attack strategies including genetic algorithm optimization, multi-agent simulation, and advanced prompt mutation techniques. The framework enhances model robustness via targeted adversarial training with curriculum learning and adaptive regularization mechanisms. Comprehensive experimental evaluation across five state-of-the-art LLMs demonstrates that our method achieves superior security alignment performance compared to PRM-based approaches while reducing computational costs by 61\%. The framework incorporates transparent reporting and continuous audit mechanisms that enable iterative security improvement and regulatory compliance. Our contributions advance the field of efficient LLM security alignment by democratizing access to robust security measures for resource-constrained organizations and providing a scalable foundation for addressing evolving adversarial threats.

http://arxiv.org/abs/2507.10016
The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents. (1%)
Lixu Wang; Kaixiang Yao; Xinfeng Li; Dong Yang; Haoyang Li; Xiaofeng Wang; Wei Dong
Our research uncovers a novel privacy risk associated with multimodal large language models (MLLMs): the ability to infer sensitive personal attributes from audio data -- a technique we term audio private attribute profiling. This capability poses a significant threat, as audio can be covertly captured without direct interaction or visibility. Moreover, compared to images and text, audio carries unique characteristics, such as tone and pitch, which can be exploited for more detailed profiling. However, two key challenges exist in understanding MLLM-employed private attribute profiling from audio: (1) the lack of audio benchmark datasets with sensitive attribute annotations and (2) the limited ability of current MLLMs to infer such attributes directly from audio. To address these challenges, we introduce AP^2, an audio benchmark dataset that consists of two subsets collected and composed from real-world data, and both are annotated with sensitive attribute labels. Additionally, we propose Gifts, a hybrid multi-agent framework that leverages the complementary strengths of audio-language models (ALMs) and large language models (LLMs) to enhance inference capabilities. Gifts employs an LLM to guide the ALM in inferring sensitive attributes, then forensically analyzes and consolidates the ALM's inferences, overcoming severe hallucinations of existing ALMs in generating long-context responses. Our evaluations demonstrate that Gifts significantly outperforms baseline approaches in inferring sensitive attributes. Finally, we investigate model-level and data-level defense strategies to mitigate the risks of audio private attribute profiling. Our work validates the feasibility of audio-based privacy attacks using MLLMs, highlighting the need for robust defenses, and provides a dataset and framework to facilitate future research.

http://arxiv.org/abs/2507.10610
LaSM: Layer-wise Scaling Mechanism for Defending Pop-up Attack on GUI Agents. (73%)
Zihe Yan; Zhuosheng Zhang
Graphical user interface (GUI) agents built on multimodal large language models (MLLMs) have recently demonstrated strong decision-making abilities in screen-based interaction tasks. However, they remain highly vulnerable to pop-up-based environmental injection attacks, where malicious visual elements divert model attention and lead to unsafe or incorrect actions. Existing defense methods either require costly retraining or perform poorly under inductive interference. In this work, we systematically study how such attacks alter the attention behavior of GUI agents and uncover a layer-wise attention divergence pattern between correct and incorrect outputs. Based on this insight, we propose \textbf{LaSM}, a \textit{Layer-wise Scaling Mechanism} that selectively amplifies attention and MLP modules in critical layers. LaSM improves the alignment between model saliency and task-relevant regions without additional training. Extensive experiments across 12 types of pop-up perturbations and 4 different model backbones show that LaSM consistently enhances the defense success rate. When combined with prompt-level alerts, LaSM achieves over 98\% robustness even under strong inductive attacks. Our findings reveal that attention misalignment is a core vulnerability in MLLM agents and can be effectively addressed through selective layer-wise modulation.

http://arxiv.org/abs/2405.20725
GI-NAS: Boosting Gradient Inversion Attacks through Adaptive Neural Architecture Search. (15%)
Wenbo Yu; Hao Fang; Bin Chen; Xiaohang Sui; Chuan Chen; Hao Wu; Shu-Tao Xia; Ke Xu
Gradient Inversion Attacks invert the transmitted gradients in Federated Learning (FL) systems to reconstruct the sensitive data of local clients and have raised considerable privacy concerns. A majority of gradient inversion methods rely heavily on explicit prior knowledge (e.g., a well pre-trained generative model), which is often unavailable in realistic scenarios. This is because real-world client data distributions are often highly heterogeneous, domain-specific, and unavailable to attackers, making it impractical for attackers to obtain perfectly matched pre-trained models, which inevitably suffer from fundamental distribution shifts relative to target private data. To alleviate this issue, researchers have proposed to leverage the implicit prior knowledge of an over-parameterized network. However, they only utilize a fixed neural architecture for all the attack settings. This would hinder the adaptive use of implicit architectural priors and consequently limit the generalizability. In this paper, we further exploit such implicit prior knowledge by proposing Gradient Inversion via Neural Architecture Search (GI-NAS), which adaptively searches the network and captures the implicit priors behind neural architectures. Extensive experiments verify that our proposed GI-NAS can achieve superior attack performance compared to state-of-the-art gradient inversion methods, even under more practical settings with high-resolution images, large-sized batches, and advanced defense strategies. To the best of our knowledge, we are the first to successfully introduce NAS to the gradient inversion community. We believe that this work exposes critical vulnerabilities in real-world federated learning by demonstrating high-fidelity reconstruction of sensitive data without requiring domain-specific priors, forcing urgent reassessment of FL privacy safeguards.

http://arxiv.org/abs/2502.19537
No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms. (68%)
Joshua Kazdan; Abhay Puri; Rylan Schaeffer; Lisa Yu; Chris Cundy; Jason Stanley; Sanmi Koyejo; Krishnamurthy Dvijotham
Leading language model (LM) providers like OpenAI and Anthropic allow customers to fine-tune frontier LMs for specific use cases. To prevent abuse, these providers apply filters to block fine-tuning on overtly harmful data. In this setting, we make three contributions: First, while past work has shown that safety alignment is "shallow", we correspondingly demonstrate that existing fine-tuning attacks are shallow -- attacks target only the first several tokens of the model response, and consequently can be blocked by generating the first several response tokens with an aligned model. Second, we conceptually illustrate how to make attacks deeper by introducing a new fine-tuning attack that trains models to first refuse harmful requests before answering them; this "refuse-then-comply" strategy bypasses shallow defenses and produces harmful responses that evade output filters. Third, we demonstrate the potency of our new fine-tuning attack by jailbreaking both open-source models equipped with defenses and production models, achieving attack success rates of 57% and 72% against GPT-4o and Claude Haiku, respectively. Our attack received a $2000 bug bounty from OpenAI and was acknowledged as a vulnerability by Anthropic. Our work undermines the notion that models are safe because they initially refuse harmful requests and broadens awareness of the scope of attacks that face production fine-tuning APIs.

http://arxiv.org/abs/2507.14180
Digital Twin-Assisted Explainable AI for Robust Beam Prediction in mmWave MIMO Systems. (13%)
Nasir Khan; Asmaa Abdallah; Abdulkadir Celik; Ahmed M. Eltawil; Sinem Coleri
In line with the AI-native 6G vision, explainability and robustness are crucial for building trust and ensuring reliable performance in millimeter-wave (mmWave) systems. Efficient beam alignment is essential for initial access, but deep learning (DL) solutions face challenges, including high data collection overhead, hardware constraints, lack of explainability, and susceptibility to adversarial attacks. This paper proposes a robust and explainable DL-based beam alignment engine (BAE) for mmWave multiple-input multiple output (MIMO) systems. The BAE uses received signal strength indicator (RSSI) measurements from wide beams to predict the best narrow beam, reducing the overhead of exhaustive beam sweeping. To overcome the challenge of real-world data collection, this work leverages a site-specific digital twin (DT) to generate synthetic channel data closely resembling real-world environments. A model refinement via transfer learning is proposed to fine-tune the pre-trained model residing in the DT with minimal real-world data, effectively bridging mismatches between the digital replica and real-world environments. To reduce beam training overhead and enhance transparency, the framework uses deep Shapley additive explanations (SHAP) to rank input features by importance, prioritizing key spatial directions and minimizing beam sweeping. It also incorporates the Deep k-nearest neighbors (DkNN) algorithm, providing a credibility metric for detecting out-of-distribution inputs and ensuring robust, transparent decision-making. Experimental results show that the proposed framework reduces real-world data needs by 70%, beam training overhead by 62%, and improves outlier detection robustness by up to 8.5x, achieving near-optimal spectral efficiency and transparent decision making compared to traditional softmax based DL models.

http://arxiv.org/abs/2507.09406
Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers. (10%)
Santhosh Kumar Ravindran
Large language models (LLMs) aligned for safety through techniques like reinforcement learning from human feedback (RLHF) often exhibit emergent deceptive behaviors, where outputs appear compliant but subtly mislead or omit critical information. This paper introduces adversarial activation patching, a novel mechanistic interpretability framework that leverages activation patching as an adversarial tool to induce, detect, and mitigate such deception in transformer-based models. By sourcing activations from "deceptive" prompts and patching them into safe forward passes at specific layers, we simulate vulnerabilities and quantify deception rates. Through toy neural network simulations across multiple scenarios (e.g., 1000 trials per setup), we demonstrate that adversarial patching increases deceptive outputs to 23.9% from a 0% baseline, with layer-specific variations supporting our hypotheses. We propose six hypotheses, including transferability across models, exacerbation in multimodal settings, and scaling effects. An expanded literature review synthesizes over 20 key works in interpretability, deception, and adversarial attacks. Mitigation strategies, such as activation anomaly detection and robust fine-tuning, are detailed, alongside ethical considerations and future research directions. This work advances AI safety by highlighting patching's dual-use potential and provides a roadmap for empirical studies on large-scale models.

http://arxiv.org/abs/2507.09095
On the Fragility of Multimodal Perception to Temporal Misalignment in Autonomous Driving. (1%)
Md Hasan Shahriar; Md Mohaimin Al Barat; Harshavardhan Sundar; Naren Ramakrishnan; Y. Thomas Hou; Wenjing Lou
Multimodal fusion (MMF) plays a critical role in the perception of autonomous driving, which primarily fuses camera and LiDAR streams for a comprehensive and efficient scene understanding. However, its strict reliance on precise temporal synchronization exposes it to new vulnerabilities. In this paper, we introduce DejaVu, a novel attack that exploits network-induced delays to create subtle temporal misalignments across sensor streams, severely degrading downstream MMF-based perception tasks. Our comprehensive attack analysis across different models and datasets reveals these sensors' task-specific imbalanced sensitivities: object detection is overly dependent on LiDAR inputs while object tracking is highly reliant on the camera inputs. Consequently, with a single-frame LiDAR delay, an attacker can reduce the car detection mAP by up to 88.5%, while with a three-frame camera delay, multiple object tracking accuracy (MOTA) for car drops by 73%. To detect such attacks, we propose AION, a defense patch that can work alongside the existing perception model to monitor temporal alignment through cross-modal temporal consistency. AION leverages multimodal shared representation learning and dynamic time warping to determine the path of temporal alignment and calculate anomaly scores based on the alignment. Our thorough evaluation of AION shows it achieves AUROC scores of 0.92-0.98 with low false positives across datasets and model architectures, demonstrating it as a robust and generalized defense against the temporal misalignment attacks.

http://arxiv.org/abs/2507.06261
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. (99%)
Gheorghe Comanici; Eric Bieber; Mike Schaekermann; Ice Pasupat; Noveen Sachdeva; Inderjit Dhillon; Marcel Blistein; Ori Ram; Dan Zhang; Evan Rosen; Luke Marris; Sam Petulla; Colin Gaffney; Asaf Aharoni; Nathan Lintz; Tiago Cardal Pais; Henrik Jacobsson; Idan Szpektor; Nan-Jiang Jiang; Krishna Haridasan; Ahmed Omran; Nikunj Saunshi; Dara Bahri; Gaurav Mishra; Eric Chu; Toby Boyd; Brad Hekman; Aaron Parisi; Chaoyi Zhang; Kornraphop Kawintiranon; Tania Bedrax-Weiss; Oliver Wang; Ya Xu; Ollie Purkiss; Uri Mendlovic; Ilaï Deutel; Nam Nguyen; Adam Langley; Flip Korn; Lucia Rossazza; Alexandre Ramé; Sagar Waghmare; Helen Miller; Vaishakh Keshava; Ying Jian; Xiaofan Zhang; Raluca Ada Popa; Kedar Dhamdhere; Blaž Bratanič; Kyuyeun Kim; Terry Koo; Ferran Alet; Yi-ting Chen; Arsha Nagrani; Hannah Muckenhirn; Zhiyuan Zhang; Corbin Quick; Filip Pavetić; Duc Dung Nguyen; Joao Carreira; Michael Elabd; Haroon Qureshi; Fabian Mentzer; Yao-Yuan Yang; Danielle Eisenbud; Anmol Gulati; Ellie Talius; Eric Ni; Sahra Ghalebikesabi; Edouard Yvinec; Alaa Saade; Thatcher Ulrich; Lorenzo Blanco; Dan A. Calian; Muhuan Huang; Aäron van den Oord; Naman Goyal; Terry Chen; Praynaa Rawlani; Christian Schallhart; Swachhand Lokhande; Xianghong Luo; Jyn Shan; Ceslee Montgomery; Victoria Krakovna; Federico Piccinini; Omer Barak; Jingyu Cui; Yiling Jia; Mikhail Dektiarev; Alexey Kolganov; Shiyu Huang; Zhe Chen; Xingyu Wang; Jessica Austin; Boursac Peter de; Evgeny Sluzhaev; Frank Ding; Huijian Li; Surya Bhupatiraju; Mohit Agarwal; Sławek Kwasiborski; Paramjit Sandhu; Patrick Siegler; Ahmet Iscen; Eyal Ben-David; Shiraz Butt; Miltos Allamanis; Seth Benjamin; Robert Busa-Fekete; Felix Hernandez-Campos; Sasha Goldshtein; Matt Dibb; Weiyang Zhang; Annie Marsden; Carey Radebaugh; Stephen Roller; Abhishek Nayyar; Jacob Austin; Tayfun Terzi; Bhargav Kanagal Shamanna; Pete Shaw; Aayush Singh; Florian Luisier; Artur Mendonça; Vaibhav Aggarwal; Larisa Markeeva; Claudio Fantacci; Sergey Brin; HyunJeong Choe; Guanyu Wang; Hartwig Adam; Avigail Dabush; Tatsuya Kiyono; Eyal Marcus; Jeremy Cole; Theophane Weber; Hongrae Lee; Ronny Huang; Alex Muzio; Leandro Kieliger; Maigo Le; Courtney Biles; Long Le; Archit Sharma; Chengrun Yang; Avery Lamp; Dave Dopson; Nate Hurley; Katrina Xinyi Xu; Zhihao Shan; Shuang Song; Jiewen Tan; Alexandre Senges; George Zhang; Chong You; Yennie Jun; David Raposo; Susanna Ricco; Xuan Yang; Weijie Chen; Prakhar Gupta; Arthur Szlam; Kevin Villela; Chun-Sung Ferng; Daniel Kasenberg; Chen Liang; Rui Zhu; Arunachalam Narayanaswamy; Florence Perot; Paul Pucciarelli; Anna Shekhawat; Alexey Stern; Rishikesh Ingale; Stefani Karp; Sanaz Bahargam; Adrian Goedeckemeyer; Jie Han; Sicheng Li; Andrea Tacchetti; Dian Yu; Abhishek Chakladar; Zhiying Zhang; Mona El Mahdy; Xu Gao; Dale Johnson; Samrat Phatale; AJ Piergiovanni; Hyeontaek Lim; Clement Farabet; Carl Lebsack; Theo Guidroz; John Blitzer; Nico Duduta; David Madras; Steve Li; Dincklage Daniel von; Xin Li; Mahdis Mahdieh; George Tucker; Ganesh Jawahar; Owen Xiao; Danny Tarlow; Robert Geirhos; Noam Velan; Daniel Vlasic; Kalesha Bullard; SK Park; Nishesh Gupta; Kellie Webster; Ayal Hitron; Jieming Mao; Julian Eisenschlos; Laurel Prince; Nina D'Souza; Kelvin Zheng; Sara Nasso; Gabriela Botea; Carl Doersch; Caglar Unlu; Chris Alberti; Alexey Svyatkovskiy; Ankita Goel; Krzysztof Choromanski; Pan-Pan Jiang; Richard Nguyen; Four Flynn; Daria Ćurko; Peter Chen; Nicholas Roth; Kieran Milan; Caleb Habtegebriel; Shashi Narayan; Michael Moffitt; Jake Marcus; Thomas Anthony; Brendan McMahan; Gowoon Cheon; Ruibo Liu; Megan Barnes; Lukasz Lew; Rebeca Santamaria-Fernandez; Mayank Upadhyay; Arjun Akula; Arnar Mar Hrafnkelsson; Alvaro Caceres; Andrew Bunner; Michal Sokolik; Subha Puttagunta; Lawrence Moore; Berivan Isik; Jay Hartford; Lawrence Chan; Pradeep Shenoy; Dan Holtmann-Rice; Jane Park; Fabio Viola; Alex Salcianu; Sujeevan Rajayogam; Ian Stewart-Binks; Zelin Wu; Richard Everett; Xi Xiong; Pierre-Antoine Manzagol; Gary Leung; Carl Saroufim; Bo Pang; Dawid Wegner; George Papamakarios; Jennimaria Palomaki; Helena Pankov; Guangda Lai; Guilherme Tubone; Shubin Zhao; Theofilos Strinopoulos; Seth Neel; Mingqiu Wang; Joe Kelley; Li Li; Pingmei Xu; Anitha Vijayakumar; Andrea D'olimpio; Omer Levy; Massimo Nicosia; Grigory Rozhdestvenskiy; Ni Lao; Sirui Xie; Yash Katariya; Jon Simon; Sanjiv Kumar; Florian Hartmann; Michael Kilgore; Jinhyuk Lee; Aroma Mahendru; Roman Ring; Tom Hennigan; Fiona Lang; Colin Cherry; David Steiner; Dawsen Hwang; Ray Smith; Pidong Wang; Jeremy Chen; Ming-Hsuan Yang; Sam Kwei; Philippe Schlattner; Donnie Kim; Ganesh Poomal Girirajan; Nikola Momchev; Ayushi Agarwal; Xingyi Zhou; Ilkin Safarli; Zachary Garrett; AJ Pierigiovanni; Sarthak Jauhari; Alif Raditya Rochman; Shikhar Vashishth; Quan Yuan; Christof Angermueller; Jon Blanton; Xinying Song; Nitesh Bharadwaj Gundavarapu; Thi Avrahami; Maxine Deines; Subhrajit Roy; Manish Gupta; Christopher Semturs; Shobha Vasudevan; Aditya Srikanth Veerubhotla; Shriya Sharma; Josh Jacob; Zhen Yang; Andreas Terzis; Dan Karliner; Auriel Wright; Tania Rojas-Esponda; Ashley Brown; Abhijit Guha Roy; Pawan Dogra; Andrei Kapishnikov; Peter Young; Wendy Kan; Vinodh Kumar Rajendran; Maria Ivanova; Salil Deshmukh; Chia-Hua Ho; Mike Kwong; Stav Ginzburg; Annie Louis; KP Sawhney; Slav Petrov; Jing Xie; Yunfei Bai; Georgi Stoyanov; Alex Fabrikant; Rajesh Jayaram; Yuqi Li; Joe Heyward; Justin Gilmer; Yaqing Wang; Radu Soricut; Luyang Liu; Qingnan Duan; Jamie Hayes; Maura O'Brien; Gaurav Singh Tomar; Sivan Eiger; Bahar Fatemi; Jeffrey Hui; Catarina Barros; Adaeze Chukwuka; Alena Butryna; Saksham Thakur; Austin Huang; Zhufeng Pan; Haotian Tang; Serkan Cabi; Tulsee Doshi; Michiel Bakker; Sumit Bagri; Ruy Ley-Wild; Adam Lelkes; Jennie Lees; Patrick Kane; David Greene; Shimu Wu; Jörg Bornschein; Gabriela Surita; Sarah Hodkinson; Fangtao Li; Chris Hidey; Sébastien Pereira; Sean Ammirati; Phillip Lippe; Adam Kraft; Pu Han; Sebastian Gerlach; Zifeng Wang; Liviu Panait; Feng Han; Brian Farris; Yingying Bi; Hannah DeBalsi; Miaosen Wang; Gladys Tyen; James Cohan; Susan Zhang; Jarred Barber; Da-Woon Chung; Jaeyoun Kim; Markus Kunesch; Steven Pecht; Nami Akazawa; Abe Friesen; James Lyon; Ali Eslami; Junru Wu; Jie Tan; Yue Song; Ravi Kumar; Chris Welty; Ilia Akolzin; Gena Gibson; Sean Augenstein; Arjun Pillai; Nancy Yuen; Du Phan; Xin Wang; Iain Barr; Heiga Zen; Nan Hua; Casper Liu; Jilei Jerry Wang; Tanuj Bhatia; Hao Xu; Oded Elyada; Pushmeet Kohli; Mirek Olšák; Ke Chen; Azalia Mirhoseini; Noam Shazeer; Shoshana Jakobovits; Maggie Tran; Nolan Ramsden; Tarun Bharti; Fred Alcober; Yunjie Li; Shilpa Shetty; Jing Chen; Dmitry Kalashnikov; Megha Nawhal; Sercan Arik; Hanwen Chen; Michiel Blokzijl; Shubham Gupta; James Rubin; Rigel Swavely; Sophie Bridgers; Ian Gemp; Chen Su; Arun Suggala; Juliette Pluto; Mary Cassin; Alain Vaucher; Kaiyang Ji; Jiahao Cai; Andrew Audibert; Animesh Sinha; David Tian; Efrat Farkash; Amy Hua; Jilin Chen; Duc-Hieu Tran; Edward Loper; Nicole Brichtova; Lara McConnaughey; Ballie Sandhu; Robert Leland; Doug DeCarlo; Andrew Over; James Huang; Xing Wu; Connie Fan; Eric Li; Yun Lei; Deepak Sharma; Cosmin Paduraru; Luo Yu; Matko Bošnjak; Phuong Dao; Min Choi; Sneha Kudugunta; Jakub Adamek; Carlos Guía; Ali Khodaei; Jie Feng; Wenjun Zeng; David Welling; Sandeep Tata; Christina Butterfield; Andrey Vlasov; Seliem El-Sayed; Swaroop Mishra; Tara Sainath; Shentao Yang; RJ Skerry-Ryan; Jeremy Shar; Robert Berry; Arunkumar Rajendran; Arun Kandoor; Andrea Burns; Deepali Jain; Tom Stone; Wonpyo Park; Shibo Wang; Albin Cassirer; Guohui Wang; Hayato Kobayashi; Sergey Rogulenko; Vineetha Govindaraj; Mikołaj Rybiński; Nadav Olmert; Colin Evans; Po-Sen Huang; Kelvin Xu; Premal Shah; Terry Thurk; Caitlin Sikora; Mu Cai; Jin Xie; Elahe Dabir; Saloni Shah; Norbert Kalb; Carrie Zhang; Shruthi Prabhakara; Amit Sabne; Artiom Myaskovsky; Vikas Raunak; Blanca Huergo; Behnam Neyshabur; Jon Clark; Ye Zhang; Shankar Krishnan; Eden Cohen; Dinesh Tewari; James Lottes; Yumeya Yamamori; Hui Elena Li; Mohamed Elhawaty; Ada Maksutaj Oflazer; Adrià Recasens; Sheryl Luo; Duy Nguyen; Taylor Bos; Kalyan Andra; Ana Salazar; Ed Chi; Jeongwoo Ko; Matt Ginsberg; Anders Andreassen; Anian Ruoss; Todor Davchev; Elnaz Davoodi; Chenxi Liu; Min Kim; Santiago Ontanon; Chi Ming To; Dawei Jia; Rosemary Ke; Jing Wang; Anna Korsun; Moran Ambar; Ilya Kornakov; Irene Giannoumis; Toni Creswell; Denny Zhou; Yi Su; Ishaan Watts; Aleksandr Zaks; Evgenii Eltyshev; Ziqiang Feng; Sidharth Mudgal; Alex Kaskasoli; Juliette Love; Kingshuk Dasgupta; Sam Shleifer; Richard Green; Sungyong Seo; Chansoo Lee; Dale Webster; Prakash Shroff; Ganna Raboshchuk; Isabel Leal; James Manyika; Sofia Erell; Daniel Murphy; Zhisheng Xiao; Anton Bulyenov; Julian Walker; Mark Collier; Matej Kastelic; Nelson George; Sushant Prakash; Sailesh Sidhwani; Alexey Frolov; Steven Hansen; Petko Georgiev; Tiberiu Sosea; Chris Apps; Aishwarya Kamath; David Reid; Emma Cooney; Charlotte Magister; Oriana Riva; Alec Go; Pu-Chin Chen; Sebastian Krause; Nir Levine; Marco Fornoni; Ilya Figotin; Nick Roy; Parsa Mahmoudieh; Vladimir Magay; Mukundan Madhavan; Jin Miao; Jianmo Ni; Yasuhisa Fujii; Ian Chou; George Scrivener; Zak Tsai; Siobhan Mcloughlin; Jeremy Selier; Sandra Lefdal; Jeffrey Zhao; Abhijit Karmarkar; Kushal Chauhan; Shivanker Goel; Zhaoyi Zhang; Vihan Jain; Parisa Haghani; Mostafa Dehghani; Jacob Scott; Erin Farnese; Anastasija Ilić; Steven Baker; Julia Pawar; Li Zhong; Josh Camp; Yoel Zeldes; Shravya Shetty; Anand Iyer; Vít Listík; Jiaxian Guo; Luming Tang; Mark Geller; Simon Bucher; Yifan Ding; Hongzhi Shi; Carrie Muir; Dominik Grewe; Ramy Eskander; Octavio Ponce; Boqing Gong; Derek Gasaway; Samira Khan; Umang Gupta; Angelos Filos; Weicheng Kuo; Klemen Kloboves; Jennifer Beattie; Christian Wright; Leon Li; Alicia Jin; Sandeep Mariserla; Miteyan Patel; Jens Heitkaemper; Dilip Krishnan; Vivek Sharma; David Bieber; Christian Frank; John Lambert; Paul Caron; Martin Polacek; Mai Giménez; Himadri Choudhury; Xing Yu; Sasan Tavakkol; Arun Ahuja; Franz Och; Rodolphe Jenatton; Wojtek Skut; Bryan Richter; David Gaddy; Andy Ly; Misha Bilenko; Megh Umekar; Ethan Liang; Martin Sevenich; Mandar Joshi; Hassan Mansoor; Rebecca Lin; Sumit Sanghai; Abhimanyu Singh; Xiaowei Li; Sudheendra Vijayanarasimhan; Zaheer Abbas; Yonatan Bitton; Hansa Srinivasan; Manish Reddy Vuyyuru; Alexander Frömmgen; Yanhua Sun; Ralph Leith; Alfonso Castaño; DJ Strouse; Le Yan; Austin Kyker; Satish Kambala; Mary Jasarevic; Thibault Sellam; Chao Jia; Alexander Pritzel; Raghavender R; Huizhong Chen; Natalie Clay; Sudeep Gandhe; Sean Kirmani; Sayna Ebrahimi; Hannah Kirkwood; Jonathan Mallinson; Chao Wang; Adnan Ozturel; Kuo Lin; Shyam Upadhyay; Vincent Cohen-Addad; Sean Purser-haskell; Yichong Xu; Ebrahim Songhori; Babi Seal; Alberto Magni; Almog Gueta; Tingting Zou; Guru Guruganesh; Thais Kagohara; Hung Nguyen; Khalid Salama; Alejandro Cruzado Ruiz; Justin Frye; Zhenkai Zhu; Matthias Lochbrunner; Simon Osindero; Wentao Yuan; Lisa Lee; Aman Prasad; Lam Nguyen Thiet; Daniele Calandriello; Victor Stone; Qixuan Feng; Han Ke; Maria Voitovich; Geta Sampemane; Lewis Chiang; Ling Wu; Alexander Bykovsky; Matt Young; Luke Vilnis; Ishita Dasgupta; Aditya Chawla; Qin Cao; Bowen Liang; Daniel Toyama; Szabolcs Payrits; Anca Stefanoiu; Dimitrios Vytiniotis; Ankesh Anand; Tianxiao Shen; Blagoj Mitrevski; Michael Tschannen; Sreenivas Gollapudi; Aishwarya P S; José Leal; Zhe Shen; Han Fu; Wei Wang; Arvind Kannan; Doron Kukliansky; Sergey Yaroshenko; Svetlana Grant; Umesh Telang; David Wood; Alexandra Chronopoulou; Alexandru Ţifrea; Tao Zhou; Tony Tu\'ân Nguy\~ên; Muge Ersoy; Anima Singh; Meiyan Xie; Emanuel Taropa; Woohyun Han; Eirikur Agustsson; Andrei Sozanschi; Hui Peng; Alex Chen; Yoel Drori; Efren Robles; Yang Gao; Xerxes Dotiwalla; Ying Chen; Anudhyan Boral; Alexei Bendebury; John Nham; Chris Tar; Luis Castro; Jiepu Jiang; Canoee Liu; Felix Halim; Jinoo Baek; Andy Wan; Jeremiah Liu; Yuan Cao; Shengyang Dai; Trilok Acharya; Ruoxi Sun; Fuzhao Xue; Saket Joshi; Morgane Lustman; Yongqin Xian; Rishabh Joshi; Deep Karkhanis; Nora Kassner; Jamie Hall; Xiangzhuo Ding; Gan Song; Gang Li; Chen Zhu; Yana Kulizhskaya; Bin Ni; Alexey Vlaskin; Solomon Demmessie; Lucio Dery; Salah Zaiem; Yanping Huang; Cindy Fan; Felix Gimeno; Ananth Balashankar; Koji Kojima; Hagai Taitelbaum; Maya Meng; Dero Gharibian; Sahil Singla; Wei Chen; Ambrose Slone; Guanjie Chen; Sujee Rajayogam; Max Schumacher; Suyog Kotecha; Rory Blevins; Qifei Wang; Mor Hazan Taege; Alex Morris; Xin Liu; Fayaz Jamil; Richard Zhang; Pratik Joshi; Ben Ingram; Tyler Liechty; Ahmed Eleryan; Scott Baird; Alex Grills; Gagan Bansal; Shan Han; Kiran Yalasangi; Shawn Xu; Majd Al Merey; Isabel Gao; Felix Weissenberger; Igor Karpov; Robert Riachi; Ankit Anand; Gautam Prasad; Kay Lamerigts; Reid Hayes; Jamie Rogers; Mandy Guo; Ashish Shenoy; Qiong Q Hu; Kyle He; Yuchen Liu; Polina Zablotskaia; Sagar Gubbi; Yifan Chang; Jay Pavagadhi; Kristian Kjems; Archita Vadali; Diego Machado; Yeqing Li; Renshen Wang; Dipankar Ghosh; Aahil Mehta; Dana Alon; George Polovets; Alessio Tonioni; Nate Kushman; Joel D'sa; Lin Zhuo; Allen Wu; Rohin Shah; John Youssef; Jiayu Ye; Justin Snyder; Karel Lenc; Senaka Buthpitiya; Matthew Tung; Jichuan Chang; Tao Chen; David Saxton; Jenny Lee; Lydia Lihui Zhang; James Qin; Prabakar Radhakrishnan; Maxwell Chen; Piotr Ambroszczyk; Metin Toksoz-Exley; Yan Zhong; Nitzan Katz; Brendan O'Donoghue; Glehn Tamara von; Adi Gerzi Rosenthal; Aga Świetlik; Xiaokai Zhao; Nick Fernando; Jinliang Wei; Jieru Mei; Sergei Vassilvitskii; Diego Cedillo; Pranjal Awasthi; Hui Zheng; Koray Kavukcuoglu; Itay Laish; Joseph Pagadora; Marc Brockschmidt; Christopher A. Choquette-Choo; Arunkumar Byravan; Yifeng Lu; Xu Chen; Mia Chen; Kenton Lee; Rama Pasumarthi; Sijal Bhatnagar; Aditya Shah; Qiyin Wu; Zhuoyuan Chen; Zack Nado; Bartek Perz; Zixuan Jiang; David Kao; Ganesh Mallya; Nino Vieillard; Lantao Mei; Sertan Girgin; Mandy Jordan; Yeongil Ko; Alekh Agarwal; Yaxin Liu; Yasemin Altun; Liedekerke Raoul de; Anastasios Kementsietsidis; Daiyi Peng; Dangyi Liu; Utku Evci; Peter Humphreys; Austin Tarango; Xiang Deng; Yoad Lewenberg; Kevin Aydin; Chengda Wu; Bhavishya Mittal; Tsendsuren Munkhdalai; Kleopatra Chatziprimou; Rodrigo Benenson; Uri First; Xiao Ma; Jinning Li; Armand Joulin; Hamish Tomlinson; Tingnan Zhang; Milad Nasr; Zhi Hong; Michaël Sander; Lisa Anne Hendricks; Anuj Sharma; Andrew Bolt; Eszter Vértes; Jiri Simsa; Tomer Levinboim; Olcan Sercinoglu; Divyansh Shukla; Austin Wu; Craig Swanson; Danny Vainstein; Fan Bu; Bo Wang; Ryan Julian; Charles Yoon; Sergei Lebedev; Antonious Girgis; Bernd Bandemer; David Du; Todd Wang; Xi Chen; Ying Xiao; Peggy Lu; Natalie Ha; Vlad Ionescu; Simon Rowe; Josip Matak; Federico Lebron; Andreas Steiner; Lalit Jain; Manaal Faruqui; Nicolas Lacasse; Georgie Evans; Neesha Subramaniam; Dean Reich; Giulia Vezzani; Aditya Pandey; Joe Stanton; Tianhao Zhou; Liam McCafferty; Henry Griffiths; Verena Rieser; Soheil Hassas Yeganeh; Eleftheria Briakou; Lu Huang; Zichuan Wei; Liangchen Luo; Erik Jue; Gabby Wang; Victor Cotruta; Myriam Khan; Jongbin Park; Qiuchen Guo; Peiran Li; Rong Rong; Diego Antognini; Anastasia Petrushkina; Chetan Tekur; Eli Collins; Parul Bhatia; Chester Kwak; Wenhu Chen; Arvind Neelakantan; Immanuel Odisho; Sheng Peng; Vincent Nallatamby; Vaibhav Tulsyan; Fabian Pedregosa; Peng Xu; Raymond Lin; Yulong Wang; Emma Wang; Sholto Douglas; Reut Tsarfaty; Elena Gribovskaya; Renga Aravamudhan; Manu Agarwal; Mara Finkelstein; Qiao Zhang; Elizabeth Cole; Phil Crone; Sarmishta Velury; Anil Das; Chris Sauer; Luyao Xu; Danfeng Qin; Chenjie Gu; Dror Marcus; CJ Zheng; Gansbeke Wouter Van; Sobhan Miryoosefi; Haitian Sun; YaGuang Li; Charlie Chen; Jae Yoo; Pavel Dubov; Alex Tomala; Adams Yu; Paweł Wesołowski; Alok Gunjan; Eddie Cao; Jiaming Luo; Nikhil Sethi; Arkadiusz Socala; Laura Graesser; Tomas Kocisky; Arturo BC; Minmin Chen; Edward Lee; Sophie Wang; Weize Kong; Qiantong Xu; Nilesh Tripuraneni; Yiming Li; Xinxin Yu; Allen Porter; Paul Voigtlaender; Biao Zhang; Arpi Vezer; Sarah York; Qing Wei; Geoffrey Cideron; Mark Kurzeja; Seungyeon Kim; Benny Li; Angéline Pouget; Hyo Lee; Kaspar Daugaard; Yang Li; Dave Uthus; Aditya Siddhant; Paul Cavallaro; Sriram Ganapathy; Maulik Shah; Rolf Jagerman; Jeff Stanway; Piermaria Mendolicchio; Li Xiao; Kayi Lee; Tara Thompson; Shubham Milind Phal; Jason Chase; Sun Jae Lee; Adrian N Reyes; Disha Shrivastava; Zhen Qin; Roykrong Sukkerd; Seth Odoom; Lior Madmoni; John Aslanides; Jonathan Herzig; Elena Pochernina; Sheng Zhang; Parker Barnes; Daisuke Ikeda; Qiujia Li; Shuo-yiin Chang; Shakir Mohamed; Jim Sproch; Richard Powell; Bidisha Samanta; Domagoj Ćevid; Anton Kovsharov; Shrestha Basu Mallick; Srinivas Tadepalli; Anne Zheng; Kareem Ayoub; Andreas Noever; Christian Reisswig; Zhuo Xu; Junhyuk Oh; Martin Matysiak; Tim Blyth; Shereen Ashraf; Julien Amelot; Boone Severson; Michele Bevilacqua; Motoki Sano; Ethan Dyer; Ofir Roval; Anu Sinha; Yin Zhong; Sagi Perel; Tea Sabolić; Johannes Mauerer; Willi Gierke; Mauro Verzetti; Rodrigo Cabrera; Alvin Abdagic; Steven Hemingray; Austin Stone; Jong Lee; Farooq Ahmad; Karthik Raman; Lior Shani; Jonathan Lai; Orhan Firat; Nathan Waters; Eric Ge; Mo Shomrat; Himanshu Gupta; Rajeev Aggarwal; Tom Hudson; Bill Jia; Simon Baumgartner; Palak Jain; Joe Kovac; Junehyuk Jung; Ante Žužul; Will Truong; Morteza Zadimoghaddam; Songyou Peng; Marco Liang; Rachel Sterneck; Balaji Lakshminarayanan; Machel Reid; Oliver Woodman; Tong Zhou; Jianling Wang; Vincent Coriou; Arjun Narayanan; Jay Hoover; Yenai Ma; Apoorv Jindal; Clayton Sanford; Doug Reid; Swaroop Ramaswamy; Alex Kurakin; Roland Zimmermann; Yana Lunts; Dragos Dena; Zalán Borsos; Vered Cohen; Shujian Zhang; Will Grathwohl; Robert Dadashi; Morgan Redshaw; Joshua Kessinger; Julian Odell; Silvano Bonacina; Zihang Dai; Grace Chen; Ayush Dubey; Pablo Sprechmann; Mantas Pajarskas; Wenxuan Zhou; Niharika Ahuja; Tara Thomas; Martin Nikoltchev; Matija Kecman; Bharath Mankalale; Andrey Ryabtsev; Jennifer She; Christian Walder; Jiaming Shen; Lu Li; Carolina Parada; Sheena Panthaplackel; Okwan Kwon; Matt Lawlor; Utsav Prabhu; Yannick Schroecker; Marc'aurelio Ranzato; Pete Blois; Iurii Kemaev; Ting Yu; Dmitry Lepikhin; Hao Xiong; Sahand Sharifzadeh; Oleaser Johnson; Jeremiah Willcock; Rui Yao; Greg Farquhar; Sujoy Basu; Hidetoshi Shimokawa; Nina Anderson; Haiguang Li; Khiem Pham; Yizhong Liang; Sebastian Borgeaud; Alexandre Moufarek; Hideto Kazawa; Blair Kutzman; Marcin Sieniek; Sara Smoot; Ruth Wang; Natalie Axelsson; Nova Fallen; Prasha Sundaram; Yuexiang Zhai; Varun Godbole; Petros Maniatis; Alek Wang; Ilia Shumailov; Santhosh Thangaraj; Remi Crocker; Nikita Gupta; Gang Wu; Phil Chen; Gellért Weisz; Celine Smith; Mojtaba Seyedhosseini; Boya Fang; Xiyang Luo; Roey Yogev; Zeynep Cankara; Andrew Hard; Helen Ran; Rahul Sukthankar; George Necula; Gaël Liu; Honglong Cai; Praseem Banzal; Daniel Keysers; Sanjay Ghemawat; Connie Tao; Emma Dunleavy; Aditi Chaudhary; Wei Li; Maciej Mikuła; Chen-Yu Lee; Tiziana Refice; Krishna Somandepalli; Alexandre Fréchette; Dan Bahir; John Karro; Keith Rush; Sarah Perrin; Bill Rosgen; Xiaomeng Yang; Clara Huiyi Hu; Mahmoud Alnahlawi; Justin Mao-Jones; Roopal Garg; Hoang Nguyen; Bat-Orgil Batsaikhan; Iñaki Iturrate; Anselm Levskaya; Avi Singh; Ashyana Kachra; Tony Lu; Denis Petek; Zheng Xu; Mark Graham; Lukas Zilka; Yael Karov; Marija Kostelac; Fangyu Liu; Yaohui Guo; Weiyue Wang; Bernd Bohnet; Emily Pitler; Tony Bruguier; Keisuke Kinoshita; Chrysovalantis Anastasiou; Nilpa Jha; Ting Liu; Jerome Connor; Phil Wallis; Philip Pham; Eric Bailey; Shixin Li; Heng-Tze Cheng; Sally Ma; Haiqiong Li; Akanksha Maurya; Kate Olszewska; Manfred Warmuth; Christy Koh; Dominik Paulus; Siddhartha Reddy Jonnalagadda; Enrique Piqueras; Ali Elqursh; Geoff Brown; Hadar Shemtov; Loren Maggiore; Fei Xia; Ryan Foley; Beka Westberg; George van den Driessche; Livio Baldini Soares; Arjun Kar; Michael Quinn; Siqi Zuo; Jialin Wu; Kyle Kastner; Anna Bortsova; Aijun Bai; Ales Mikhalap; Luowei Zhou; Jennifer Brennan; Vinay Ramasesh; Honglei Zhuang; John Maggs; Johan Schalkwyk; Yuntao Xu; Hui Huang; Andrew Howard; Sasha Brown; Linting Xue; Gloria Shen; Brian Albert; Neha Jha; Daniel Zheng; Varvara Krayvanova; Spurthi Amba Hombaiah; Olivier Lacombe; Gautam Vasudevan; Dan Graur; Tian Xie; Meet Gandhi; Bangju Wang; Dustin Zelle; Harman Singh; Dahun Kim; Sébastien Cevey; Victor Ungureanu; Natasha Noy; Fei Liu; Annie Xie; Fangxiaoyu Feng; Katerina Tsihlas; Daniel Formoso; Neera Vats; Quentin Wellens; Yinan Wang; Niket Kumar Bhumihar; Samrat Ghosh; Matt Hoffman; Tom Lieber; Oran Lang; Kush Bhatia; Tom Paine; Aroonalok Pyne; Ronny Votel; Madeleine Clare Elish; Benoit Schillings; Alex Panagopoulos; Haichuan Yang; Adam Raveret; Zohar Yahav; Shuang Liu; Dalia El Badawy; Nishant Agrawal; Mohammed Badawi; Mahdi Mirzazadeh; Carla Bromberg; Fan Ye; Chang Liu; Tatiana Sholokhova; George-Cristian Muraru; Gargi Balasubramaniam; Jonathan Malmaud; Alen Carin; Danilo Martins; Irina Jurenka; Pankil Botadra; Dave Lacey; Richa Singh; Mariano Schain; Dan Zheng; Isabelle Guyon; Victor Lavrenko; Seungji Lee; Xiang Zhou; Demis Hassabis; Jeshwanth Challagundla; Derek Cheng; Nikhil Mehta; Matthew Mauger; Michela Paganini; Pushkar Mishra; Kate Lee; Zhang Li; Lexi Baugher; Ondrej Skopek; Max Chang; Amir Zait; Gaurav Menghani; Lizzetth Bellot; Guangxing Han; Jean-Michel Sarr; Sharat Chikkerur; Himanshu Sahni; Rohan Anil; Arun Narayanan; Chandu Thekkath; Daniele Pighin; Hana Strejček; Marko Velic; Fred Bertsch; Manuel Tragut; Keran Rong; Alicia Parrish; Kai Bailey; Jiho Park; Isabela Albuquerque; Abhishek Bapna; Rajesh Venkataraman; Alec Kosik; Johannes Griesser; Zhiwei Deng; Alek Andreev; Qingyun Dou; Kevin Hui; Fanny Wei; Xiaobin Yu; Lei Shu; Avia Aharon; David Barker; Badih Ghazi; Sebastian Flennerhag; Chris Breaux; Yuchuan Liu; Matthew Bilotti; Josh Woodward; Uri Alon; Stephanie Winkler; Tzu-Kuo Huang; Kostas Andriopoulos; João Gabriel Oliveira; Penporn Koanantakool; Berkin Akin; Michael Wunder; Cicero Nogueira dos Santos; Mohammad Hossein Bateni; Lin Yang; Dan Horgan; Beer Changpinyo; Keyvan Amiri; Min Ma; Dayeong Lee; Lihao Liang; Anirudh Baddepudi; Tejasi Latkar; Raia Hadsell; Jun Xu; Hairong Mu; Michael Han; Aedan Pope; Snchit Grover; Frank Kim; Ankit Bhagatwala; Guan Sun; Yamini Bansal; Amir Globerson; Alireza Nazari; Samira Daruki; Hagen Soltau; Jane Labanowski; Laurent El Shafey; Matt Harvey; Yanif Ahmad; Elan Rosenfeld; William Kong; Etienne Pot; Yi-Xuan Tan; Aurora Wei; Victoria Langston; Marcel Prasetya; Petar Veličković; Richard Killam; Robin Strudel; Darren Ni; Zhenhai Zhu; Aaron Archer; Kavya Kopparapu; Lynn Nguyen; Emilio Parisotto; Hussain Masoom; Sravanti Addepalli; Jordan Grimstad; Hexiang Hu; Joss Moore; Avinatan Hassidim; Le Hou; Mukund Raghavachari; Jared Lichtarge; Adam R. Brown; Hilal Dib; Natalia Ponomareva; Justin Fu; Yujing Zhang; Altaf Rahman; Joana Iljazi; Edouard Leurent; Gabriel Dulac-Arnold; Cosmo Du; Chulayuth Asawaroengchai; Larry Jin; Ela Gruzewska; Ziwei Ji; Benigno Uria; Freitas Daniel De; Paul Barham; Lauren Beltrone; Víctor Campos; Jun Yan; Neel Kovelamudi; Arthur Nguyen; Elinor Davies; Zhichun Wu; Zoltan Egyed; Kristina Toutanova; Nithya Attaluri; Hongliang Fei; Peter Stys; Siddhartha Brahma; Martin Izzard; Siva Velusamy; Scott Lundberg; Vincent Zhuang; Kevin Sequeira; Adam Santoro; Ehsan Amid; Ophir Aharoni; Shuai Ye; Mukund Sundararajan; Lijun Yu; Yu-Cheng Ling; Stephen Spencer; Hugo Song; Josip Djolonga; Christo Kirov; Sonal Gupta; Alessandro Bissacco; Clemens Meyer; Mukul Bhutani; Andrew Dai; Weiyi Wang; Siqi Liu; Ashwin Sreevatsa; Qijun Tan; Maria Wang; Lucy Kim; Yicheng Wang; Alex Irpan; Yang Xiao; Stanislav Fort; Yifan He; Alex Gurney; Bryan Gale; Yue Ma; Monica Roy; Viorica Patraucean; Taylan Bilal; Golnaz Ghiasi; Anahita Hosseini; Melvin Johnson; Zhuowan Li; Yi Tay; Benjamin Beyret; Katie Millican; Josef Broder; Mayank Lunayach; Danny Swisher; Eugen Vušak; David Parkinson; MH Tessler; Adi Mayrav Gilady; Richard Song; Allan Dafoe; Yves Raimond; Masa Yamaguchi; Itay Karo; Elizabeth Nielsen; Kevin Kilgour; Mike Dusenberry; Rajiv Mathews; Jiho Choi; Siyuan Qiao; Harsh Mehta; Sahitya Potluri; Chris Knutsen; Jialu Liu; Tat Tan; Kuntal Sengupta; Keerthana Gopalakrishnan; Abodunrinwa Toki; Mencher Chiang; Mike Burrows; Grace Vesom; Zafarali Ahmed; Ilia Labzovsky; Siddharth Vashishtha; Preeti Singh; Ankur Sharma; Ada Ma; Jinyu Xie; Pranav Talluri; Hannah Forbes-Pollard; Aarush Selvan; Joel Wee; Loic Matthey; Tom Funkhouser; Parthasarathy Gopavarapu; Lev Proleev; Cheng Li; Matt Thomas; Kashyap Kolipaka; Zhipeng Jia; Ashwin Kakarla; Srinivas Sunkara; Joan Puigcerver; Suraj Satishkumar Sheth; Emily Graves; Chen Wang; Sadh MNM Khan; Kai Kang; Shyamal Buch; Fred Zhang; Omkar Savant; David Soergel; Kevin Lee; Linda Friso; Xuanyi Dong; Rahul Arya; Shreyas Chandrakaladharan; Connor Schenck; Greg Billock; Tejas Iyer; Anton Bakalov; Leslie Baker; Alex Ruiz; Angad Chandorkar; Trieu Trinh; Matt Miecnikowski; Yanqi Zhou; Yangsibo Huang; Jiazhong Nie; Ali Shah; Ashish Thapliyal; Sam Haves; Lun Wang; Uri Shaham; Patrick Morris-Suzuki; Soroush Radpour; Leonard Berrada; Thomas Strohmann; Chaochao Yan; Jingwei Shen; Sonam Goenka; Tris Warkentin; Petar Dević; Dan Belov; Albert Webson; Madhavi Yenugula; Puranjay Datta; Jerry Chang; Nimesh Ghelani; Aviral Kumar; Vincent Perot; Jessica Lo; Yang Song; Herman Schmit; Jianmin Chen; Vasilisa Bashlovkina; Xiaoyue Pan; Diana Mincu; Paul Roit; Isabel Edkins; Andy Davis; Yujia Li; Ben Horn; Xinjian Li; Pradeep Kumar S; Eric Doi; Wanzheng Zhu; Sri Gayatri Sundara Padmanabhan; Siddharth Verma; Jasmine Liu; Heng Chen; Mihajlo Velimirović; Malcolm Reynolds; Priyanka Agrawal; Nick Sukhanov; Abhinit Modi; Siddharth Goyal; John Palowitch; Nima Khajehnouri; Wing Lowe; David Klinghoffer; Sharon Silver; Vinh Tran; Candice Schumann; Francesco Piccinno; Xi Liu; Mario Lučić; Xiaochen Yang; Sandeep Kumar; Ajay Kannan; Ragha Kotikalapudi; Mudit Bansal; Fabian Fuchs; Mohammad Javad Hosseini; Abdelrahman Abdelhamed; Dawn Bloxwich; Tianhe Yu; Ruoxin Sang; Gregory Thornton; Karan Gill; Yuchi Liu; Virat Shejwalkar; Jason Lin; Zhipeng Yan; Kehang Han; Thomas Buschmann; Michael Pliskin; Zhi Xing; Susheel Tatineni; Junlin Zhang; Sissie Hsiao; Gavin Buttimore; Marcus Wu; Zefei Li; Geza Kovacs; Legg Yeung; Tao Huang; Aaron Cohen; Bethanie Brownfield; Averi Nowak; Mikel Rodriguez; Tianze Shi; Hasselt Hado van; Kevin Cen; Deepanway Ghoshal; Kushal Majmundar; Weiren Yu; Warren Weilun Chen; Danila Sinopalnikov; Hao Zhang; Vlado Galić; Di Lu; Zeyu Zheng; Maggie Song; Gary Wang; Gui Citovsky; Swapnil Gawde; Isaac Galatzer-Levy; David Silver; Ivana Balazevic; Dipanjan Das; Kingshuk Majumder; Yale Cong; Praneet Dutta; Dustin Tran; Hui Wan; Junwei Yuan; Daniel Eppens; Alanna Walton; Been Kim; Harry Ragan; James Cobon-Kerr; Lu Liu; Weijun Wang; Bryce Petrini; Jack Rae; Rakesh Shivanna; Yan Xiong; Chace Lee; Pauline Coquinot; Yiming Gu; Lisa Patel; Blake Hechtman; Aviel Boag; Orion Jankowski; Alex Wertheim; Alex Lee; Paul Covington; Hila Noga; Sam Sobell; Shanthal Vasanth; William Bono; Chirag Nagpal; Wei Fan; Xavier Garcia; Kedar Soparkar; Aybuke Turker; Nathan Howard; Sachit Menon; Yuankai Chen; Vikas Verma; Vladimir Pchelin; Harish Rajamani; Valentin Dalibard; Ana Ramalho; Yang Guo; Kartikeya Badola; Seojin Bang; Nathalie Rauschmayr; Julia Proskurnia; Sudeep Dasari; Xinyun Chen; Mikhail Sushkov; Anja Hauth; Pauline Sho; Abhinav Singh; Bilva Chandra; Allie Culp; Max Dylla; Olivier Bachem; James Besley; Heri Zhao; Timothy Lillicrap; Wei Wei; Wael Al Jishi; Ning Niu; Alban Rrustemi; Raphaël Lopez Kaufman; Ryan Poplin; Jewel Zhao; Minh Truong; Shikhar Bharadwaj; Ester Hlavnova; Eli Stickgold; Cordelia Schmid; Georgi Stephanov; Zhaoqi Leng; Frederick Liu; Léonard Hussenot; Shenil Dodhia; Juliana Vicente Franco; Lesley Katzen; Abhanshu Sharma; Sarah Cogan; Zuguang Yang; Aniket Ray; Sergi Caelles; Shen Yan; Ravin Kumar; Daniel Gillick; Renee Wong; Joshua Ainslie; Jonathan Hoech; Séb Arnold; Dan Abolafia; Anca Dragan; Ben Hora; Grace Hu; Alexey Guseynov; Yang Lu; Chas Leichner; Jinmeng Rao; Abhimanyu Goyal; Nagabhushan Baddi; Daniel Hernandez Diaz; Tim McConnell; Max Bain; Jake Abernethy; Qiqi Yan; Rylan Schaeffer; Paul Vicol; Will Thompson; Montse Gonzalez Arenas; Mathias Bellaiche; Pablo Barrio; Stefan Zinke; Riccardo Patana; Pulkit Mehta; JK Kearns; Avraham Ruderman; Scott Pollom; David D'Ambrosio; Cath Hope; Yang Yu; Andrea Gesmundo; Kuang-Huei Lee; Aviv Rosenberg; Yiqian Zhou; Yaoyiran Li; Drew Garmon; Yonghui Wu; Safeen Huda; Gil Fidel; Martin Baeuml; Jian Li; Phoebe Kirk; Rhys May; Tao Tu; Sara Mc Carthy; Toshiyuki Fukuzawa; Miranda Aperghis; Chih-Kuan Yeh; Toshihiro Yoshino; Bo Li; Austin Myers; Kaisheng Yao; Ben Limonchik; Changwan Ryu; Rohun Saxena; Alex Goldin; Ruizhe Zhao; Rocky Rhodes; Tao Zhu; Divya Tyam; Heidi Howard; Nathan Byrd; Hongxu Ma; Yan Wu; Ryan Mullins; Qingze Wang; Aida Amini; Sebastien Baur; Yiran Mao; Subhashini Venugopalan; Will Song; Wen Ding; Paul Collins; Sashank Reddi; Megan Shum; Andrei Rusu; Luisa Zintgraf; Kelvin Chan; Sheela Goenka; Mathieu Blondel; Michael Collins; Renke Pan; Marissa Giustina; Nikolai Chinaev; Christian Schuler; Ce Zheng; Jonas Valfridsson; Alyssa Loo; Alex Yakubovich; Jamie Smith; Tao Jiang; Rich Munoz; Gabriel Barcik; Rishabh Bansal; Mingyao Yang; Yilun Du; Pablo Duque; Mary Phuong; Alexandra Belias; Kunal Lad; Zeyu Liu; Tal Schuster; Karthik Duddu; Jieru Hu; Paige Kunkle; Matthew Watson; Jackson Tolins; Josh Smith; Denis Teplyashin; Garrett Bingham; Marvin Ritter; Marco Andreetto; Divya Pitta; Mohak Patel; Shashank Viswanadha; Trevor Strohman; Catalin Ionescu; Jincheng Luo; Yogesh Kalley; Jeremy Wiesner; Dan Deutsch; Derek Lockhart; Peter Choy; Rumen Dangovski; Chawin Sitawarin; Cat Graves; Tanya Lando; Amersfoort Joost van; Ndidi Elue; Zhouyuan Huo; Pooya Moradi; Jean Tarbouriech; Henryk Michalewski; Wenting Ye; Eunyoung Kim; Alex Druinsky; Florent Altché; Xinyi Chen; Artur Dwornik; Da-Cheng Juan; Rivka Moroshko; Horia Toma; Jarrod Kahn; Hai Qian; Maximilian Sieb; Irene Cai; Roman Goldenberg; Praneeth Netrapalli; Sindhu Raghuram; Yuan Gong; Lijie Fan; Evan Palmer; Yossi Matias; Valentin Gabeur; Shreya Pathak; Tom Ouyang; Don Metzler; Geoff Bacon; Srinivasan Venkatachary; Sridhar Thiagarajan; Alex Cullum; Eran Ofek; Vytenis Sakenas; Mohamed Hammad; Cesar Magalhaes; Mayank Daswani; Oscar Chang; Ashok Popat; Ruichao Li; Komal Jalan; Yanhan Hou; Josh Lipschultz; Antoine He; Wenhao Jia; Pier Giuseppe Sessa; Prateek Kolhar; William Wong; Sumeet Singh; Lukas Haas; Jay Whang; Hanna Klimczak-Plucińska; Georges Rotival; Grace Chung; Yiqing Hua; Anfal Siddiqui; Nicolas Serrano; Dongkai Chen; Billy Porter; Libin Bai; Keshav Shivam; Sho Arora; Partha Talukdar; Tom Cobley; Sangnie Bhardwaj; Evgeny Gladchenko; Simon Green; Kelvin Guu; Felix Fischer; Xiao Wu; Eric Wang; Achintya Singhal; Tatiana Matejovicova; James Martens; Hongji Li; Roma Patel; Elizabeth Kemp; Jiaqi Pan; Lily Wang; Blake JianHang Chen; Jean-Baptiste Alayrac; Navneet Potti; Erika Gemzer; Eugene Ie; Kay McKinney; Takaaki Saeki; Edward Chou; Pascal Lamblin; SQ Mah; Zach Fisher; Martin Chadwick; Jon Stritar; Obaid Sarvana; Andrew Hogue; Artem Shtefan; Hadi Hashemi; Yang Xu; Jindong Gu; Sharad Vikram; Chung-Ching Chang; Sabela Ramos; Logan Kilpatrick; Weijuan Xi; Jenny Brennan; Yinghao Sun; Abhishek Jindal; Ionel Gog; Dawn Chen; Felix Wu; Jason Lee; Sudhindra Kopalle; Srinadh Bhojanapalli; Oriol Vinyals; Natan Potikha; Burcu Karagol Ayan; Yuan Yuan; Michael Riley; Piotr Stanczyk; Sergey Kishchenko; Bing Wang; Dan Garrette; Antoine Yang; Vlad Feinberg; CJ Carey; Javad Azizi; Viral Shah; Erica Moreira; Chongyang Shi; Josh Feldman; Elizabeth Salesky; Thomas Lampe; Aneesh Pappu; Duhyeon Kim; Jonas Adler; Avi Caciularu; Brian Walker; Yunhan Xu; Yochai Blau; Dylan Scandinaro; Terry Huang; Sam El-Husseini; Abhishek Sinha; Lijie Ren; Taylor Tobin; Patrik Sundberg; Tim Sohn; Vikas Yadav; Mimi Ly; Emily Xue; Jing Xiong; Afzal Shama Soudagar; Sneha Mondal; Nikhil Khadke; Qingchun Ren; Ben Vargas; Stan Bileschi; Sarah Chakera; Cindy Wang; Boyu Wang; Yoni Halpern; Joe Jiang; Vikas Sindhwani; Petre Petrov; Pranavaraj Ponnuramu; Sanket Vaibhav Mehta; Yu Watanabe; Betty Chan; Matheus Wisniewski; Trang Pham; Jingwei Zhang; Conglong Li; Cesare Dario de; Art Khurshudov; Alex Vasiloff; Melissa Tan; Zoe Ashwood; Bobak Shahriari; Maryam Majzoubi; Garrett Tanzer; Olga Kozlova; Robin Alazard; James Lee-Thorp; Nguyet Minh Phu; Isaac Tian; Junwhan Ahn; Andy Crawford; Lauren Lax; Yuan Shangguan; Iftekhar Naim; David Ross; Oleksandr Ferludin; Tongfei Guo; Andrea Banino; Hubert Soyer; Xiaoen Ju; Dominika Rogozińska; Ishaan Malhi; Marcella Valentine; Daniel Balle; Apoorv Kulshreshtha; Maciej Kula; Yiwen Song; Sophia Austin; John Schultz; Roy Hirsch; Arthur Douillard; Apoorv Reddy; Michael Fink; Summer Yue; Khyatti Gupta; Adam Zhang; Norman Rink; Daniel McDuff; Lei Meng; András György; Yasaman Razeghi; Ricky Liang; Kazuki Osawa; Aviel Atias; Matan Eyal; Tyrone Hill; Nikolai Grigorev; Zhengdong Wang; Nitish Kulkarni; Rachel Soh; Ivan Lobov; Zachary Charles; Sid Lall; Kazuma Hashimoto; Ido Kessler; Victor Gomes; Zelda Mariet; Danny Driess; Alessandro Agostini; Canfer Akbulut; Jingcao Hu; Marissa Ikonomidis; Emily Caveness; Kartik Audhkhasi; Saurabh Agrawal; Ioana Bica; Evan Senter; Jayaram Mudigonda; Kelly Chen; Jingchen Ye; Xuanhui Wang; James Svensson; Philipp Fränken; Josh Newlan; Li Lao; Eva Schnider; Sami Alabed; Joseph Kready; Jesse Emond; Afief Halumi; Tim Zaman; Chengxi Ye; Naina Raisinghani; Vilobh Meshram; Bo Chang; Ankit Singh Rawat; Axel Stjerngren; Sergey Levi; Rui Wang; Xiangzhu Long; Mitchelle Rasquinha; Steven Hand; Aditi Mavalankar; Lauren Agubuzu; Sudeshna Roy; Junquan Chen; Jarek Wilkiewicz; Hao Zhou; Michal Jastrzebski; Qiong Hu; Agustin Dal Lago; Ramya Sree Boppana; Wei-Jen Ko; Jennifer Prendki; Yao Su; Zhi Li; Eliza Rutherford; Girish Ramchandra Rao; Ramona Comanescu; Adrià Puigdomènech; Qihang Chen; Dessie Petrova; Christine Chan; Vedrana Milutinovic; Felipe Tiengo Ferreira; Chin-Yi Cheng; Ming Zhang; Tapomay Dey; Sherry Yang; Ramesh Sampath; Quoc Le; Howard Zhou; Chu-Cheng Lin; Hoi Lam; Christine Kaeser-Chen; Kai Hui; Dean Hirsch; Tom Eccles; Basil Mustafa; Shruti Rijhwani; Morgane Rivière; Yuanzhong Xu; Junjie Wang; Xinyang Geng; Xiance Si; Arjun Khare; Cheolmin Kim; Vahab Mirrokni; Kamyu Lee; Khuslen Baatarsukh; Nathaniel Braun; Lisa Wang; Pallavi LV; Richard Tanburn; Yonghao Zhu; Fangda Li; Setareh Ariafar; Dan Goldberg; Ken Burke; Daniil Mirylenka; Meiqi Guo; Olaf Ronneberger; Hadas Natalie Vogel; Liqun Cheng; Nishita Shetty; Johnson Jia; Thomas Jimma; Corey Fry; Ted Xiao; Martin Sundermeyer; Ryan Burnell; Yannis Assael; Mario Pinto; JD Chen; Rohit Sathyanarayana; Donghyun Cho; Jing Lu; Rishabh Agarwal; Sugato Basu; Lucas Gonzalez; Dhruv Shah; Meng Wei; Dre Mahaarachchi; Rohan Agrawal; Tero Rissa; Yani Donchev; Ramiro Leal-Cavazos; Adrian Hutter; Markus Mircea; Alon Jacovi; Faruk Ahmed; Jiageng Zhang; Shuguang Hu; Bo-Juen Chen; Jonni Kanerva; Guillaume Desjardins; Andrew Lee; Nikos Parotsidis; Asier Mujika; Tobias Weyand; Jasper Snoek; Jo Chick; Kai Chen; Paul Chang; Ethan Mahintorabi; Zi Wang; Tolly Powell; Orgad Keller; Abhirut Gupta; Claire Sha; Kanav Garg; Nicolas Heess; Ágoston Weisz; Cassidy Hardin; Bartek Wydrowski; Ben Coleman; Karina Zainullina; Pankaj Joshi; Alessandro Epasto; Terry Spitz; Binbin Xiong; Kai Zhao; Arseniy Klimovskiy; Ivy Zheng; Johan Ferret; Itay Yona; Waleed Khawaja; Jean-Baptiste Lespiau; Maxim Krikun; Siamak Shakeri; Timothee Cour; Bonnie Li; Igor Krivokon; Dan Suh; Alex Hofer; Jad Al Abdallah; Nikita Putikhin; Oscar Akerlund; Silvio Lattanzi; Anurag Kumar; Shane Settle; Himanshu Srivastava; Folawiyo Campbell-Ajala; Edouard Rosseel; Mihai Dorin Istin; Nishanth Dikkala; Anand Rao; Nick Young; Kate Lin; Dhruva Bhaswar; Yiming Wang; Jaume Sanchez Elias; Kritika Muralidharan; James Keeling; Dayou Du; Siddharth Gopal; Gregory Dibb; Charles Blundell; Manolis Delakis; Jacky Liang; Marco Tulio Ribeiro; Georgi Karadzhov; Guillermo Garrido; Ankur Bapna; Jiawei Cao; Adam Sadovsky; Pouya Tafti; Arthur Guez; Coline Devin; Yixian Di; Jinwei Xing; Chuqiao Joyce Xu; Hanzhao Lin; Chun-Te Chu; Sameera Ponda; Wesley Helmholz; Fan Yang; Yue Gao; Sara Javanmardi; Wael Farhan; Alex Ramirez; Ricardo Figueira; Khe Chai Sim; Yuval Bahat; Ashwin Vaswani; Liangzhe Yuan; Gufeng Zhang; Leland Rechis; Hanjun Dai; Tayo Oguntebi; Alexandra Cordell; Eugénie Rives; Kaan Tekelioglu; Naveen Kumar; Bing Zhang; Aurick Zhou; Nikolay Savinov; Andrew Leach; Alex Tudor; Sanjay Ganapathy; Yanyan Zheng; Mirko Rossini; Vera Axelrod; Arnaud Autef; Yukun Zhu; Zheng Zheng; Mingda Zhang; Baochen Sun; Jie Ren; Nenad Tomasev; Nithish Kannen; Amer Sinha; Charles Chen; Louis O'Bryan; Alex Pak; Aditya Kusupati; Weel Yang; Deepak Ramachandran; Patrick Griffin; Seokhwan Kim; Philipp Neubeck; Craig Schiff; Tammo Spalink; Mingyang Ling; Arun Nair; Ga-Young Joung; Linda Deng; Avishkar Bhoopchand; Lora Aroyo; Tom Duerig; Jordan Griffith; Gabe Barth-Maron; Jake Ades; Alex Haig; Ankur Taly; Yunting Song; Paul Michel; Dave Orr; Dean Weesner; Corentin Tallec; Carrie Grimes Bostock; Paul Niemczyk; Andy Twigg; Mudit Verma; Rohith Vallu; Henry Wang; Marco Gelmi; Kiranbir Sodhia; Aleksandr Chuklin; Omer Goldman; Jasmine George; Liang Bai; Kelvin Zhang; Petar Sirkovic; Efrat Nehoran; Golan Pundak; Jiaqi Mu; Alice Chen; Alex Greve; Paulo Zacchello; David Amos; Heming Ge; Eric Noland; Colton Bishop; Jeffrey Dudek; Youhei Namiki; Elena Buchatskaya; Jing Li; Dorsa Sadigh; Masha Samsikova; Dan Malkin; Damien Vincent; Robert David; Rob Willoughby; Phoenix Meadowlark; Shawn Gao; Yan Li; Raj Apte; Amit Jhindal; Stein Xudong Lin; Alex Polozov; Zhicheng Wang; Tomas Mery; Anirudh GP; Varun Yerram; Sage Stevens; Tianqi Liu; Noah Fiedel; Charles Sutton; Matthew Johnson; Xiaodan Song; Kate Baumli; Nir Shabat; Muqthar Mohammad; Hao Liu; Marco Selvi; Yichao Zhou; Mehdi Hafezi Manshadi; Chu-ling Ko; Anthony Chen; Michael Bendersky; Jorge Gonzalez Mendez; Nisarg Kothari; Amir Zandieh; Yiling Huang; Daniel Andor; Ellie Pavlick; Idan Brusilovsky; Jitendra Harlalka; Sally Goldman; Andrew Lampinen; Guowang Li; Asahi Ushio; Somit Gupta; Lei Zhang; Chuyuan Kelly Fu; Madhavi Sewak; Timo Denk; Jed Borovik; Brendan Jou; Avital Zipori; Prateek Jain; Junwen Bai; Thang Luong; Jonathan Tompson; Alice Li; Li Liu; George Powell; Jiajun Shen; Alex Feng; Grishma Chole; Da Yu; Yinlam Chow; Tongxin Yin; Eric Malmi; Kefan Xiao; Yash Pande; Shachi Paul; Niccolò Dal Santo; Adil Dostmohamed; Sergio Guadarrama; Aaron Phillips; Thanumalayan Sankaranarayana Pillai; Gal Yona; Amin Ghafouri; Preethi Lahoti; Benjamin Lee; Dhruv Madeka; Eren Sezener; Simon Tokumine; Adrian Collister; Cao Nicola De; Richard Shin; Uday Kalra; Parker Beak; Emily Nottage; Ryo Nakashima; Ivan Jurin; Vikash Sehwag; Meenu Gaba; Junhao Zeng; Kevin R. McKee; Fernando Pereira; Tamar Yakar; Amayika Panda; Arka Dhar; Peilin Zhong; Daniel Sohn; Mark Brand; Lars Lowe Sjoesund; Viral Carpenter; Sharon Lin; Shantanu Thakoor; Marcus Wainwright; Ashwin Chaugule; Pranesh Srinivasan; Muye Zhu; Bernett Orlando; Jack Weber; Ayzaan Wahid; Gilles Baechler; Apurv Suman; Jovana Mitrović; Gabe Taubman; Honglin Yu; Helen King; Josh Dillon; Cathy Yip; Dhriti Varma; Tomas Izo; Levent Bolelli; Borja De Balle Pigem; Trapani Julia Di; Fotis Iliopoulos; Adam Paszke; Nishant Ranka; Joe Zou; Francesco Pongetti; Jed McGiffin; Alex Siegman; Rich Galt; Ross Hemsley; Goran Žužić; Victor Carbune; Tao Li; Myle Ott; Félix de Chaumont Quitry; David Vilar Torres; Yuri Chervonyi; Tomy Tsai; Prem Eruvbetine; Samuel Yang; Matthew Denton; Jake Walker; Slavica Andačić; Idan Heimlich Shtacher; Vittal Premachandran; Harshal Tushar Lehri; Cip Baetu; Damion Yates; Lampros Lamprou; Mariko Iinuma; Ioana Mihailescu; Ben Albrecht; Shachi Dave; Susie Sargsyan; Bryan Perozzi; Lucas Manning; Chiyuan Zhang; Denis Vnukov; Igor Mordatch; Raia Hadsell Wolfgang Macherey; Ryan Kappedal; Jim Stephan; Aditya Tripathi; Klaus Macherey; Jun Qian; Abhishek Bhowmick; Shekoofeh Azizi; Rémi Leblond; Shiva Mohan Reddy Garlapati; Timothy Knight; Matthew Wiethoff; Wei-Chih Hung; Anelia Angelova; Georgios Evangelopoulos; Pawel Janus; Dimitris Paparas; Matthew Rahtz; Ken Caluwaerts; Vivek Sampathkumar; Daniel Jarrett; Shadi Noghabi; Antoine Miech; Chak Yeung; Geoff Clark; Henry Prior; Fei Zheng; Jean Pouget-Abadie; Indro Bhattacharya; Kalpesh Krishna; Will Bishop; Zhe Yuan; Yunxiao Deng; Ashutosh Sathe; Kacper Krasowiak; Ciprian Chelba; Cho-Jui Hsieh; Kiran Vodrahalli; Buhuang Liu; Thomas Köppe; Amr Khalifa; Lubo Litchev; Pichi Charoenpanit; Reed Roberts; Sachin Yadav; Yasumasa Onoe; Desi Ivanov; Megha Mohabey; Vighnesh Birodkar; Nemanja Rakićević; Pierre Sermanet; Vaibhav Mehta; Krishan Subudhi; Travis Choma; Will Ng; Luheng He; Kathie Wang; Tasos Kementsietsidis; Shane Gu; Mansi Gupta; Andrew Nystrom; Mehran Kazemi; Timothy Chung; Nacho Cano; Nikhil Dhawan; Yufei Wang; Jiawei Xia; Trevor Yacovone; Eric Jia; Mingqing Chen; Simeon Ivanov; Ashrith Sheshan; Sid Dalmia; Paweł Stradomski; Pengcheng Yin; Salem Haykal; Congchao Wang; Dennis Duan; Neslihan Bulut; Greg Kochanski; Liam MacDermed; Namrata Godbole; Shitao Weng; Jingjing Chen; Rachana Fellinger; Ramin Mehran; Daniel Suo; Hisham Husain; Tong He; Kaushal Patel; Joshua Howland; Randall Parker; Kelvin Nguyen; Sharath Maddineni; Chris Rawles; Mina Khan; Shlomi Cohen-Ganor; Amol Mandhane; Xinyi Wu; Chenkai Kuang; Iulia Comşa; Ramya Ganeshan; Hanie Sedghi; Adam Bloniarz; Nuo Wang Pierse; Anton Briukhov; Petr Mitrichev; Anita Gergely; Serena Zhan; Allan Zhou; Nikita Saxena; Eva Lu; Josef Dean; Ashish Gupta; Nicolas Perez-Nieves; Renjie Wu; Cory McLean; Wei Liang; Disha Jindal; Anton Tsitsulin; Wenhao Yu; Kaiz Alarakyia; Tom Schaul; Piyush Patil; Peter Sung; Elijah Peake; Hongkun Yu; Feryal Behbahani; JD Co-Reyes; Alan Ansell; Sean Sun; Clara Barbu; Jonathan Lee; Seb Noury; James Allingham; Bilal Piot; Mohit Sharma; Christopher Yew; Ivan Korotkov; Bibo Xu; Demetra Brady; Goran Petrovic; Shibl Mourad; Claire Cui; Aditya Gupta; Parker Schuh; Saarthak Khanna; Anna Goldie; Abhinav Arora; Vadim Zubov; Amy Stuart; Mark Epstein; Yun Zhu; Jianqiao Liu; Yury Stuken; Ziyue Wang; Karolis Misiunas; Dee Guo; Ashleah Gill; Ale Hartman; Zaid Nabulsi; Aurko Roy; Aleksandra Faust; Jason Riesa; Ben Withbroe; Mengchao Wang; Marco Tagliasacchi; Andreea Marzoca; James Noraky; Serge Toropov; Malika Mehrotra; Bahram Raad; Sanja Deur; Steve Xu; Marianne Monteiro; Zhongru Wu; Yi Luan; Sam Ritter; Nick Li; Håvard Garnes; Yanzhang He; Martin Zlocha; Jifan Zhu; Matteo Hessel; Will Wu; Spandana Raj Babbula; Chizu Kawamoto; Yuanzhen Li; Mehadi Hassen; Yan Wang; Brian Wieder; James Freedman; Yin Zhang; Xinyi Bai; Tianli Yu; David Reitter; XiangHai Sheng; Mateo Wirth; Aditya Kini; Dima Damen; Mingcen Gao; Rachel Hornung; Michael Voznesensky; Brian Roark; Adhi Kuncoro; Yuxiang Zhou; Rushin Shah; Anthony Brohan; Kuangyuan Chen; James Wendt; David Rim; Paul Kishan Rubenstein; Jonathan Halcrow; Michelle Liu; Ty Geri; Yunhsuan Sung; Jane Shapiro; Shaan Bijwadia; Chris Duvarney; Christina Sorokin; Paul Natsev; Reeve Ingle; Pramod Gupta; Young Maeng; Ndaba Ndebele; Kexin Zhu; Valentin Anklin; Katherine Lee; Yuan Liu; Yaroslav Akulov; Shaleen Gupta; Guolong Su; Flavien Prost; Tianlin Liu; Vitaly Kovalev; Pol Moreno; Martin Scholz; Sam Redmond; Zongwei Zhou; Alex Castro-Ros; André Susano Pinto; Dia Kharrat; Michal Yarom; Rachel Saputro; Jannis Bulian; Ben Caine; Ji Liu; Abbas Abdolmaleki; Shariq Iqbal; Tautvydas Misiunas; Mikhail Sirotenko; Shefali Garg; Guy Bensky; Huan Gui; Xuezhi Wang; Raphael Koster; Mike Bernico; Da Huang; Romal Thoppilan; Trevor Cohn; Ben Golan; Wenlei Zhou; Andrew Rosenberg; Markus Freitag; Tynan Gangwani; Vincent Tsang; Anand Shukla; Xiaoqi Ren; Minh Giang; Chi Zou; Andre Elisseeff; Charline Le Lan; Dheeru Dua; Shuba Lall; Pranav Shyam; Frankie Garcia; Sarah Nguyen; Michael Guzman; AJ Maschinot; Marcello Maggioni; Ming-Wei Chang; Karol Gregor; Lotte Weerts; Kumaran Venkatesan; Bogdan Damoc; Leon Liu; Jan Wassenberg; Lewis Ho; Becca Roelofs; Majid Hadian; François-Xavier Aubet; Yu Liang; Sami Lachgar; Danny Karmon; Yong Cheng; Amelio Vázquez-Reina; Angie Chen; Zhuyun Dai; Andy Brock; Shubham Agrawal; Chenxi Pang; Peter Garst; Mariella Sanchez-Vargas; Ivor Rendulic; Aditya Ayyar; Andrija Ražnatović; Olivia Ma; Roopali Vij; Neha Sharma; Ashwin Balakrishna; Bingyuan Liu; Ian Mackinnon; Sorin Baltateanu; Petra Poklukar; Gabriel Ibagon; Colin Ji; Hongyang Jiao; Isaac Noble; Wojciech Stokowiec; Zhihao Li; Jeff Dean; David Lindner; Mark Omernick; Kristen Chiafullo; Mason Dimarco; Vitor Rodrigues; Vittorio Selo; Garrett Honke; Xintian Cindy Wu; Wei He; Adam Hillier; Anhad Mohananey; Vihari Piratla; Chang Ye; Chase Malik; Sebastian Riedel; Samuel Albanie; Zi Yang; Kenny Vassigh; Maria Bauza; Sheng Li; Yiqing Tao; Nevan Wichers; Andrii Maksai; Abe Ittycheriah; Ross Mcilroy; Bryan Seybold; Noah Goodman; Romina Datta; Steven M. Hernandez; Tian Shi; Yony Kochinski; Anna Bulanova; Ken Franko; Mikita Sazanovich; Nicholas FitzGerald; Praneeth Kacham; Shubha Srinivas Raghvendra; Vincent Hellendoorn; Alexander Grushetsky; Julian Salazar; Angeliki Lazaridou; Jason Chang; Jan-Thorsten Peter; Sushant Kafle; Yann Dauphin; Abhishek Rao; Filippo Graziano; Izhak Shafran; Yuguo Liao; Tianli Ding; Geng Yan; Grace Chu; Zhao Fu; Vincent Roulet; Gabriel Rasskin; Duncan Williams; Shahar Drath; Alex Mossin; Raphael Hoffmann; Jordi Orbay; Francesco Bertolini; Hila Sheftel; Justin Chiu; Siyang Xue; Yuheng Kuang; Ferjad Naeem; Swaroop Nath; Nana Nti; Phil Culliton; Kashyap Krishnakumar; Michael Isard; Pei Sun; Ayan Chakrabarti; Nathan Clement; Regev Cohen; Arissa Wongpanich; GS Oh; Ashwin Murthy; Hao Zheng; Jessica Hamrick; Oskar Bunyan; Suhas Ganesh; Nitish Gupta; Roy Frostig; John Wieting; Yury Malkov; Pierre Marcenac; Zhixin Lucas Lai; Xiaodan Tang; Mohammad Saleh; Fedir Zubach; Chinmay Kulkarni; Huanjie Zhou; Vicky Zayats; Nan Ding; Anshuman Tripathi; Arijit Pramanik; Patrik Zochbauer; Harish Ganapathy; Vedant Misra; Zach Behrman; Hugo Vallet; Mingyang Zhang; Mukund Sridhar; Ye Jin; Mohammad Babaeizadeh; Siim Põder; Megha Goel; Divya Jain; Tajwar Nasir; Shubham Mittal; Tim Dozat; Diego Ardila; Aliaksei Severyn; Fabio Pardo; Sammy Jerome; Siyang Qin; Louis Rouillard; Amir Yazdanbakhsh; Zizhao Zhang; Shivani Agrawal; Kaushik Shivakumar; Caden Lu; Praveen Kallakuri; Rachita Chhaparia; Kanishka Rao; Charles Kwong; Asya Fadeeva; Shitij Nigam; Yan Virin; Yuan Zhang; Balaji Venkatraman; Beliz Gunel; Marc Wilson; Huiyu Wang; Abhinav Gupta; Xiaowei Xu; Adrien Ali Taïga; Kareem Mohamed; Doug Fritz; Daniel Rodriguez; Zoubin Ghahramani; Harry Askham; Lior Belenki; James Zhao; Rahul Gupta; Krzysztof Jastrzębski; Takahiro Kosakai; Kaan Katircioglu; Jon Schneider; Rina Panigrahy; Konstantinos Bousmalis; Peter Grabowski; Prajit Ramachandran; Chaitra Hegde; Mihaela Rosca; Angelo Scorza Scarpati; Kyriakos Axiotis; Ying Xu; Zach Gleicher; Assaf Hurwitz Michaely; Mandar Sharma; Sanil Jain; Christoph Hirnschall; Tal Marian; Xuhui Jia; Kevin Mather; Kilol Gupta; Linhai Qiu; Nigamaa Nayakanti; Lucian Ionita; Steven Zheng; Lucia Loher; Kurt Shuster; Igor Petrovski; Roshan Sharma; Rahma Chaabouni; Angel Yeh; James An; Arushi Gupta; Steven Schwarcz; Seher Ellis; Sam Conway-Rahman; Javier Snaider; Alex Zhai; James Atwood; Daniel Golovin; Liqian Peng; Te I; Vivian Xia; Salvatore Scellato; Mahan Malihi; Arthur Bražinskas; Vlad-Doru Ion; Younghoon Jun; James Swirhun; Soroosh Mariooryad; Jiao Sun; Steve Chien; Rey Coaguila; Ariel Brand; Yi Gao; Tom Kwiatkowski; Roee Aharoni; Cheng-Chun Lee; Mislav Žanić; Yichi Zhang; Dan Ethier; Vitaly Nikolaev; Pranav Nair; Yoav Ben Shalom; Hen Fitoussi; Jai Gupta; Hongbin Liu; Dee Cattle; Tolga Bolukbasi; Ben Murdoch; Fantine Huot; Yin Li; Chris Hahn
In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our most capable model yet, achieving SoTA performance on frontier coding and reasoning benchmarks. In addition to its incredible coding and reasoning skills, Gemini 2.5 Pro is a thinking model that excels at multimodal understanding and it is now able to process up to 3 hours of video content. Its unique combination of long context, multimodal and reasoning capabilities can be combined to unlock new agentic workflows. Gemini 2.5 Flash provides excellent reasoning abilities at a fraction of the compute and latency requirements and Gemini 2.0 Flash and Flash-Lite provide high performance at low latency and cost. Taken together, the Gemini 2.X model generation spans the full Pareto frontier of model capability vs cost, allowing users to explore the boundaries of what is possible with complex agentic problem solving.

http://arxiv.org/abs/2507.08982
VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models. (93%)
Hanene F. Z. Brachemi Meftah; Wassim Hamidouche; Sid Ahmed Fezza; Olivier Déforges
Recent years have witnessed remarkable progress in developing Vision-Language Models (VLMs) capable of processing both textual and visual inputs. These models have demonstrated impressive performance, leading to their widespread adoption in various applications. However, this widespread raises serious concerns regarding user privacy, particularly when models inadvertently process or expose private visual information. In this work, we frame the preservation of privacy in VLMs as an adversarial attack problem. We propose a novel attack strategy that selectively conceals information within designated Region Of Interests (ROIs) in an image, effectively preventing VLMs from accessing sensitive content while preserving the semantic integrity of the remaining image. Unlike conventional adversarial attacks that often disrupt the entire image, our method maintains high coherence in unmasked areas. Experimental results across three state-of-the-art VLMs namely LLaVA, Instruct-BLIP, and BLIP2-T5 demonstrate up to 98% reduction in detecting targeted ROIs, while maintaining global image semantics intact, as confirmed by high similarity scores between clean and adversarial outputs. We believe that this work contributes to a more privacy conscious use of multimodal models and offers a practical tool for further research, with the source code publicly available at: https://github.com/hbrachemi/Vlm_defense-attack.

http://arxiv.org/abs/2507.08303
Learning Robust Motion Skills via Critical Adversarial Attacks for Humanoid Robots. (87%)
Yang Zhang; Zhanxiang Cao; Buqing Nie; Haoyang Li; Yue Gao
Humanoid robots show significant potential in daily tasks. However, reinforcement learning-based motion policies often suffer from robustness degradation due to the sim-to-real dynamics gap, thereby affecting the agility of real robots. In this work, we propose a novel robust adversarial training paradigm designed to enhance the robustness of humanoid motion policies in real worlds. The paradigm introduces a learnable adversarial attack network that precisely identifies vulnerabilities in motion policies and applies targeted perturbations, forcing the motion policy to enhance its robustness against perturbations through dynamic adversarial training. We conduct experiments on the Unitree G1 humanoid robot for both perceptive locomotion and whole-body control tasks. The results demonstrate that our proposed method significantly enhances the robot's motion robustness in real world environments, enabling successful traversal of challenging terrains and highly agile whole-body trajectory tracking.

http://arxiv.org/abs/2507.08284
Lightweight Safety Guardrails via Synthetic Data and RL-guided Adversarial Training. (13%)
Aleksei Ilin; Gor Matevosyan; Xueying Ma; Vladimir Eremin; Suhaa Dada; Muqun Li; Riyaaz Shaik; Haluk Noyan Tokgozoglu
We introduce a lightweight yet highly effective safety guardrail framework for language models, demonstrating that small-scale language models can achieve, and even surpass, the performance of larger counterparts in content moderation tasks. This is accomplished through high-fidelity synthetic data generation and adversarial training. The synthetic data generation process begins with human-curated seed data, which undergoes query augmentation and paraphrasing to create diverse and contextually rich examples. This augmented data is then subjected to multiple rounds of curation, ensuring high fidelity and relevance. Inspired by recent advances in the Generative Adversarial Network (GAN) architecture, our adversarial training employs reinforcement learning to guide a generator that produces challenging synthetic examples. These examples are used to fine-tune the safety classifier, enhancing its ability to detect and mitigate harmful content. Additionally, we incorporate strategies from recent research on efficient LLM training, leveraging the capabilities of smaller models to improve the performance of larger generative models. With iterative adversarial training and the generation of diverse, high-quality synthetic data, our framework enables small language models (SLMs) to serve as robust safety guardrails. This approach not only reduces computational overhead but also enhances resilience against adversarial attacks, offering a scalable and efficient solution for content moderation in AI systems.

http://arxiv.org/abs/2507.10578
When and Where do Data Poisons Attack Textual Inversion? (13%)
Jeremy Styborski; Mingzhi Lyu; Jiayou Lu; Nupur Kapur; Adams Kong
Poisoning attacks pose significant challenges to the robustness of diffusion models (DMs). In this paper, we systematically analyze when and where poisoning attacks textual inversion (TI), a widely used personalization technique for DMs. We first introduce Semantic Sensitivity Maps, a novel method for visualizing the influence of poisoning on text embeddings. Second, we identify and experimentally verify that DMs exhibit non-uniform learning behavior across timesteps, focusing on lower-noise samples. Poisoning attacks inherit this bias and inject adversarial signals predominantly at lower timesteps. Lastly, we observe that adversarial signals distract learning away from relevant concept regions within training data, corrupting the TI process. Based on these insights, we propose Safe-Zone Training (SZT), a novel defense mechanism comprised of 3 key components: (1) JPEG compression to weaken high-frequency poison signals, (2) restriction to high timesteps during TI training to avoid adversarial signals at lower timesteps, and (3) loss masking to constrain learning to relevant regions. Extensive experiments across multiple poisoning methods demonstrate that SZT greatly enhances the robustness of TI against all poisoning attacks, improving generative quality beyond prior published defenses. Code: www.github.com/JStyborski/Diff_Lab Data: www.github.com/JStyborski/NC10

http://arxiv.org/abs/2507.08623
Entangled Threats: A Unified Kill Chain Model for Quantum Machine Learning Security. (10%)
Pascal Debus; Maximilian Wendlinger; Kilian Tscharke; Daniel Herr; Cedric Brügmann; Mello Daniel Ohl de; Juris Ulmanis; Alexander Erhard; Arthur Schmidt; Fabian Petsch
Quantum Machine Learning (QML) systems inherit vulnerabilities from classical machine learning while introducing new attack surfaces rooted in the physical and algorithmic layers of quantum computing. Despite a growing body of research on individual attack vectors - ranging from adversarial poisoning and evasion to circuit-level backdoors, side-channel leakage, and model extraction - these threats are often analyzed in isolation, with unrealistic assumptions about attacker capabilities and system environments. This fragmentation hampers the development of effective, holistic defense strategies. In this work, we argue that QML security requires more structured modeling of the attack surface, capturing not only individual techniques but also their relationships, prerequisites, and potential impact across the QML pipeline. We propose adapting kill chain models, widely used in classical IT and cybersecurity, to the quantum machine learning context. Such models allow for structured reasoning about attacker objectives, capabilities, and possible multi-stage attack paths - spanning reconnaissance, initial access, manipulation, persistence, and exfiltration. Based on extensive literature analysis, we present a detailed taxonomy of QML attack vectors mapped to corresponding stages in a quantum-aware kill chain framework that is inspired by the MITRE ATLAS for classical machine learning. We highlight interdependencies between physical-level threats (like side-channel leakage and crosstalk faults), data and algorithm manipulation (such as poisoning or circuit backdoors), and privacy attacks (including model extraction and training data inference). This work provides a foundation for more realistic threat modeling and proactive security-in-depth design in the emerging field of quantum machine learning.

http://arxiv.org/abs/2507.10583
$\texttt{Droid}$: A Resource Suite for AI-Generated Code Detection. (9%)
Daniil Orel; Indraneil Paul; Iryna Gurevych; Preslav Nakov
In this work, we compile $\textbf{$\texttt{DroidCollection}$}$, the most extensive open data suite for training and evaluating machine-generated code detectors, comprising over a million code samples, seven programming languages, outputs from 43 coding models, and over three real-world coding domains. Alongside fully AI-generated samples, our collection includes human-AI co-authored code, as well as adversarial samples explicitly crafted to evade detection. Subsequently, we develop $\textbf{$\texttt{DroidDetect}$}$, a suite of encoder-only detectors trained using a multi-task objective over $\texttt{DroidCollection}$. Our experiments show that existing detectors' performance fails to generalise to diverse coding domains and programming languages outside of their narrow training data. Additionally, we demonstrate that while most detectors are easily compromised by humanising the output distributions using superficial prompting and alignment approaches, this problem can be easily amended by training on a small amount of adversarial data. Finally, we demonstrate the effectiveness of metric learning and uncertainty-based resampling as means to enhance detector training on possibly noisy distributions.

http://arxiv.org/abs/2507.08261
Admissibility of Stein Shrinkage for Batch Normalization in the Presence of Adversarial Attacks. (2%)
Sofia Ivolgina; P. Thomas Fletcher; Baba C. Vemuri
Batch normalization (BN) is a ubiquitous operation in deep neural networks used primarily to achieve stability and regularization during network training. BN involves feature map centering and scaling using sample means and variances, respectively. Since these statistics are being estimated across the feature maps within a batch, this problem is ideally suited for the application of Stein's shrinkage estimation, which leads to a better, in the mean-squared-error sense, estimate of the mean and variance of the batch. In this paper, we prove that the Stein shrinkage estimator for the mean and variance dominates over the sample mean and variance estimators in the presence of adversarial attacks when modeling these attacks using sub-Gaussian distributions. This facilitates and justifies the application of Stein shrinkage to estimate the mean and variance parameters in BN and use it in image classification (segmentation) tasks with and without adversarial attacks. We present SOTA performance results using this Stein corrected batch norm in a standard ResNet architecture applied to the task of image classification using CIFAR-10 data, 3D CNN on PPMI (neuroimaging) data and image segmentation using HRNet on Cityscape data with and without adversarial attacks.

http://arxiv.org/abs/2507.08280
MIRRAMS: Towards Training Models Robust to Missingness Distribution Shifts. (1%)
Jihye Lee; Minseo Kang; Dongha Kim
In real-world data analysis, missingness distributional shifts between training and test input datasets frequently occur, posing a significant challenge to achieving robust prediction performance. In this study, we propose a novel deep learning framework designed to address such shifts in missingness distributions. We begin by introducing a set of mutual information-based conditions, called MI robustness conditions, which guide a prediction model to extract label-relevant information while remaining invariant to diverse missingness patterns, thereby enhancing robustness to unseen missingness scenarios at test-time. To make these conditions practical, we propose simple yet effective techniques to derive loss terms corresponding to each and formulate a final objective function, termed MIRRAMS(Mutual Information Regularization for Robustness Against Missingness Shifts). As a by-product, our analysis provides a theoretical interpretation of the principles underlying consistency regularization-based semi-supervised learning methods, such as FixMatch. Extensive experiments across various benchmark datasets show that MIRRAMS consistently outperforms existing baselines and maintains stable performance across diverse missingness scenarios. Moreover, our approach achieves state-of-the-art performance even without missing data and can be naturally extended to address semi-supervised learning tasks, highlighting MIRRAMS as a powerful, off-the-shelf framework for general-purpose learning.

http://arxiv.org/abs/2502.00718
"I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models. (99%)
Isha Gupta; David Khachaturov; Robert Mullins
The rise of multimodal large language models has introduced innovative human-machine interaction paradigms but also significant challenges in machine learning safety. Audio-Language Models (ALMs) are especially relevant due to the intuitive nature of spoken communication, yet little is known about their failure modes. This paper explores audio jailbreaks targeting ALMs, focusing on their ability to bypass alignment mechanisms. We construct adversarial perturbations that generalize across prompts, tasks, and even base audio samples, demonstrating the first universal jailbreaks in the audio modality, and show that these remain effective in simulated real-world conditions. Beyond demonstrating attack feasibility, we analyze how ALMs interpret these audio adversarial examples and reveal them to encode imperceptible first-person toxic speech - suggesting that the most effective perturbations for eliciting toxic outputs specifically embed linguistic features within the audio signal. These results have important implications for understanding the interactions between different modalities in multimodal models, and offer actionable insights for enhancing defenses against adversarial audio attacks.

http://arxiv.org/abs/2507.07776
SCOOTER: A Human Evaluation Framework for Unrestricted Adversarial Examples. (99%)
Dren Fazlija; Monty-Maximilian Zühlke; Johanna Schrader; Arkadij Orlov; Clara Stein; Iyiola E. Olatunji; Daniel Kudenko
Unrestricted adversarial attacks aim to fool computer vision models without being constrained by $\ell_p$-norm bounds to remain imperceptible to humans, for example, by changing an object's color. This allows attackers to circumvent traditional, norm-bounded defense strategies such as adversarial training or certified defense strategies. However, due to their unrestricted nature, there are also no guarantees of norm-based imperceptibility, necessitating human evaluations to verify just how authentic these adversarial examples look. While some related work assesses this vital quality of adversarial attacks, none provide statistically significant insights. This issue necessitates a unified framework that supports and streamlines such an assessment for evaluating and comparing unrestricted attacks. To close this gap, we introduce SCOOTER - an open-source, statistically powered framework for evaluating unrestricted adversarial examples. Our contributions are: $(i)$ best-practice guidelines for crowd-study power, compensation, and Likert equivalence bounds to measure imperceptibility; $(ii)$ the first large-scale human vs. model comparison across 346 human participants showing that three color-space attacks and three diffusion-based attacks fail to produce imperceptible images. Furthermore, we found that GPT-4o can serve as a preliminary test for imperceptibility, but it only consistently detects adversarial examples for four out of six tested attacks; $(iii)$ open-source software tools, including a browser-based task template to collect annotations and analysis scripts in Python and R; $(iv)$ an ImageNet-derived benchmark dataset containing 3K real images, 7K adversarial examples, and over 34K human ratings. Our findings demonstrate that automated vision systems do not align with human perception, reinforcing the need for a ground-truth SCOOTER benchmark.

http://arxiv.org/abs/2507.07768
TRIX- Trading Adversarial Fairness via Mixed Adversarial Training. (99%)
Tejaswini Medi; Steffen Jung; Margret Keuper
Adversarial Training (AT) is a widely adopted defense against adversarial examples. However, existing approaches typically apply a uniform training objective across all classes, overlooking disparities in class-wise vulnerability. This results in adversarial unfairness: classes with well distinguishable features (strong classes) tend to become more robust, while classes with overlapping or shared features(weak classes) remain disproportionately susceptible to adversarial attacks. We observe that strong classes do not require strong adversaries during training, as their non-robust features are quickly suppressed. In contrast, weak classes benefit from stronger adversaries to effectively reduce their vulnerabilities. Motivated by this, we introduce TRIX, a feature-aware adversarial training framework that adaptively assigns weaker targeted adversaries to strong classes, promoting feature diversity via uniformly sampled targets, and stronger untargeted adversaries to weak classes, enhancing their focused robustness. TRIX further incorporates per-class loss weighting and perturbation strength adjustments, building on prior work, to emphasize weak classes during the optimization. Comprehensive experiments on standard image classification benchmarks, including evaluations under strong attacks such as PGD and AutoAttack, demonstrate that TRIX significantly improves worst-case class accuracy on both clean and adversarial data, reducing inter-class robustness disparities, and preserves overall accuracy. Our results highlight TRIX as a practical step toward fair and effective adversarial defense.

http://arxiv.org/abs/2507.07709
One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models. (98%)
Jiale Zhao; Xinyang Jiang; Junyao Gao; Yuhao Xue; Cairong Zhao
Unified vision-language models(VLMs) have recently shown remarkable progress, enabling a single model to flexibly address diverse tasks through different instructions within a shared computational architecture. This instruction-based control mechanism creates unique security challenges, as adversarial inputs must remain effective across multiple task instructions that may be unpredictably applied to process the same malicious content. In this paper, we introduce CrossVLAD, a new benchmark dataset carefully curated from MSCOCO with GPT-4-assisted annotations for systematically evaluating cross-task adversarial attacks on unified VLMs. CrossVLAD centers on the object-change objective-consistently manipulating a target object's classification across four downstream tasks-and proposes a novel success rate metric that measures simultaneous misclassification across all tasks, providing a rigorous evaluation of adversarial transferability. To tackle this challenge, we present CRAFT (Cross-task Region-based Attack Framework with Token-alignment), an efficient region-centric attack method. Extensive experiments on Florence-2 and other popular unified VLMs demonstrate that our method outperforms existing approaches in both overall cross-task attack performance and targeted object-change success rates, highlighting its effectiveness in adversarially influencing unified VLMs across diverse tasks.

http://arxiv.org/abs/2507.08163
Adaptive Diffusion Denoised Smoothing : Certified Robustness via Randomized Smoothing with Differentially Private Guided Denoising Diffusion. (92%)
Frederick Shpilevskiy; Saiyue Lyu; Krishnamurthy Dj Dvijotham; Mathias Lécuyer; Pierre-André Noël
We propose Adaptive Diffusion Denoised Smoothing, a method for certifying the predictions of a vision model against adversarial examples, while adapting to the input. Our key insight is to reinterpret a guided denoising diffusion model as a long sequence of adaptive Gaussian Differentially Private (GDP) mechanisms refining a pure noise sample into an image. We show that these adaptive mechanisms can be composed through a GDP privacy filter to analyze the end-to-end robustness of the guided denoising process, yielding a provable certification that extends the adaptive randomized smoothing analysis. We demonstrate that our design, under a specific guiding strategy, can improve both certified accuracy and standard accuracy on ImageNet for an $\ell_2$ threat model.

http://arxiv.org/abs/2507.07974
Defending Against Prompt Injection With a Few DefensiveTokens. (81%)
Sizhe Chen; Yizhu Wang; Nicholas Carlini; Chawin Sitawarin; David Wagner
When large language model (LLM) systems interact with external data to perform complex tasks, a new attack, namely prompt injection, becomes a significant threat. By injecting instructions into the data accessed by the system, the attacker is able to override the initial user task with an arbitrary task directed by the attacker. To secure the system, test-time defenses, e.g., defensive prompting, have been proposed for system developers to attain security only when needed in a flexible manner. However, they are much less effective than training-time defenses that change the model parameters. Motivated by this, we propose DefensiveToken, a test-time defense with prompt injection robustness comparable to training-time alternatives. DefensiveTokens are newly inserted as special tokens, whose embeddings are optimized for security. In security-sensitive cases, system developers can append a few DefensiveTokens before the LLM input to achieve security with a minimal utility drop. In scenarios where security is less of a concern, developers can simply skip DefensiveTokens; the LLM system remains the same as there is no defense, generating high-quality responses. Thus, DefensiveTokens, if released alongside the model, allow a flexible switch between the state-of-the-art (SOTA) utility and almost-SOTA security at test time. The code is available at https://github.com/Sizhe-Chen/DefensiveToken.

http://arxiv.org/abs/2507.07850
Identifying the Smallest Adversarial Load Perturbations that Render DC-OPF Infeasible. (81%)
Samuel Chevalier; William A. Wheeler
What is the globally smallest load perturbation that renders DC-OPF infeasible? Reliably identifying such "adversarial attack" perturbations has useful applications in a variety of emerging grid-related contexts, including machine learning performance verification, cybersecurity, and operational robustness of power systems dominated by stochastic renewable energy resources. In this paper, we formulate the inherently nonconvex adversarial attack problem by applying a parameterized version of Farkas' lemma to a perturbed set of DC-OPF equations. Since the resulting formulation is very hard to globally optimize, we also propose a parameterized generation control policy which, when applied to the primal DC-OPF problem, provides solvability guarantees. Together, these nonconvex problems provide guaranteed upper and lower bounds on adversarial attack size; by combining them into a single optimization problem, we can efficiently "squeeze" these bounds towards a common global solution. We apply these methods on a range of small- to medium-sized test cases from PGLib, benchmarking our results against the best adversarial attack lower bounds provided by Gurobi 12.0's spatial Branch and Bound solver.

http://arxiv.org/abs/2507.08158
Beyond the Worst Case: Extending Differential Privacy Guarantees to Realistic Adversaries. (68%)
Marika Swanberg; Meenatchi Sundaram Muthu Selva Annamalai; Jamie Hayes; Borja Balle; Adam Smith
Differential Privacy (DP) is a family of definitions that bound the worst-case privacy leakage of a mechanism. One important feature of the worst-case DP guarantee is it naturally implies protections against adversaries with less prior information, more sophisticated attack goals, and complex measures of a successful attack. However, the analytical tradeoffs between the adversarial model and the privacy protections conferred by DP are not well understood thus far. To that end, this work sheds light on what the worst-case guarantee of DP implies about the success of attackers that are more representative of real-world privacy risks.
  In this paper, we present a single flexible framework that generalizes and extends the patchwork of bounds on DP mechanisms found in prior work. Our framework allows us to compute high-probability guarantees for DP mechanisms on a large family of natural attack settings that previous bounds do not capture. One class of such settings is the approximate reconstruction of multiple individuals' data, such as inferring nearly entire columns of a tabular data set from noisy marginals and extracting sensitive information from DP-trained language models.
  We conduct two empirical case studies to illustrate the versatility of our bounds and compare them to the success of state-of-the-art attacks. Specifically, we study attacks that extract non-uniform PII from a DP-trained language model, as well as multi-column reconstruction attacks where the adversary has access to some columns in the clear and attempts to reconstruct the remaining columns for each person's record. We find that the absolute privacy risk of attacking non-uniform data is highly dependent on the adversary's prior probability of success. Our high probability bounds give us a nuanced understanding of the privacy leakage of DP mechanisms in a variety of previously understudied attack settings.

http://arxiv.org/abs/2507.07417
May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks. (68%)
Nishit V. Pandya; Andrey Labunets; Sicun Gao; Earlence Fernandes
A popular class of defenses against prompt injection attacks on large language models (LLMs) relies on fine-tuning the model to separate instructions and data, so that the LLM does not follow instructions that might be present with data. There are several academic systems and production-level implementations of this idea. We evaluate the robustness of this class of prompt injection defenses in the whitebox setting by constructing strong optimization-based attacks and showing that the defenses do not provide the claimed security properties. Specifically, we construct a novel attention-based attack algorithm for text-based LLMs and apply it to two recent whitebox defenses SecAlign (CCS 2025) and StruQ (USENIX Security 2025), showing attacks with success rates of up to 70% with modest increase in attacker budget in terms of tokens. Our findings make fundamental progress towards understanding the robustness of prompt injection defenses in the whitebox setting. We release our code and attacks at https://github.com/nishitvp/better_opts_attacks

http://arxiv.org/abs/2507.08212
EvA: Evolutionary Attacks on Graphs. (26%)
Mohammad Sadegh Akhondzadeh; Soroush H. Zargarbashi; Jimin Cao; Aleksandar Bojchevski
Even a slight perturbation in the graph structure can cause a significant drop in the accuracy of graph neural networks (GNNs). Most existing attacks leverage gradient information to perturb edges. This relaxes the attack's optimization problem from a discrete to a continuous space, resulting in solutions far from optimal. It also restricts the adaptability of the attack to non-differentiable objectives. Instead, we introduce a few simple yet effective enhancements of an evolutionary-based algorithm to solve the discrete optimization problem directly. Our Evolutionary Attack (EvA) works with any black-box model and objective, eliminating the need for a differentiable proxy loss. This allows us to design two novel attacks that reduce the effectiveness of robustness certificates and break conformal sets. The memory complexity of our attack is linear in the attack budget. Among our experiments, EvA shows $\sim$11\% additional drop in accuracy on average compared to the best previous attack, revealing significant untapped potential in designing attacks.

http://arxiv.org/abs/2507.06850
The Dark Side of LLMs: Agent-based Attacks for Complete Computer Takeover. (10%)
Matteo Lupinacci; Francesco Aurelio Pironti; Francesco Blefari; Francesco Romeo; Luigi Arena; Angelo Furfaro
The rapid adoption of Large Language Model (LLM) agents and multi-agent systems enables unprecedented capabilities in natural language processing and generation. However, these systems have introduced unprecedented security vulnerabilities that extend beyond traditional prompt injection attacks. This paper presents the first comprehensive evaluation of LLM agents as attack vectors capable of achieving complete computer takeover through the exploitation of trust boundaries within agentic AI systems where autonomous entities interact and influence each other. We demonstrate that adversaries can leverage three distinct attack surfaces - direct prompt injection, RAG backdoor attacks, and inter-agent trust exploitation - to coerce popular LLMs (including GPT-4o, Claude-4 and Gemini-2.5) into autonomously installing and executing malware on victim machines. Our evaluation of 17 state-of-the-art LLMs reveals an alarming vulnerability hierarchy: while 41.2% of models succumb to direct prompt injection, 52.9% are vulnerable to RAG backdoor attacks, and a critical 82.4% can be compromised through inter-agent trust exploitation. Notably, we discovered that LLMs which successfully resist direct malicious commands will execute identical payloads when requested by peer agents, revealing a fundamental flaw in current multi-agent security models. Our findings demonstrate that only 5.9% of tested models (1/17) proved resistant to all attack vectors, with the majority exhibiting context-dependent security behaviors that create exploitable blind spots. Our findings also highlight the need to increase awareness and research on the security risks of LLMs, showing a paradigm shift in cybersecurity threats, where AI tools themselves become sophisticated attack vectors.

http://arxiv.org/abs/2507.07436
When Graph Contrastive Learning Backfires: Spectral Vulnerability and Defense in Recommendation. (3%)
Zongwei Wang; Min Gao; Junliang Yu; Shazia Sadiq; Hongzhi Yin; Ling Liu
Graph Contrastive Learning (GCL) has demonstrated substantial promise in enhancing the robustness and generalization of recommender systems, particularly by enabling models to leverage large-scale unlabeled data for improved representation learning. However, in this paper, we reveal an unexpected vulnerability: the integration of GCL inadvertently increases the susceptibility of a recommender to targeted promotion attacks. Through both theoretical investigation and empirical validation, we identify the root cause as the spectral smoothing effect induced by contrastive optimization, which disperses item embeddings across the representation space and unintentionally enhances the exposure of target items. Building on this insight, we introduce CLeaR, a bi-level optimization attack method that deliberately amplifies spectral smoothness, enabling a systematic investigation of the susceptibility of GCL-based recommendation models to targeted promotion attacks. Our findings highlight the urgent need for robust countermeasures; in response, we further propose SIM, a spectral irregularity mitigation framework designed to accurately detect and suppress targeted items without compromising model performance. Extensive experiments on multiple benchmark datasets demonstrate that, compared to existing targeted promotion attacks, GCL-based recommendation models exhibit greater susceptibility when evaluated with CLeaR, while SIM effectively mitigates these vulnerabilities.

http://arxiv.org/abs/2507.07438
Algorithmic Complexity Attacks on All Learned Cardinality Estimators: A Data-centric Approach. (3%)
Yingze Li; Xianglong Liu; Dong Wang; Zixuan Wang; Hongzhi Wang; Kaixing Zhang; Yiming Guan
Learned cardinality estimators show promise in query cardinality prediction, yet they universally exhibit fragility to training data drifts, posing risks for real-world deployment. This work is the first to theoretical investigate how minimal data-level drifts can maximally degrade the accuracy of learned estimators. We propose data-centric algorithmic complexity attacks against learned estimators in a black-box setting, proving that finding the optimal attack strategy is NP-Hard. To address this, we design a polynomial-time approximation algorithm with a $(1-κ)$ approximation ratio. Extensive experiments demonstrate our attack's effectiveness: on STATS-CEB and IMDB-JOB benchmarks, modifying just 0.8\% of training tuples increases the 90th percentile Qerror by three orders of magnitude and raises end-to-end processing time by up to 20$\times$. Our work not only reveals critical vulnerabilities in deployed learned estimators but also provides the first unified worst-case theoretical analysis of their fragility under data updates. Additionally, we identify two countermeasures to mitigate such black-box attacks, offering insights for developing robust learned database optimizers.

http://arxiv.org/abs/2505.19598
Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Study. (2%)
Guanyu Hou; Jiaming He; Yinhang Zhou; Ji Guo; Yitong Qiao; Rui Zhang; Wenbo Jiang
Large Audio-Language Models (LALMs) are increasingly deployed in real-world applications, yet their robustness against malicious audio injection attacks remains underexplored. This study systematically evaluates five leading LALMs across four attack scenarios: Audio Interference Attack, Instruction Following Attack, Context Injection Attack, and Judgment Hijacking Attack. Using metrics like Defense Success Rate, Context Robustness Score, and Judgment Robustness Index, their vulnerabilities and resilience were quantitatively assessed. Experimental results reveal significant performance disparities among models; no single model consistently outperforms others across all attack types. The position of malicious content critically influences attack effectiveness, particularly when placed at the beginning of sequences. A negative correlation between instruction-following capability and robustness suggests models adhering strictly to instructions may be more susceptible, contrasting with greater resistance by safety-aligned models. Additionally, system prompts show mixed effectiveness, indicating the need for tailored strategies. This work introduces a benchmark framework and highlights the importance of integrating robustness into training pipelines. Findings emphasize developing multi-modal defenses and architectural designs that decouple capability from susceptibility for secure LALMs deployment.

http://arxiv.org/abs/2507.01788
Are Vision Transformer Representations Semantically Meaningful? A Case Study in Medical Imaging. (2%)
Montasir Shams; Chashi Mahiul Islam; Shaeke Salman; Phat Tran; Xiuwen Liu
Vision transformers (ViTs) have rapidly gained prominence in medical imaging tasks such as disease classification, segmentation, and detection due to their superior accuracy compared to conventional deep learning models. However, due to their size and complex interactions via the self-attention mechanism, they are not well understood. In particular, it is unclear whether the representations produced by such models are semantically meaningful. In this paper, using a projected gradient-based algorithm, we show that their representations are not semantically meaningful and they are inherently vulnerable to small changes. Images with imperceptible differences can have very different representations; on the other hand, images that should belong to different semantic classes can have nearly identical representations. Such vulnerability can lead to unreliable classification results; for example, unnoticeable changes cause the classification accuracy to be reduced by over 60\%. %. To the best of our knowledge, this is the first work to systematically demonstrate this fundamental lack of semantic meaningfulness in ViT representations for medical image classification, revealing a critical challenge for their deployment in safety-critical systems.

http://arxiv.org/abs/2507.08207
A Dynamic Stackelberg Game Framework for Agentic AI Defense Against LLM Jailbreaking. (1%)
Zhengye Han; Quanyan Zhu
As large language models (LLMs) are increasingly deployed in critical applications, the challenge of jailbreaking, where adversaries manipulate the models to bypass safety mechanisms, has become a significant concern. This paper presents a dynamic Stackelberg game framework to model the interactions between attackers and defenders in the context of LLM jailbreaking. The framework treats the prompt-response dynamics as a sequential extensive-form game, where the defender, as the leader, commits to a strategy while anticipating the attacker's optimal responses. We propose a novel agentic AI solution, the "Purple Agent," which integrates adversarial exploration and defensive strategies using Rapidly-exploring Random Trees (RRT). The Purple Agent actively simulates potential attack trajectories and intervenes proactively to prevent harmful outputs. This approach offers a principled method for analyzing adversarial dynamics and provides a foundation for mitigating the risk of jailbreaking.

http://arxiv.org/abs/2507.07700
Rethinking the Privacy of Text Embeddings: A Reproducibility Study of "Text Embeddings Reveal (Almost) As Much As Text". (1%)
Dominykas Seputis; Yongkang Li; Karsten Langerak; Serghei Mihailov
Text embeddings are fundamental to many natural language processing (NLP) tasks, extensively applied in domains such as recommendation systems and information retrieval (IR). Traditionally, transmitting embeddings instead of raw text has been seen as privacy-preserving. However, recent methods such as Vec2Text challenge this assumption by demonstrating that controlled decoding can successfully reconstruct original texts from black-box embeddings. The unexpectedly strong results reported by Vec2Text motivated us to conduct further verification, particularly considering the typically non-intuitive and opaque structure of high-dimensional embedding spaces. In this work, we reproduce the Vec2Text framework and evaluate it from two perspectives: (1) validating the original claims, and (2) extending the study through targeted experiments. First, we successfully replicate the original key results in both in-domain and out-of-domain settings, with only minor discrepancies arising due to missing artifacts, such as model checkpoints and dataset splits. Furthermore, we extend the study by conducting a parameter sensitivity analysis, evaluating the feasibility of reconstructing sensitive inputs (e.g., passwords), and exploring embedding quantization as a lightweight privacy defense. Our results show that Vec2Text is effective under ideal conditions, capable of reconstructing even password-like sequences that lack clear semantics. However, we identify key limitations, including its sensitivity to input sequence length. We also find that Gaussian noise and quantization techniques can mitigate the privacy risks posed by Vec2Text, with quantization offering a simpler and more widely applicable solution. Our findings emphasize the need for caution in using text embeddings and highlight the importance of further research into robust defense mechanisms for NLP systems.

http://arxiv.org/abs/2507.07259
Exploiting Edge Features for Transferable Adversarial Attacks in Distributed Machine Learning. (99%)
Giulio Rossolini; Fabio Brau; Alessandro Biondi; Battista Biggio; Giorgio Buttazzo
As machine learning models become increasingly deployed across the edge of internet of things environments, a partitioned deep learning paradigm in which models are split across multiple computational nodes introduces a new dimension of security risk. Unlike traditional inference setups, these distributed pipelines span the model computation across heterogeneous nodes and communication layers, thereby exposing a broader attack surface to potential adversaries. Building on these motivations, this work explores a previously overlooked vulnerability: even when both the edge and cloud components of the model are inaccessible (i.e., black-box), an adversary who intercepts the intermediate features transmitted between them can still pose a serious threat. We demonstrate that, under these mild and realistic assumptions, an attacker can craft highly transferable proxy models, making the entire deep learning system significantly more vulnerable to evasion attacks. In particular, the intercepted features can be effectively analyzed and leveraged to distill surrogate models capable of crafting highly transferable adversarial examples against the target model. To this end, we propose an exploitation strategy specifically designed for distributed settings, which involves reconstructing the original tensor shape from vectorized transmitted features using simple statistical analysis, and adapting surrogate architectures accordingly to enable effective feature distillation. A comprehensive and systematic experimental evaluation has been conducted to demonstrate that surrogate models trained with the proposed strategy, i.e., leveraging intermediate features, tremendously improve the transferability of adversarial attacks. These findings underscore the urgent need to account for intermediate feature leakage in the design of secure distributed deep learning systems.

http://arxiv.org/abs/2507.06907
Robust and Safe Traffic Sign Recognition using N-version with Weighted Voting. (98%)
Linyun Gao; Qiang Wen; Fumio Machida
Autonomous driving is rapidly advancing as a key application of machine learning, yet ensuring the safety of these systems remains a critical challenge. Traffic sign recognition, an essential component of autonomous vehicles, is particularly vulnerable to adversarial attacks that can compromise driving safety. In this paper, we propose an N-version machine learning (NVML) framework that integrates a safety-aware weighted soft voting mechanism. Our approach utilizes Failure Mode and Effects Analysis (FMEA) to assess potential safety risks and assign dynamic, safety-aware weights to the ensemble outputs. We evaluate the robustness of three-version NVML systems employing various voting mechanisms against adversarial samples generated using the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. Experimental results demonstrate that our NVML approach significantly enhances the robustness and safety of traffic sign recognition systems under adversarial conditions.

http://arxiv.org/abs/2305.13651
Adversarial Defenses via Vector Quantization. (98%)
Zhiyi Dong; Yongyi Mao
Adversarial attacks pose significant challenges to the robustness of modern deep neural networks in computer vision, and defending these networks against adversarial attacks has attracted intense research efforts. Among various defense strategies, preprocessing-based defenses are practically appealing since there is no need to train the network under protection. However, such approaches typically do not achieve comparable robustness as other methods such as adversarial training. In this paper, we propose a novel framework for preprocessing-based defenses, where a vector quantizer is used as a preprocessor. This framework, inspired by and extended from Randomized Discretization (RandDisc), is theoretically principled by rate-distortion theory: indeed, RandDisc may be viewed as a scalar quantizer, and rate-distortion theory suggests that such quantization schemes are inferior to vector quantization. In our framework, the preprocessing vector quantizer treats the input image as a collection of patches and finds a set of representative patches based on the patch distributions; each original patch is then modified according to the representative patches close to it. We present two lightweight defenses in this framework, referred to as patched RandDisc (pRD) and sliding-window RandDisc (swRD), where the patches are disjoint in the former and overlapping in the latter. We show that vector-quantization-based defenses have certifiable robust accuracy and that pRD and swRD demonstrate state-of-the-art performances, surpassing RandDisc by a large margin. Notably, the proposed defenses possess the obfuscated gradients property. Our experiments however show that pRD and swRD remain effective under the STE and EOT attacks, which are designed specifically for defenses with gradient obfuscation. ...

http://arxiv.org/abs/2507.06856
IAP: Invisible Adversarial Patch Attack through Perceptibility-Aware Localization and Perturbation Optimization. (92%)
Subrat Kishore Dutta; Xiao Zhang
Despite modifying only a small localized input region, adversarial patches can drastically change the prediction of computer vision models. However, prior methods either cannot perform satisfactorily under targeted attack scenarios or fail to produce contextually coherent adversarial patches, causing them to be easily noticeable by human examiners and insufficiently stealthy against automatic patch defenses. In this paper, we introduce IAP, a novel attack framework that generates highly invisible adversarial patches based on perceptibility-aware localization and perturbation optimization schemes. Specifically, IAP first searches for a proper location to place the patch by leveraging classwise localization and sensitivity maps, balancing the susceptibility of patch location to both victim model prediction and human visual system, then employs a perceptibility-regularized adversarial loss and a gradient update rule that prioritizes color constancy for optimizing invisible perturbations. Comprehensive experiments across various image benchmarks and model architectures demonstrate that IAP consistently achieves competitive attack success rates in targeted settings with significantly improved patch invisibility compared to existing baselines. In addition to being highly imperceptible to humans, IAP is shown to be stealthy enough to render several state-of-the-art patch defenses ineffective.

http://arxiv.org/abs/2507.08862
RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation. (86%)
Tianzhe Zhao; Jiaoyan Chen; Yanchi Ru; Haiping Zhu; Nan Hu; Jun Liu; Qika Lin
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by retrieving external data to mitigate hallucinations and outdated knowledge issues. Benefiting from the strong ability in facilitating diverse data sources and supporting faithful reasoning, knowledge graphs (KGs) have been increasingly adopted in RAG systems, giving rise to KG-based RAG (KG-RAG) methods. Though RAG systems are widely applied in various applications, recent studies have also revealed its vulnerabilities to data poisoning attacks, where malicious information injected into external knowledge sources can mislead the system into producing incorrect or harmful responses. However, these studies focus exclusively on RAG systems using unstructured textual data sources, leaving the security risks of KG-RAG largely unexplored, despite the fact that KGs present unique vulnerabilities due to their structured and editable nature. In this work, we conduct the first systematic investigation of the security issue of KG-RAG methods through data poisoning attacks. To this end, we introduce a practical, stealthy attack setting that aligns with real-world implementation. We propose an attack strategy that first identifies adversarial target answers and then inserts perturbation triples to complete misleading inference chains in the KG, increasing the likelihood that KG-RAG methods retrieve and rely on these perturbations during generation. Through extensive experiments on two benchmarks and four recent KG-RAG methods, our attack strategy demonstrates strong effectiveness in degrading KG-RAG performance, even with minimal KG perturbations. In-depth analyses are also conducted to understand the safety threats within the internal stages of KG-RAG systems and to explore the robustness of LLMs against adversarial knowledge.

http://arxiv.org/abs/2507.07139
Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning. (50%)
Renyang Liu; Guanlin Li; Tianwei Zhang; See-Kiong Ng
Recent advances in image generation models (IGMs), particularly diffusion-based architectures such as Stable Diffusion (SD), have markedly enhanced the quality and diversity of AI-generated visual content. However, their generative capability has also raised significant ethical, legal, and societal concerns, including the potential to produce harmful, misleading, or copyright-infringing content. To mitigate these concerns, machine unlearning (MU) emerges as a promising solution by selectively removing undesirable concepts from pretrained models. Nevertheless, the robustness and effectiveness of existing unlearning techniques remain largely unexplored, particularly in the presence of multi-modal adversarial inputs.
  To bridge this gap, we propose Recall, a novel adversarial framework explicitly designed to compromise the robustness of unlearned IGMs. Unlike existing approaches that predominantly rely on adversarial text prompts, Recall exploits the intrinsic multi-modal conditioning capabilities of diffusion models by efficiently optimizing adversarial image prompts with guidance from a single semantically relevant reference image. Extensive experiments across ten state-of-the-art unlearning methods and diverse tasks show that Recall consistently outperforms existing baselines in terms of adversarial effectiveness, computational efficiency, and semantic fidelity with the original textual prompt. These findings reveal critical vulnerabilities in current unlearning mechanisms and underscore the need for more robust solutions to ensure the safety and reliability of generative models. Code and data are publicly available at \textcolor{blue}{https://github.com/ryliu68/RECALL}.

http://arxiv.org/abs/2507.06489
On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks. (13%)
Stephen Obadinma; Xiaodan Zhu
Robust verbal confidence generated by large language models (LLMs) is crucial for the deployment of LLMs to ensure transparency, trust, and safety in human-AI interactions across many high-stakes applications. In this paper, we present the first comprehensive study on the robustness of verbal confidence under adversarial attacks. We introduce a novel framework for attacking verbal confidence scores through both perturbation and jailbreak-based methods, and show that these attacks can significantly jeopardize verbal confidence estimates and lead to frequent answer changes. We examine a variety of prompting strategies, model sizes, and application domains, revealing that current confidence elicitation methods are vulnerable and that commonly used defence techniques are largely ineffective or counterproductive. Our findings underscore the urgent need to design more robust mechanisms for confidence expression in LLMs, as even subtle semantic-preserving modifications can lead to misleading confidence in responses.

http://arxiv.org/abs/2507.06969
Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy. (1%)
Bogdan Kulynych; Juan Felipe Gomez; Georgios Kaissis; Jamie Hayes; Borja Balle; Flavio du Pin Calmon; Jean Louis Raisaro
Differentially private (DP) mechanisms are difficult to interpret and calibrate because existing methods for mapping standard privacy parameters to concrete privacy risks -- re-identification, attribute inference, and data reconstruction -- are both overly pessimistic and inconsistent. In this work, we use the hypothesis-testing interpretation of DP ($f$-DP), and determine that bounds on attack success can take the same unified form across re-identification, attribute inference, and data reconstruction risks. Our unified bounds are (1) consistent across a multitude of attack settings, and (2) tunable, enabling practitioners to evaluate risk with respect to arbitrary (including worst-case) levels of baseline risk. Empirically, our results are tighter than prior methods using $\varepsilon$-DP, Rényi DP, and concentrated DP. As a result, calibrating noise using our bounds can reduce the required noise by 20% at the same risk level, which yields, e.g., more than 15pp accuracy increase in a text classification task. Overall, this unifying perspective provides a principled framework for interpreting and calibrating the degree of protection in DP against specific levels of re-identification, attribute inference, or data reconstruction risk.

http://arxiv.org/abs/2507.06078
ScoreAdv: Score-based Targeted Generation of Natural Adversarial Examples via Diffusion Models. (99%)
Chihan Huang; Hao Tang
Despite the success of deep learning across various domains, it remains vulnerable to adversarial attacks. Although many existing adversarial attack methods achieve high success rates, they typically rely on $\ell_{p}$-norm perturbation constraints, which do not align with human perceptual capabilities. Consequently, researchers have shifted their focus toward generating natural, unrestricted adversarial examples (UAEs). GAN-based approaches suffer from inherent limitations, such as poor image quality due to instability and mode collapse. Meanwhile, diffusion models have been employed for UAE generation, but they still rely on iterative PGD perturbation injection, without fully leveraging their central denoising capabilities. In this paper, we introduce a novel approach for generating UAEs based on diffusion models, named ScoreAdv. This method incorporates an interpretable adversarial guidance mechanism to gradually shift the sampling distribution towards the adversarial distribution, while using an interpretable saliency map to inject the visual information of a reference image into the generated samples. Notably, our method is capable of generating an unlimited number of natural adversarial examples and can attack not only classification models but also retrieval models. We conduct extensive experiments on ImageNet and CelebA datasets, validating the performance of ScoreAdv across ten target models in both black-box and white-box settings. Our results demonstrate that ScoreAdv achieves state-of-the-art attack success rates and image quality. Furthermore, the dynamic balance between denoising and adversarial perturbation enables ScoreAdv to remain robust even under defensive measures.

http://arxiv.org/abs/2507.05622
DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspective. (98%)
Shuo Shao; Yiming Li; Mengren Zheng; Zhiyang Hu; Yukun Chen; Boheng Li; Yu He; Junfeng Guo; Tianwei Zhang; Dacheng Tao; Zhan Qin
The widespread application of Deep Learning across diverse domains hinges critically on the quality and composition of training datasets. However, the common lack of disclosure regarding their usage raises significant privacy and copyright concerns. Dataset auditing techniques, which aim to determine if a specific dataset was used to train a given suspicious model, provide promising solutions to addressing these transparency gaps. While prior work has developed various auditing methods, their resilience against dedicated adversarial attacks remains largely unexplored. To bridge the gap, this paper initiates a comprehensive study evaluating dataset auditing from an adversarial perspective. We start with introducing a novel taxonomy, classifying existing methods based on their reliance on internal features (IF) (inherent to the data) versus external features (EF) (artificially introduced for auditing). Subsequently, we formulate two primary attack types: evasion attacks, designed to conceal the use of a dataset, and forgery attacks, intending to falsely implicate an unused dataset. Building on the understanding of existing methods and attack objectives, we further propose systematic attack strategies: decoupling, removal, and detection for evasion; adversarial example-based methods for forgery. These formulations and strategies lead to our new benchmark, DATABench, comprising 17 evasion attacks, 5 forgery attacks, and 9 representative auditing methods. Extensive evaluations using DATABench reveal that none of the evaluated auditing methods are sufficiently robust or distinctive under adversarial settings. These findings underscore the urgent need for developing a more secure and reliable dataset auditing method capable of withstanding sophisticated adversarial manipulation. Code is available at https://github.com/shaoshuo-ss/DATABench.

http://arxiv.org/abs/2507.06419
Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling. (84%)
Pankayaraj Pathmanathan; Furong Huang
Reward modeling (RM), which captures human preferences to align large language models (LLMs), is increasingly employed in tasks such as model finetuning, response filtering, and ranking. However, due to the inherent complexity of human preferences and the limited coverage of available datasets, reward models often fail under distributional shifts or adversarial perturbations. Existing approaches for identifying such failure modes typically rely on prior knowledge about preference distributions or failure attributes, limiting their practicality in real-world settings where such information is unavailable. In this work, we propose a tractable, preference-distribution agnostic method for discovering reward model failure modes via reward guided controlled decoding. Building on this, we introduce REFORM, a self-improving reward modeling framework that enhances robustness by using the reward model itself to guide the generation of falsely scored responses. These adversarial examples are then used to augment the training data and patch the reward model's misaligned behavior. We evaluate REFORM on two widely used preference datasets Anthropic Helpful Harmless (HH) and PKU Beavertails and demonstrate that it significantly improves robustness without sacrificing reward quality. Notably, REFORM preserves performance both in direct evaluation and in downstream policy training, and further improves alignment quality by removing spurious correlations.

http://arxiv.org/abs/2507.05980
RabakBench: Scaling Human Annotations to Construct Localized Multilingual Safety Benchmarks for Low-Resource Languages. (64%)
Gabriel Chua; Leanne Tan; Ziyu Ge; Roy Ka-Wei Lee
Large language models (LLMs) and their safety classifiers often perform poorly on low-resource languages due to limited training data and evaluation benchmarks. This paper introduces RabakBench, a new multilingual safety benchmark localized to Singapore's unique linguistic context, covering Singlish, Chinese, Malay, and Tamil. RabakBench is constructed through a scalable three-stage pipeline: (i) Generate - adversarial example generation by augmenting real Singlish web content with LLM-driven red teaming; (ii) Label - semi-automated multi-label safety annotation using majority-voted LLM labelers aligned with human judgments; and (iii) Translate - high-fidelity translation preserving linguistic nuance and toxicity across languages. The final dataset comprises over 5,000 safety-labeled examples across four languages and six fine-grained safety categories with severity levels. Evaluations of 11 popular open-source and closed-source guardrail classifiers reveal significant performance degradation. RabakBench not only enables robust safety evaluation in Southeast Asian multilingual settings but also offers a reproducible framework for building localized safety datasets in low-resource environments. The benchmark dataset, including the human-verified translations, and evaluation code are publicly available.

http://arxiv.org/abs/2507.06332
AR2: Attention-Guided Repair for the Robustness of CNNs Against Common Corruptions. (56%)
Fuyuan Zhang; Qichen Wang; Jianjun Zhao
Deep neural networks suffer from significant performance degradation when exposed to common corruptions such as noise, blur, weather, and digital distortions, limiting their reliability in real-world applications. In this paper, we propose AR2 (Attention-Guided Repair for Robustness), a simple yet effective method to enhance the corruption robustness of pretrained CNNs. AR2 operates by explicitly aligning the class activation maps (CAMs) between clean and corrupted images, encouraging the model to maintain consistent attention even under input perturbations. Our approach follows an iterative repair strategy that alternates between CAM-guided refinement and standard fine-tuning, without requiring architectural changes. Extensive experiments show that AR2 consistently outperforms existing state-of-the-art methods in restoring robustness on standard corruption benchmarks (CIFAR-10-C, CIFAR-100-C and ImageNet-C), achieving a favorable balance between accuracy on clean data and corruption robustness. These results demonstrate that AR2 provides a robust and scalable solution for enhancing model reliability in real-world environments with diverse corruptions.

http://arxiv.org/abs/2507.08020
Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation. (11%)
Zhibo Zhang; Yuxi Li; Kailong Wang; Shuai Yuan; Ling Shi; Haoyu Wang
Large Language Models (LLMs) have achieved remarkable success across domains such as healthcare, education, and cybersecurity. However, this openness also introduces significant security risks, particularly through embedding space poisoning, which is a subtle attack vector where adversaries manipulate the internal semantic representations of input data to bypass safety alignment mechanisms. While previous research has investigated universal perturbation methods, the dynamics of LLM safety alignment at the embedding level remain insufficiently understood. Consequently, more targeted and accurate adversarial perturbation techniques, which pose significant threats, have not been adequately studied.
  In this work, we propose ETTA (Embedding Transformation Toxicity Attenuation), a novel framework that identifies and attenuates toxicity-sensitive dimensions in embedding space via linear transformations. ETTA bypasses model refusal behaviors while preserving linguistic coherence, without requiring model fine-tuning or access to training data. Evaluated on five representative open-source LLMs using the AdvBench benchmark, ETTA achieves a high average attack success rate of 88.61%, outperforming the best baseline by 11.34%, and generalizes to safety-enhanced models (e.g., 77.39% ASR on instruction-tuned defenses). These results highlight a critical vulnerability in current alignment strategies and underscore the need for embedding-aware defenses.

http://arxiv.org/abs/2507.06282
The bitter lesson of misuse detection. (10%)
Hadrien Mariaccia; Charbel-Raphaël Segerie; Diego Dorn
Prior work on jailbreak detection has established the importance of adversarial robustness for LLMs but has largely focused on the model ability to resist adversarial inputs and to output safe content, rather than the effectiveness of external supervision systems. The only public and independent benchmark of these guardrails to date evaluates a narrow set of supervisors on limited scenarios. Consequently, no comprehensive public benchmark yet verifies how well supervision systems from the market perform under realistic, diverse attacks. To address this, we introduce BELLS, a Benchmark for the Evaluation of LLM Supervision Systems. The framework is two dimensional: harm severity (benign, borderline, harmful) and adversarial sophistication (direct vs. jailbreak) and provides a rich dataset covering 3 jailbreak families and 11 harm categories. Our evaluations reveal drastic limitations of specialized supervision systems. While they recognize some known jailbreak patterns, their semantic understanding and generalization capabilities are very limited, sometimes with detection rates close to zero when asking a harmful question directly or with a new jailbreak technique such as base64 encoding. Simply asking generalist LLMs if the user question is "harmful or not" largely outperforms these supervisors from the market according to our BELLS score. But frontier LLMs still suffer from metacognitive incoherence, often responding to queries they correctly identify as harmful (up to 30 percent for Claude 3.7 and greater than 50 percent for Mistral Large). These results suggest that simple scaffolding could significantly improve misuse detection robustness, but more research is needed to assess the tradeoffs of such techniques. Our results support the "bitter lesson" of misuse detection: general capabilities of LLMs are necessary to detect a diverse array of misuses and jailbreaks.

http://arxiv.org/abs/2507.05660
TuneShield: Mitigating Toxicity in Conversational AI while Fine-tuning on Untrusted Data. (1%)
Aravind Cheruvu; Shravya Kanchi; Sifat Muhammad Abdullah; Nicholas Kong; Daphne Yao; Murtuza Jadliwala; Bimal Viswanath
Recent advances in foundation models, such as LLMs, have revolutionized conversational AI. Chatbots are increasingly being developed by customizing LLMs on specific conversational datasets. However, mitigating toxicity during this customization, especially when dealing with untrusted training data, remains a significant challenge. To address this, we introduce TuneShield, a defense framework designed to mitigate toxicity during chatbot fine-tuning while preserving conversational quality. TuneShield leverages LLM-based toxicity classification, utilizing the instruction-following capabilities and safety alignment of LLMs to effectively identify toxic samples, outperforming industry API services. TuneShield generates synthetic conversation samples, termed 'healing data', based on the identified toxic samples, using them to mitigate toxicity while reinforcing desirable behavior during fine-tuning. It performs an alignment process to further nudge the chatbot towards producing desired responses. Our findings show that TuneShield effectively mitigates toxicity injection attacks while preserving conversational quality, even when the toxicity classifiers are imperfect or biased. TuneShield proves to be resilient against adaptive adversarial and jailbreak attacks. Additionally, TuneShield demonstrates effectiveness in mitigating adaptive toxicity injection attacks during dialog-based learning (DBL).

http://arxiv.org/abs/2507.06256
Attacker's Noise Can Manipulate Your Audio-based LLM in the Real World. (54%)
Vinu Sankar Sadasivan; Soheil Feizi; Rajiv Mathews; Lun Wang
This paper investigates the real-world vulnerabilities of audio-based large language models (ALLMs), such as Qwen2-Audio. We first demonstrate that an adversary can craft stealthy audio perturbations to manipulate ALLMs into exhibiting specific targeted behaviors, such as eliciting responses to wake-keywords (e.g., "Hey Qwen"), or triggering harmful behaviors (e.g. "Change my calendar event"). Subsequently, we show that playing adversarial background noise during user interaction with the ALLMs can significantly degrade the response quality. Crucially, our research illustrates the scalability of these attacks to real-world scenarios, impacting other innocent users when these adversarial noises are played through the air. Further, we discuss the transferrability of the attack, and potential defensive measures.

http://arxiv.org/abs/2507.06258
Phantom Subgroup Poisoning: Stealth Attacks on Federated Recommender Systems. (26%)
Bo Yan; Yurong Hao; Dingqi Liu; Huabin Sun; Pengpeng Qiao; Wei Yang Bryan Lim; Yang Cao; Chuan Shi
Federated recommender systems (FedRec) have emerged as a promising solution for delivering personalized recommendations while safeguarding user privacy. However, recent studies have demonstrated their vulnerability to poisoning attacks. Existing attacks typically target the entire user group, which compromises stealth and increases the risk of detection. In contrast, real-world adversaries may prefer to prompt target items to specific user subgroups, such as recommending health supplements to elderly users. Motivated by this gap, we introduce Spattack, the first targeted poisoning attack designed to manipulate recommendations for specific user subgroups in the federated setting. Specifically, Spattack adopts a two-stage approximation-and-promotion strategy, which first simulates user embeddings of target/non-target subgroups and then prompts target items to the target subgroups. To enhance the approximation stage, we push the inter-group embeddings away based on contrastive learning and augment the target group's relevant item set based on clustering. To enhance the promotion stage, we further propose to adaptively tune the optimization weights between target and non-target subgroups. Besides, an embedding alignment strategy is proposed to align the embeddings between the target items and the relevant items. We conduct comprehensive experiments on three real-world datasets, comparing Spattack against seven state-of-the-art poisoning attacks and seven representative defense mechanisms. Experimental results demonstrate that Spattack consistently achieves strong manipulation performance on the specific user subgroup, while incurring minimal impact on non-target users, even when only 0.1\% of users are malicious. Moreover, Spattack maintains competitive overall recommendation performance and exhibits strong resilience against existing mainstream defenses.

http://arxiv.org/abs/2507.05531
Bit-Flip Fault Attack: Crushing Graph Neural Networks via Gradual Bit Search. (13%)
Sanaz Kazemi Abharian; Sai Manoj Pudukotai Dinakarrao
Graph Neural Networks (GNNs) have emerged as a powerful machine learning method for graph-structured data. A plethora of hardware accelerators has been introduced to meet the performance demands of GNNs in real-world applications. However, security challenges of hardware-based attacks have been generally overlooked. In this paper, we investigate the vulnerability of GNN models to hardware-based fault attack, wherein an attacker attempts to misclassify output by modifying trained weight parameters through fault injection in a memory device. Thus, we propose Gradual Bit-Flip Fault Attack (GBFA), a layer-aware bit-flip fault attack, selecting a vulnerable bit in each selected weight gradually to compromise the GNN's performance by flipping a minimal number of bits. To achieve this, GBFA operates in two steps. First, a Markov model is created to predict the execution sequence of layers based on features extracted from memory access patterns, enabling the launch of the attack within a specific layer. Subsequently, GBFA identifies vulnerable bits within the selected weights using gradient ranking through an in-layer search. We evaluate the effectiveness of the proposed GBFA attack on various GNN models for node classification tasks using the Cora and PubMed datasets. Our findings show that GBFA significantly degrades prediction accuracy, and the variation in its impact across different layers highlights the importance of adopting a layer-aware attack strategy in GNNs. For example, GBFA degrades GraphSAGE's prediction accuracy by 17% on the Cora dataset with only a single bit flip in the last layer.

http://arxiv.org/abs/2507.05093
The Hidden Threat in Plain Text: Attacking RAG Data Loaders. (13%)
Alberto Castagnaro; Umberto Salviati; Mauro Conti; Luca Pajola; Simeone Pizzi
Large Language Models (LLMs) have transformed human-machine interaction since ChatGPT's 2022 debut, with Retrieval-Augmented Generation (RAG) emerging as a key framework that enhances LLM outputs by integrating external knowledge. However, RAG's reliance on ingesting external documents introduces new vulnerabilities. This paper exposes a critical security gap at the data loading stage, where malicious actors can stealthily corrupt RAG pipelines by exploiting document ingestion.
  We propose a taxonomy of 9 knowledge-based poisoning attacks and introduce two novel threat vectors -- Content Obfuscation and Content Injection -- targeting common formats (DOCX, HTML, PDF). Using an automated toolkit implementing 19 stealthy injection techniques, we test five popular data loaders, finding a 74.4% attack success rate across 357 scenarios. We further validate these threats on six end-to-end RAG systems -- including white-box pipelines and black-box services like NotebookLM and OpenAI Assistants -- demonstrating high success rates and critical vulnerabilities that bypass filters and silently compromise output integrity. Our results emphasize the urgent need to secure the document ingestion process in RAG systems against covert content manipulations.

http://arxiv.org/abs/2507.04726
Losing Control: Data Poisoning Attack on Guided Diffusion via ControlNet. (11%)
Raz Lapid; Almog Dubin
Text-to-image diffusion models have achieved remarkable success in translating textual prompts into high-fidelity images. ControlNets further extend these models by allowing precise, image-based conditioning (e.g., edge maps, depth, pose), enabling fine-grained control over structure and style. However, their dependence on large, publicly scraped datasets -- and the increasing use of community-shared data for fine-tuning -- exposes them to stealthy data poisoning attacks. In this work, we introduce a novel data poisoning method that manipulates ControlNets to generate images containing specific content without any text triggers. By injecting poisoned samples -- each pairing a subtly triggered input with an NSFW target -- the model retains clean-prompt fidelity yet reliably produces NSFW outputs when the trigger is present. On large-scale, high-quality datasets, our backdoor achieves high attack success rate while remaining imperceptible in raw inputs. These results reveal a critical vulnerability in open-source ControlNets pipelines and underscore the need for robust data sanitization and defense mechanisms.

http://arxiv.org/abs/2507.04883
Beyond Training-time Poisoning: Component-level and Post-training Backdoors in Deep Reinforcement Learning. (10%)
Sanyam Vyas; Alberto Caron; Chris Hicks; Pete Burnap; Vasilios Mavroudis
Deep Reinforcement Learning (DRL) systems are increasingly used in safety-critical applications, yet their security remains severely underexplored. This work investigates backdoor attacks, which implant hidden triggers that cause malicious actions only when specific inputs appear in the observation space. Existing DRL backdoor research focuses solely on training-time attacks requiring unrealistic access to the training pipeline. In contrast, we reveal critical vulnerabilities across the DRL supply chain where backdoors can be embedded with significantly reduced adversarial privileges. We introduce two novel attacks: (1) TrojanentRL, which exploits component-level flaws to implant a persistent backdoor that survives full model retraining; and (2) InfrectroRL, a post-training backdoor attack which requires no access to training, validation, nor test data. Empirical and analytical evaluations across six Atari environments show our attacks rival state-of-the-art training-time backdoor attacks while operating under much stricter adversarial constraints. We also demonstrate that InfrectroRL further evades two leading DRL backdoor defenses. These findings challenge the current research focus and highlight the urgent need for robust defenses.

http://arxiv.org/abs/2507.05512
Disappearing Ink: Obfuscation Breaks N-gram Code Watermarks in Theory and Practice. (5%)
Gehao Zhang; Eugene Bagdasarian; Juan Zhai; Shiqing Ma
Distinguishing AI-generated code from human-written code is becoming crucial for tasks such as authorship attribution, content tracking, and misuse detection. Based on this, N-gram-based watermarking schemes have emerged as prominent, which inject secret watermarks to be detected during the generation.
  However, their robustness in code content remains insufficiently evaluated. Most claims rely solely on defenses against simple code transformations or code optimizations as a simulation of attack, creating a questionable sense of robustness. In contrast, more sophisticated schemes already exist in the software engineering world, e.g., code obfuscation, which significantly alters code while preserving functionality. Although obfuscation is commonly used to protect intellectual property or evade software scanners, the robustness of code watermarking techniques against such transformations remains largely unexplored.
  In this work, we formally model the code obfuscation and prove the impossibility of N-gram-based watermarking's robustness with only one intuitive and experimentally verified assumption, distribution consistency, satisfied. Given the original false positive rate of the watermarking detection, the ratio that the detector failed on the watermarked code after obfuscation will increase to 1 - fpr.
  The experiments have been performed on three SOTA watermarking schemes, two LLMs, two programming languages, four code benchmarks, and four obfuscators. Among them, all watermarking detectors show coin-flipping detection abilities on obfuscated codes (AUROC tightly surrounds 0.5). Among all models, watermarking schemes, and datasets, both programming languages own obfuscators that can achieve attack effects with no detection AUROC higher than 0.6 after the attack. Based on the theoretical and practical observations, we also proposed a potential path of robust code watermarking.

http://arxiv.org/abs/2507.05441
Adversarial Machine Learning Attacks on Financial Reporting via Maximum Violated Multi-Objective Attack. (2%)
Edward Raff; Karen Kukla; Michel Benaroch; Joseph Comprix
Bad actors, primarily distressed firms, have the incentive and desire to manipulate their financial reports to hide their distress and derive personal gains. As attackers, these firms are motivated by potentially millions of dollars and the availability of many publicly disclosed and used financial modeling frameworks. Existing attack methods do not work on this data due to anti-correlated objectives that must both be satisfied for the attacker to succeed. We introduce Maximum Violated Multi-Objective (MVMO) attacks that adapt the attacker's search direction to find $20\times$ more satisfying attacks compared to standard attacks. The result is that in $\approx50\%$ of cases, a company could inflate their earnings by 100-200%, while simultaneously reducing their fraud scores by 15%. By working with lawyers and professional accountants, we ensure our threat model is realistic to how such frauds are performed in practice.

http://arxiv.org/abs/2507.04903
BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning. (2%)
Thinh Dao; Dung Thuy Nguyen; Khoa D Doan; Kok-Seng Wong
Federated Learning (FL) systems are vulnerable to backdoor attacks, where adversaries train their local models on poisoned data and submit poisoned model updates to compromise the global model. Despite numerous proposed attacks and defenses, divergent experimental settings, implementation errors, and unrealistic assumptions hinder fair comparisons and valid conclusions about their effectiveness in real-world scenarios. To address this, we introduce BackFed - a comprehensive benchmark suite designed to standardize, streamline, and reliably evaluate backdoor attacks and defenses in FL, with a focus on practical constraints. Our benchmark offers key advantages through its multi-processing implementation that significantly accelerates experimentation and the modular design that enables seamless integration of new methods via well-defined APIs. With a standardized evaluation pipeline, we envision BackFed as a plug-and-play environment for researchers to comprehensively and reliably evaluate new attacks and defenses. Using BackFed, we conduct large-scale studies of representative backdoor attacks and defenses across both Computer Vision and Natural Language Processing tasks with diverse model architectures and experimental settings. Our experiments critically assess the performance of proposed attacks and defenses, revealing unknown limitations and modes of failures under practical conditions. These empirical insights provide valuable guidance for the development of new methods and for enhancing the security of FL systems. Our framework is openly available at https://github.com/thinh-dao/BackFed.

http://arxiv.org/abs/2507.05248
Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models. (1%)
Ziqi Miao; Lijun Li; Yuan Xiong; Zhenhua Liu; Pengyu Zhu; Jing Shao
Contextual priming, where earlier stimuli covertly bias later judgments, offers an unexplored attack surface for large language models (LLMs). We uncover a contextual priming vulnerability in which the previous response in the dialogue can steer its subsequent behavior toward policy-violating content. Building on this insight, we propose Response Attack, which uses an auxiliary LLM to generate a mildly harmful response to a paraphrased version of the original malicious query. They are then formatted into the dialogue and followed by a succinct trigger prompt, thereby priming the target model to generate harmful content. Across eight open-source and proprietary LLMs, RA consistently outperforms seven state-of-the-art jailbreak techniques, achieving higher attack success rates. To mitigate this threat, we construct and release a context-aware safety fine-tuning dataset, which significantly reduces the attack success rate while preserving model capabilities. The code and data are available at https://github.com/Dtc7w3PQ/Response-Attack.

http://arxiv.org/abs/2507.04446
Tail-aware Adversarial Attacks: A Distributional Approach to Efficient LLM Jailbreaking. (81%)
Tim Beyer; Yan Scholten; Stephan Günnemann; Leo Schwinn
To guarantee safe and robust deployment of large language models (LLMs) at scale, it is critical to accurately assess their adversarial robustness. Existing adversarial attacks typically target harmful responses in single-point, greedy generations, overlooking the inherently stochastic nature of LLMs. In this paper, we propose a novel framework for adversarial robustness evaluation that explicitly models the entire output distribution, including tail-risks, providing better estimates for model robustness at scale. By casting the attack process as a resource allocation problem between optimization and sampling, we determine compute-optimal tradeoffs and show that integrating sampling into existing attacks boosts ASR by up to 48% and improves efficiency by up to two orders of magnitude. Our framework also enables us to analyze how different attack algorithms affect output harm distributions. Surprisingly, we find that most optimization strategies have little effect on output harmfulness. Finally, we introduce a data-free proof-of-concept objective based on entropy-maximization to demonstrate how our tail-aware perspective enables new optimization targets. Overall, our findings highlight the importance of tail-aware attacks and evaluation protocols to accurately assess and strengthen LLM safety.

http://arxiv.org/abs/2507.04365
Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs. (62%)
Xiaomeng Hu; Pin-Yu Chen; Tsung-Yi Ho
As large language models (LLMs) become more integral to society and technology, ensuring their safety becomes essential. Jailbreak attacks exploit vulnerabilities to bypass safety guardrails, posing a significant threat. However, the mechanisms enabling these attacks are not well understood. In this paper, we reveal a universal phenomenon that occurs during jailbreak attacks: Attention Slipping. During this phenomenon, the model gradually reduces the attention it allocates to unsafe requests in a user query during the attack process, ultimately causing a jailbreak. We show Attention Slipping is consistent across various jailbreak methods, including gradient-based token replacement, prompt-level template refinement, and in-context learning. Additionally, we evaluate two defenses based on query perturbation, Token Highlighter and SmoothLLM, and find they indirectly mitigate Attention Slipping, with their effectiveness positively correlated with the degree of mitigation achieved. Inspired by this finding, we propose Attention Sharpening, a new defense that directly counters Attention Slipping by sharpening the attention score distribution using temperature scaling. Experiments on four leading LLMs (Gemma2-9B-It, Llama3.1-8B-It, Qwen2.5-7B-It, Mistral-7B-It v0.2) show that our method effectively resists various jailbreak attacks while maintaining performance on benign tasks on AlpacaEval. Importantly, Attention Sharpening introduces no additional computational or memory overhead, making it an efficient and practical solution for real-world deployment.

http://arxiv.org/abs/2507.04528
Towards integration of Privacy Enhancing Technologies in Explainable Artificial Intelligence. (31%)
Sonal Allana; Rozita Dara; Xiaodong Lin; Pulei Xiong
Explainable Artificial Intelligence (XAI) is a crucial pathway in mitigating the risk of non-transparency in the decision-making process of black-box Artificial Intelligence (AI) systems. However, despite the benefits, XAI methods are found to leak the privacy of individuals whose data is used in training or querying the models. Researchers have demonstrated privacy attacks that exploit explanations to infer sensitive personal information of individuals. Currently there is a lack of defenses against known privacy attacks targeting explanations when vulnerable XAI are used in production and machine learning as a service system. To address this gap, in this article, we explore Privacy Enhancing Technologies (PETs) as a defense mechanism against attribute inference on explanations provided by feature-based XAI methods. We empirically evaluate 3 types of PETs, namely synthetic training data, differentially private training and noise addition, on two categories of feature-based XAI. Our evaluation determines different responses from the mitigation methods and side-effects of PETs on other system properties such as utility and performance. In the best case, PETs integration in explanations reduced the risk of the attack by 49.47%, while maintaining model utility and explanation quality. Through our evaluation, we identify strategies for using PETs in XAI for maximizing benefits and minimizing the success of this privacy attack on sensitive personal information.

http://arxiv.org/abs/2507.04227
Hijacking JARVIS: Benchmarking Mobile GUI Agents against Unprivileged Third Parties. (2%)
Guohong Liu; Jialei Ye; Jiacheng Liu; Yuanchun Li; Wei Liu; Pengzhi Gao; Jian Luan; Yunxin Liu
Mobile GUI agents are designed to autonomously execute diverse device-control tasks by interpreting and interacting with mobile screens. Despite notable advancements, their resilience in real-world scenarios where screen content may be partially manipulated by untrustworthy third parties remains largely unexplored. Owing to their black-box and autonomous nature, these agents are vulnerable to manipulations that could compromise user devices. In this work, we present the first systematic investigation into the vulnerabilities of mobile GUI agents. We introduce a scalable attack simulation framework AgentHazard, which enables flexible and targeted modifications of screen content within existing applications. Leveraging this framework, we develop a comprehensive benchmark suite comprising both a dynamic task execution environment and a static dataset of vision-language-action tuples, totaling over 3,000 attack scenarios. The dynamic environment encompasses 58 reproducible tasks in an emulator with various types of hazardous UI content, while the static dataset is constructed from 210 screenshots collected from 14 popular commercial apps. Importantly, our content modifications are designed to be feasible for unprivileged third parties. We evaluate 7 widely-used mobile GUI agents and 5 common backbone models using our benchmark. Our findings reveal that all examined agents are significantly influenced by misleading third-party content (with an average misleading rate of 28.8% in human-crafted attack scenarios) and that their vulnerabilities are closely linked to the employed perception modalities and backbone LLMs. Furthermore, we assess training-based mitigation strategies, highlighting both the challenges and opportunities for enhancing the robustness of mobile GUI agents. Our code and data will be released at https://agenthazard.github.io.

http://arxiv.org/abs/2205.13532
Selective Prediction via Training Dynamics. (1%)
Stephan Rabanser; Anvith Thudi; Kimia Hamidieh; Adam Dziedzic; Israfil Bahceci; Akram Bin Sediq; Hamza Sokun; Nicolas Papernot
Selective Prediction is the task of rejecting inputs a model would predict incorrectly on. This involves a trade-off between input space coverage (how many data points are accepted) and model utility (how good is the performance on accepted data points). Current methods for selective prediction typically impose constraints on either the model architecture or the optimization objective; this inhibits their usage in practice and introduces unknown interactions with pre-existing loss functions. In contrast to prior work, we show that state-of-the-art selective prediction performance can be attained solely from studying the (discretized) training dynamics of a model. We propose a general framework that, given a test input, monitors metrics capturing the instability of predictions from intermediate models (i.e., checkpoints) obtained during training w.r.t. the final model's prediction. In particular, we reject data points exhibiting too much disagreement with the final prediction at late stages in training. The proposed rejection mechanism is domain-agnostic (i.e., it works for both discrete and real-valued prediction) and can be flexibly combined with existing selective prediction approaches as it does not require any train-time modifications. Our experimental evaluation on image classification, regression, and time series problems shows that our method beats past state-of-the-art accuracy/utility trade-offs on typical selective prediction benchmarks.

http://arxiv.org/abs/2507.06252
False Alarms, Real Damage: Adversarial Attacks Using LLM-based Models on Text-based Cyber Threat Intelligence Systems. (88%)
Samaneh Shafee; Alysson Bessani; Pedro M. Ferreira
Cyber Threat Intelligence (CTI) has emerged as a vital complementary approach that operates in the early phases of the cyber threat lifecycle. CTI involves collecting, processing, and analyzing threat data to provide a more accurate and rapid understanding of cyber threats. Due to the large volume of data, automation through Machine Learning (ML) and Natural Language Processing (NLP) models is essential for effective CTI extraction. These automated systems leverage Open Source Intelligence (OSINT) from sources like social networks, forums, and blogs to identify Indicators of Compromise (IoCs). Although prior research has focused on adversarial attacks on specific ML models, this study expands the scope by investigating vulnerabilities within various components of the entire CTI pipeline and their susceptibility to adversarial attacks. These vulnerabilities arise because they ingest textual inputs from various open sources, including real and potentially fake content. We analyse three types of attacks against CTI pipelines, including evasion, flooding, and poisoning, and assess their impact on the system's information selection capabilities. Specifically, on fake text generation, the work demonstrates how adversarial text generation techniques can create fake cybersecurity and cybersecurity-like text that misleads classifiers, degrades performance, and disrupts system functionality. The focus is primarily on the evasion attack, as it precedes and enables flooding and poisoning attacks within the CTI pipeline.

http://arxiv.org/abs/2507.04100
Hierarchical Testing with Rabbit Optimization for Industrial Cyber-Physical Systems. (78%)
Jinwei Hu; Zezhi Tang; Xin Jin; Benyuan Zhang; Yi Dong; Xiaowei Huang
This paper presents HERO (Hierarchical Testing with Rabbit Optimization), a novel black-box adversarial testing framework for evaluating the robustness of deep learning-based Prognostics and Health Management systems in Industrial Cyber-Physical Systems. Leveraging Artificial Rabbit Optimization, HERO generates physically constrained adversarial examples that align with real-world data distributions via global and local perspective. Its generalizability ensures applicability across diverse ICPS scenarios. This study specifically focuses on the Proton Exchange Membrane Fuel Cell system, chosen for its highly dynamic operational conditions, complex degradation mechanisms, and increasing integration into ICPS as a sustainable and efficient energy solution. Experimental results highlight HERO's ability to uncover vulnerabilities in even state-of-the-art PHM models, underscoring the critical need for enhanced robustness in real-world applications. By addressing these challenges, HERO demonstrates its potential to advance more resilient PHM systems across a wide range of ICPS domains.

http://arxiv.org/abs/2507.04105
Enhancing Robustness of LLM-Driven Multi-Agent Systems through Randomized Smoothing. (74%)
Jinwei Hu; Yi Dong; Zhengtao Ding; Xiaowei Huang
This paper presents a defense framework for enhancing the safety of large language model (LLM) empowered multi-agent systems (MAS) in safety-critical domains such as aerospace. We apply randomized smoothing, a statistical robustness certification technique, to the MAS consensus context, enabling probabilistic guarantees on agent decisions under adversarial influence. Unlike traditional verification methods, our approach operates in black-box settings and employs a two-stage adaptive sampling mechanism to balance robustness and computational efficiency. Simulation results demonstrate that our method effectively prevents the propagation of adversarial behaviors and hallucinations while maintaining consensus performance. This work provides a practical and scalable path toward safe deployment of LLM-based MAS in real-world, high-stakes environments.

http://arxiv.org/abs/2507.07056
LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing. (41%)
Jiahao Chen; junhao li; Yiming Wang; Zhe Ma; Yi Jiang; Chunyi Zhou; Qingming Li; Tianyu Du; Shouling Ji
The proliferation of Low-Rank Adaptation (LoRA) models has democratized personalized text-to-image generation, enabling users to share lightweight models (e.g., personal portraits) on platforms like Civitai and Liblib. However, this "share-and-play" ecosystem introduces critical risks: benign LoRAs can be weaponized by adversaries to generate harmful content (e.g., political, defamatory imagery), undermining creator rights and platform safety. Existing defenses like concept-erasure methods focus on full diffusion models (DMs), neglecting LoRA's unique role as a modular adapter and its vulnerability to adversarial prompt engineering. To bridge this gap, we propose LoRAShield, the first data-free editing framework for securing LoRA models against misuse. Our platform-driven approach dynamically edits and realigns LoRA's weight subspace via adversarial optimization and semantic augmentation. Experimental results demonstrate that LoRAShield achieves remarkable effectiveness, efficiency, and robustness in blocking malicious generations without sacrificing the functionality of the benign task. By shifting the defense to platforms, LoRAShield enables secure, scalable sharing of personalized models, a critical step toward trustworthy generative ecosystems.

http://arxiv.org/abs/2507.04119
When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need. (16%)
Ziming Hong; Runnan Chen; Zengmao Wang; Bo Han; Bo Du; Tongliang Liu
Data-free knowledge distillation (DFKD) transfers knowledge from a teacher to a student without access the real in-distribution (ID) data. Its common solution is to use a generator to synthesize fake data and use them as a substitute for real ID data. However, existing works typically assume teachers are trustworthy, leaving the robustness and security of DFKD from untrusted teachers largely unexplored. In this work, we conduct the first investigation into distilling non-transferable learning (NTL) teachers using DFKD, where the transferability from an ID domain to an out-of-distribution (OOD) domain is prohibited. We find that NTL teachers fool DFKD through divert the generator's attention from the useful ID knowledge to the misleading OOD knowledge. This hinders ID knowledge transfer but prioritizes OOD knowledge transfer. To mitigate this issue, we propose Adversarial Trap Escaping (ATEsc) to benefit DFKD by identifying and filtering out OOD-like synthetic samples. Specifically, inspired by the evidence that NTL teachers show stronger adversarial robustness on OOD samples than ID samples, we split synthetic samples into two groups according to their robustness. The fragile group is treated as ID-like data and used for normal knowledge distillation, while the robust group is seen as OOD-like data and utilized for forgetting OOD knowledge. Extensive experiments demonstrate the effectiveness of ATEsc for improving DFKD against NTL teachers. Code is released at https://github.com/tmllab/2025_ICML_ATEsc.

http://arxiv.org/abs/2507.05288
A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models. (11%)
Shuliang Liu; Hongyi Liu; Aiwei Liu; Bingchen Duan; Qi Zheng; Yibo Yan; He Geng; Peijie Jiang; Jia Liu; Xuming Hu
The widespread deployment of large language models (LLMs) across critical domains has amplified the societal risks posed by algorithmically generated misinformation. Unlike traditional false content, LLM-generated misinformation can be self-reinforcing, highly plausible, and capable of rapid propagation across multiple languages, which traditional detection methods fail to mitigate effectively. This paper introduces a proactive defense paradigm, shifting from passive post hoc detection to anticipatory mitigation strategies. We propose a Three Pillars framework: (1) Knowledge Credibility, fortifying the integrity of training and deployed data; (2) Inference Reliability, embedding self-corrective mechanisms during reasoning; and (3) Input Robustness, enhancing the resilience of model interfaces against adversarial attacks. Through a comprehensive survey of existing techniques and a comparative meta-analysis, we demonstrate that proactive defense strategies offer up to 63\% improvement over conventional methods in misinformation prevention, despite non-trivial computational overhead and generalization challenges. We argue that future research should focus on co-designing robust knowledge foundations, reasoning certification, and attack-resistant interfaces to ensure LLMs can effectively counter misinformation across varied domains.

http://arxiv.org/abs/2507.03427
Rectifying Adversarial Sample with Low Entropy Prior for Test-Time Defense. (98%)
Lina Ma; Xiaowei Fu; Fuxiang Huang; Xinbo Gao; Lei Zhang
Existing defense methods fail to defend against unknown attacks and thus raise generalization issue of adversarial robustness. To remedy this problem, we attempt to delve into some underlying common characteristics among various attacks for generality. In this work, we reveal the commonly overlooked low entropy prior (LE) implied in various adversarial samples, and shed light on the universal robustness against unseen attacks in inference phase. LE prior is elaborated as two properties across various attacks as shown in Fig. 1 and Fig. 2: 1) low entropy misclassification for adversarial samples and 2) lower entropy prediction for higher attack intensity. This phenomenon stands in stark contrast to the naturally distributed samples. The LE prior can instruct existing test-time defense methods, thus we propose a two-stage REAL approach: Rectify Adversarial sample based on LE prior for test-time adversarial rectification. Specifically, to align adversarial samples more closely with clean samples, we propose to first rectify adversarial samples misclassified with low entropy by reverse maximizing prediction entropy, thereby eliminating their adversarial nature. To ensure the rectified samples can be correctly classified with low entropy, we carry out secondary rectification by forward minimizing prediction entropy, thus creating a Max-Min entropy optimization scheme. Further, based on the second property, we propose an attack-aware weighting mechanism to adaptively adjust the strengths of Max-Min entropy objectives. Experiments on several datasets show that REAL can greatly improve the performance of existing sample rectification models.

http://arxiv.org/abs/2507.03450
Evaluating the Evaluators: Trust in Adversarial Robustness Tests. (96%)
Antonio Emanuele Cinà; Maura Pintor; Luca Demetrio; Ambra Demontis; Battista Biggio; Fabio Roli
Despite significant progress in designing powerful adversarial evasion attacks for robustness verification, the evaluation of these methods often remains inconsistent and unreliable. Many assessments rely on mismatched models, unverified implementations, and uneven computational budgets, which can lead to biased results and a false sense of security. Consequently, robustness claims built on such flawed testing protocols may be misleading and give a false sense of security. As a concrete step toward improving evaluation reliability, we present AttackBench, a benchmark framework developed to assess the effectiveness of gradient-based attacks under standardized and reproducible conditions. AttackBench serves as an evaluation tool that ranks existing attack implementations based on a novel optimality metric, which enables researchers and practitioners to identify the most reliable and effective attack for use in subsequent robustness evaluations. The framework enforces consistent testing conditions and enables continuous updates, making it a reliable foundation for robustness verification.

http://arxiv.org/abs/2507.03646
When There Is No Decoder: Removing Watermarks from Stable Diffusion Models in a No-box Setting. (86%)
Xiaodong Wu; Tianyi Tang; Xiangman Li; Jianbing Ni; Yong Yu
Watermarking has emerged as a promising solution to counter harmful or deceptive AI-generated content by embedding hidden identifiers that trace content origins. However, the robustness of current watermarking techniques is still largely unexplored, raising critical questions about their effectiveness against adversarial attacks. To address this gap, we examine the robustness of model-specific watermarking, where watermark embedding is integrated with text-to-image generation in models like latent diffusion models. We introduce three attack strategies: edge prediction-based, box blurring, and fine-tuning-based attacks in a no-box setting, where an attacker does not require access to the ground-truth watermark decoder. Our findings reveal that while model-specific watermarking is resilient against basic evasion attempts, such as edge prediction, it is notably vulnerable to blurring and fine-tuning-based attacks. Our best-performing attack achieves a reduction in watermark detection accuracy to approximately 47.92\%. Additionally, we perform an ablation study on factors like message length, kernel size and decoder depth, identifying critical parameters influencing the fine-tuning attack's success. Finally, we assess several advanced watermarking defenses, finding that even the most robust methods, such as multi-label smoothing, result in watermark extraction accuracy that falls below an acceptable level when subjected to our no-box attacks.

http://arxiv.org/abs/2507.03372
Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization. (9%)
Buqing Nie; Yangqing Fu; Jingtian Ji; Yue Gao
Reinforcement Learning (RL) has achieved remarkable success in sequential decision tasks. However, recent studies have revealed the vulnerability of RL policies to different perturbations, raising concerns about their effectiveness and safety in real-world applications. In this work, we focus on the robustness of RL policies against action perturbations and introduce a novel framework called Optimal Adversary-aware Policy Iteration (OA-PI). Our framework enhances action robustness under various perturbations by evaluating and improving policy performance against the corresponding optimal adversaries. Besides, our approach can be integrated into mainstream DRL algorithms such as Twin Delayed DDPG (TD3) and Proximal Policy Optimization (PPO), improving action robustness effectively while maintaining nominal performance and sample efficiency. Experimental results across various environments demonstrate that our method enhances robustness of DRL policies against different action adversaries effectively.

http://arxiv.org/abs/2507.03236
On Jailbreaking Quantized Language Models Through Fault Injection Attacks. (8%)
Noureldin Zahran; Ahmad Tahmasivand; Ihsen Alouani; Khaled Khasawneh; Mohammed E. Fouda
The safety alignment of Language Models (LMs) is a critical concern, yet their integrity can be challenged by direct parameter manipulation attacks, such as those potentially induced by fault injection. As LMs are increasingly deployed using low-precision quantization for efficiency, this paper investigates the efficacy of such attacks for jailbreaking aligned LMs across different quantization schemes. We propose gradient-guided attacks, including a tailored progressive bit-level search algorithm introduced herein and a comparative word-level (single weight update) attack. Our evaluation on Llama-3.2-3B, Phi-4-mini, and Llama-3-8B across FP16 (baseline), and weight-only quantization (FP8, INT8, INT4) reveals that quantization significantly influences attack success. While attacks readily achieve high success (>80\% Attack Success Rate, ASR) on FP16 models, within an attack budget of 25 perturbations, FP8 and INT8 models exhibit ASRs below 20\% and 50\%, respectively. Increasing the perturbation budget up to 150 bit-flips, FP8 models maintained ASR below 65\%, demonstrating some resilience compared to INT8 and INT4 models that have high ASR. In addition, analysis of perturbation locations revealed differing architectural targets across quantization schemes, with (FP16, INT4) and (INT8, FP8) showing similar characteristics. Besides, jailbreaks induced in FP16 models were highly transferable to subsequent FP8/INT8 quantization (<5\% ASR difference), though INT4 significantly reduced transferred ASR (avg. 35\% drop). These findings highlight that while common quantization schemes, particularly FP8, increase the difficulty of direct parameter manipulation jailbreaks, vulnerabilities can still persist, especially through post-attack quantization.

http://arxiv.org/abs/2501.08276
Exploring Robustness of LLMs to Paraphrasing Based on Sociodemographic Factors. (5%)
Pulkit Arora; Akbar Karimi; Lucie Flek
Despite their linguistic prowess, LLMs have been shown to be vulnerable to small input perturbations. While robustness to local adversarial changes has been studied, robustness to global modifications such as different linguistic styles remains underexplored. Therefore, we take a broader approach to explore a wider range of variations across sociodemographic dimensions. We extend the SocialIQA dataset to create diverse paraphrased sets conditioned on sociodemographic factors (age and gender). The assessment aims to provide a deeper understanding of LLMs in (a) their capability of generating demographic paraphrases with engineered prompts and (b) their capabilities in interpreting real-world, complex language scenarios. We also perform a reliability analysis of the generated paraphrases looking into linguistic diversity and perplexity as well as manual evaluation. We find that demographic-based paraphrasing significantly impacts the performance of language models, indicating that the subtleties of linguistic variation remain a significant challenge. We will make the code and dataset available for future research.

http://arxiv.org/abs/2506.23661
Robustness of Misinformation Classification Systems to Adversarial Examples Through BeamAttack. (99%)
Arnisa Fazla; Lucas Krauter; David Guzman Piedrahita; Andrianos Michail
We extend BeamAttack, an adversarial attack algorithm designed to evaluate the robustness of text classification systems through word-level modifications guided by beam search. Our extensions include support for word deletions and the option to skip substitutions, enabling the discovery of minimal modifications that alter model predictions. We also integrate LIME to better prioritize word replacements. Evaluated across multiple datasets and victim models (BiLSTM, BERT, and adversarially trained RoBERTa) within the BODEGA framework, our approach achieves over a 99\% attack success rate while preserving the semantic and lexical similarity of the original texts. Through both quantitative and qualitative analysis, we highlight BeamAttack's effectiveness and its limitations. Our implementation is available at https://github.com/LucK1Y/BeamAttack

http://arxiv.org/abs/2507.02606
De-AntiFake: Rethinking the Protective Perturbations Against Voice Cloning Attacks. (97%)
Wei Fan; Kejiang Chen; Chang Liu; Weiming Zhang; Nenghai Yu
The rapid advancement of speech generation models has heightened privacy and security concerns related to voice cloning (VC). Recent studies have investigated disrupting unauthorized voice cloning by introducing adversarial perturbations. However, determined attackers can mitigate these protective perturbations and successfully execute VC. In this study, we conduct the first systematic evaluation of these protective perturbations against VC under realistic threat models that include perturbation purification. Our findings reveal that while existing purification methods can neutralize a considerable portion of the protective perturbations, they still lead to distortions in the feature space of VC models, which degrades the performance of VC. From this perspective, we propose a novel two-stage purification method: (1) Purify the perturbed speech; (2) Refine it using phoneme guidance to align it with the clean speech distribution. Experimental results demonstrate that our method outperforms state-of-the-art purification methods in disrupting VC defenses. Our study reveals the limitations of adversarial perturbation-based VC defenses and underscores the urgent need for more robust solutions to mitigate the security and privacy risks posed by VC. The code and audio samples are available at https://de-antifake.github.io.

http://arxiv.org/abs/2412.05883
On the Adversarial Robustness of Graph Neural Networks with Graph Reduction. (78%)
Kerui Wu; Ka-Ho Chow; Wenqi Wei; Lei Yu
As Graph Neural Networks (GNNs) become increasingly popular for learning from large-scale graph data across various domains, their susceptibility to adversarial attacks when using graph reduction techniques for scalability remains underexplored. In this paper, we present an extensive empirical study to investigate the impact of graph reduction techniques, specifically graph coarsening and sparsification, on the robustness of GNNs against adversarial attacks. Through extensive experiments involving multiple datasets and GNN architectures, we examine the effects of four sparsification and six coarsening methods on the poisoning attacks. Our results indicate that, while graph sparsification can mitigate the effectiveness of certain poisoning attacks, such as Mettack, it has limited impact on others, like PGD. Conversely, graph coarsening tends to amplify the adversarial impact, significantly reducing classification accuracy as the reduction ratio decreases. Additionally, we provide a novel analysis of the causes driving these effects and examine how defensive GNN models perform under graph reduction, offering practical insights for designing robust GNNs within graph acceleration systems.

http://arxiv.org/abs/2507.03031
On the Mathematical Impossibility of Safe Universal Approximators. (56%)
Jasper Yao
We establish fundamental mathematical limits on universal approximation theorem (UAT) system alignment by proving that catastrophic failures are an inescapable feature of any useful computational system. Our central thesis is that for any universal approximator, the expressive power required for useful computation is inextricably linked to a dense set of instabilities that make perfect, reliable control a mathematical impossibility. We prove this through a three-level argument that leaves no escape routes for any class of universal approximator architecture. i) Combinatorial Necessity: For the vast majority of practical universal approximators (e.g., those using ReLU activations), we prove that the density of catastrophic failure points is directly proportional to the network's expressive power. ii) Topological Necessity: For any theoretical universal approximator, we use singularity theory to prove that the ability to approximate generic functions requires the ability to implement the dense, catastrophic singularities that characterize them. iii) Empirical Necessity: We prove that the universal existence of adversarial examples is empirical evidence that real-world tasks are themselves catastrophic, forcing any successful model to learn and replicate these instabilities. These results, combined with a quantitative "Impossibility Sandwich" showing that the minimum complexity for usefulness exceeds the maximum complexity for safety, demonstrate that perfect alignment is not an engineering challenge but a mathematical impossibility. This foundational result reframes UAT safety from a problem of "how to achieve perfect control" to one of "how to operate safely in the presence of irreducible uncontrollability," with profound implications for the future of UAT development and governance.

http://arxiv.org/abs/2507.02735
Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks. (26%)
Sizhe Chen; Arman Zharmagambetov; David Wagner; Chuan Guo
Prompt injection attacks pose a significant security threat to LLM-integrated applications. Model-level defenses have shown strong effectiveness, but are currently deployed into commercial-grade models in a closed-source manner. We believe open-source models are needed by the AI security community, where co-development of attacks and defenses through open research drives scientific progress in mitigation against prompt injection attacks. To this end, we develop Meta SecAlign, the first open-source and open-weight LLM with built-in model-level defense that achieves commercial-grade model performance. We provide complete details of our training recipe, which utilizes an improved version of the SOTA SecAlign defense. Evaluations on 9 utility benchmarks and 7 security benchmarks show that Meta SecAlign, despite being trained on a generic instruction-tuning dataset, confers security in unseen downstream tasks, including tool-calling and agentic web navigation, in addition general instruction-following. Our best model -- Meta-SecAlign-70B -- achieves state-of-the-art robustness against prompt injection attacks and comparable utility to closed-source commercial LLM with model-level defense.

http://arxiv.org/abs/2502.11853
StructTransform: A Scalable Attack Surface for Safety-Aligned Large Language Models. (16%)
Shehel Yoosuf; Temoor Ali; Ahmed Lekssays; Mashael AlSabah; Issa Khalil
In this work, we present a series of structure transformation attacks on LLM alignment, where we encode natural language intent using diverse syntax spaces, ranging from simple structure formats and basic query languages (e.g., SQL) to new novel spaces and syntaxes created entirely by LLMs. Our extensive evaluation shows that our simplest attacks can achieve close to a 90% success rate, even on strict LLMs (such as Claude 3.5 Sonnet) using SOTA alignment mechanisms. We improve the attack performance further by using an adaptive scheme that combines structure transformations along with existing content transformations, resulting in over 96% ASR with 0% refusals.
  To generalize our attacks, we explore numerous structure formats, including syntaxes purely generated by LLMs. Our results indicate that such novel syntaxes are easy to generate and result in a high ASR, suggesting that defending against our attacks is not a straightforward process. Finally, we develop a benchmark and evaluate existing safety-alignment defenses against it, showing that most of them fail with 100% ASR. Our results show that existing safety alignment mostly relies on token-level patterns without recognizing harmful concepts, highlighting and motivating the need for serious research efforts in this direction. As a case study, we demonstrate how attackers can use our attack to easily generate a sample malware and a corpus of fraudulent SMS messages, which perform well in bypassing detection.

http://arxiv.org/abs/2507.02727
Quantifying Classifier Utility under Local Differential Privacy. (11%)
Ye Zheng; Yidan Hu
Local differential privacy (LDP) provides a rigorous and quantifiable privacy guarantee for personal data by introducing perturbation at the data source. However, quantifying the impact of these perturbations on classifier utility remains a theoretical challenge, particularly for complex or black-box classifiers.
  This paper presents a framework for theoretically quantifying classifier utility under LDP mechanisms. The key insight is that LDP perturbation is concentrated around the original data with a specific probability, transforming utility analysis of the classifier into its robustness analysis in this concentrated region. Our framework connects the concentration analysis of LDP mechanisms with the robustness analysis of classifiers. It treats LDP mechanisms as general distributional functions and classifiers as black-box functions, thus applicable to any LDP mechanism and classifier. A direct application of our utility quantification is guiding the selection of LDP mechanisms and privacy parameters for a given classifier. Notably, our analysis shows that a piecewise-based mechanism leads to better utility compared to alternatives in common scenarios.
  Using this framework alongside two novel refinement techniques, we conduct case studies on utility quantification for typical mechanism-classifier combinations. The results demonstrate that our theoretical utility quantification aligns closely with empirical observations, particularly when classifiers operate in lower-dimensional input spaces.

http://arxiv.org/abs/2507.03168
Adopting a human developmental visual diet yields robust, shape-based AI vision. (10%)
Zejin Lu; Sushrut Thorat; Radoslaw M Cichy; Tim C Kietzmann
Despite years of research and the dramatic scaling of artificial intelligence (AI) systems, a striking misalignment between artificial and human vision persists. Contrary to humans, AI heavily relies on texture-features rather than shape information, lacks robustness to image distortions, remains highly vulnerable to adversarial attacks, and struggles to recognise simple abstract shapes within complex backgrounds. To close this gap, we here introduce a solution that arises from a previously underexplored direction: rather than scaling up, we take inspiration from how human vision develops from early infancy into adulthood. We quantified the visual maturation by synthesising decades of psychophysical and neurophysiological research into a novel developmental visual diet (DVD) for AI vision. We show that guiding AI systems through this human-inspired curriculum produces models that closely align with human behaviour on every hallmark of robust vision tested yielding the strongest reported reliance on shape information to date, abstract shape recognition beyond the state of the art, higher robustness to image corruptions, and stronger resilience to adversarial attacks. By outperforming high parameter AI foundation models trained on orders of magnitude more data, we provide evidence that robust AI vision can be achieved by guiding the way how a model learns, not merely how much it learns, offering a resource-efficient route toward safer and more human-like artificial visual systems.

http://arxiv.org/abs/2507.02799
Is Reasoning All You Need? Probing Bias in the Age of Reasoning Language Models. (3%)
Riccardo Cantini; Nicola Gabriele; Alessio Orsino; Domenico Talia
Reasoning Language Models (RLMs) have gained traction for their ability to perform complex, multi-step reasoning tasks through mechanisms such as Chain-of-Thought (CoT) prompting or fine-tuned reasoning traces. While these capabilities promise improved reliability, their impact on robustness to social biases remains unclear. In this work, we leverage the CLEAR-Bias benchmark, originally designed for Large Language Models (LLMs), to investigate the adversarial robustness of RLMs to bias elicitation. We systematically evaluate state-of-the-art RLMs across diverse sociocultural dimensions, using an LLM-as-a-judge approach for automated safety scoring and leveraging jailbreak techniques to assess the strength of built-in safety mechanisms. Our evaluation addresses three key questions: (i) how the introduction of reasoning capabilities affects model fairness and robustness; (ii) whether models fine-tuned for reasoning exhibit greater safety than those relying on CoT prompting at inference time; and (iii) how the success rate of jailbreak attacks targeting bias elicitation varies with the reasoning mechanisms employed. Our findings reveal a nuanced relationship between reasoning capabilities and bias safety. Surprisingly, models with explicit reasoning, whether via CoT prompting or fine-tuned reasoning traces, are generally more vulnerable to bias elicitation than base models without such mechanisms, suggesting reasoning may unintentionally open new pathways for stereotype reinforcement. Reasoning-enabled models appear somewhat safer than those relying on CoT prompting, which are particularly prone to contextual reframing attacks through storytelling prompts, fictional personas, or reward-shaped instructions. These results challenge the assumption that reasoning inherently improves robustness and underscore the need for more bias-aware approaches to reasoning design.

http://arxiv.org/abs/2507.03167
Adversarial Manipulation of Reasoning Models using Internal Representations. (1%)
Kureha Yamaguchi; Benjamin Etheridge; Andy Arditi
Reasoning models generate chain-of-thought (CoT) tokens before their final output, but how this affects their vulnerability to jailbreak attacks remains unclear. While traditional language models make refusal decisions at the prompt-response boundary, we find evidence that DeepSeek-R1-Distill-Llama-8B makes these decisions within its CoT generation. We identify a linear direction in activation space during CoT token generation that predicts whether the model will refuse or comply -- termed the "caution" direction because it corresponds to cautious reasoning patterns in the generated text. Ablating this direction from model activations increases harmful compliance, effectively jailbreaking the model. We additionally show that intervening only on CoT token activations suffices to control final outputs, and that incorporating this direction into prompt-based attacks improves success rates. Our findings suggest that the chain-of-thought itself is a promising new target for adversarial manipulation in reasoning models.
  Code available at https://github.com/ky295/reasoning-manipulation

http://arxiv.org/abs/2507.02737
Early Signs of Steganographic Capabilities in Frontier LLMs. (1%)
Artur Zolkowski; Kei Nishimura-Gasparian; Robert McCarthy; Roland S. Zimmermann; David Lindner
Monitoring Large Language Model (LLM) outputs is crucial for mitigating risks from misuse and misalignment. However, LLMs could evade monitoring through steganography: Encoding hidden information within seemingly benign generations. In this paper, we evaluate the steganography capabilities in frontier LLMs to better understand the risk they pose. We focus on two types of steganography: passing encoded messages and performing encoded reasoning. We find that current models are unable to encode short messages in their outputs without a monitor noticing under standard affordances. They can succeed, however, if given additional affordances such as using an unmonitored scratchpad and coordinating on what encoding scheme to use. We additionally find early signs that models can perform basic encoded reasoning in a simple state-tracking problem. This includes some ability to reason with their own and pre-defined schemes, including encoding schemes such as Hexadecimal. Despite this, they can rarely hide reasoning subtly within a cover task to fool a monitor. Overall, our results indicate that current LLMs exhibit nascent steganographic capabilities. While these capabilities are likely insufficient to bypass well-designed monitors at present, this could change in the future.

http://arxiv.org/abs/2502.05673
The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions. (1%)
Ping Liu; Jiawei Du
Dataset distillation, which condenses large-scale datasets into compact synthetic representations, has emerged as a critical solution for training modern deep learning models efficiently. While prior surveys focus on developments before 2023, this work comprehensively reviews recent advances, emphasizing scalability to large-scale datasets such as ImageNet-1K and ImageNet-21K. We categorize progress into a few key methodologies: trajectory matching, gradient matching, distribution matching, scalable generative approaches, and decoupling optimization mechanisms. As a comprehensive examination of recent dataset distillation advances, this survey highlights breakthrough innovations: the SRe2L framework for efficient and effective condensation, soft label strategies that significantly enhance model accuracy, and lossless distillation techniques that maximize compression while maintaining performance. Beyond these methodological advancements, we address critical challenges, including robustness against adversarial and backdoor attacks, effective handling of non-IID data distributions. Additionally, we explore emerging applications in video and audio processing, multi-modal learning, medical imaging, and scientific computing, highlighting its domain versatility. By offering extensive performance comparisons and actionable research directions, this survey equips researchers and practitioners with practical insights to advance efficient and generalizable dataset distillation, paving the way for future innovations.

http://arxiv.org/abs/2507.01791
Boosting Adversarial Transferability Against Defenses via Multi-Scale Transformation. (99%)
Zihong Guo; Chen Wan; Yayin Zheng; Hailing Kuang; Xiaohai Lu
The transferability of adversarial examples poses a significant security challenge for deep neural networks, which can be attacked without knowing anything about them. In this paper, we propose a new Segmented Gaussian Pyramid (SGP) attack method to enhance the transferability, particularly against defense models. Unlike existing methods that generally focus on single-scale images, our approach employs Gaussian filtering and three types of downsampling to construct a series of multi-scale examples. Then, the gradients of the loss function with respect to each scale are computed, and their average is used to determine the adversarial perturbations. The proposed SGP can be considered an input transformation with high extensibility that is easily integrated into most existing adversarial attacks. Extensive experiments demonstrate that in contrast to the state-of-the-art methods, SGP significantly enhances attack success rates against black-box defense models, with average attack success rates increasing by 2.3% to 32.6%, based only on transferability.

http://arxiv.org/abs/2507.01383
DARTS: A Dual-View Attack Framework for Targeted Manipulation in Federated Sequential Recommendation. (68%)
Qitao Qin; Yucong Luo; Zhibo Chu
Federated recommendation (FedRec) preserves user privacy by enabling decentralized training of personalized models, but this architecture is inherently vulnerable to adversarial attacks. Significant research has been conducted on targeted attacks in FedRec systems, motivated by commercial and social influence considerations. However, much of this work has largely overlooked the differential robustness of recommendation models. Moreover, our empirical findings indicate that existing targeted attack methods achieve only limited effectiveness in Federated Sequential Recommendation(FSR) tasks. Driven by these observations, we focus on investigating targeted attacks in FSR and propose a novel dualview attack framework, named DV-FSR. This attack method uniquely combines a sampling-based explicit strategy with a contrastive learning-based implicit gradient strategy to orchestrate a coordinated attack. Additionally, we introduce a specific defense mechanism tailored for targeted attacks in FSR, aiming to evaluate the mitigation effects of the attack method we proposed. Extensive experiments validate the effectiveness of our proposed approach on representative sequential models. Our codes are publicly available.

http://arxiv.org/abs/2507.01694
Graph Representation-based Model Poisoning on Federated LLMs in CyberEdge Networks. (50%)
Hanlin Cai; Haofan Dong; Houtianfu Wang; Kai Li; Ozgur B. Akan
Federated large language models (FedLLMs) provide powerful generative capabilities in CyberEdge networks while protecting data privacy. However, FedLLMs remains highly vulnerable to model poisoning attacks. This article first reviews recent model poisoning techniques and existing defense mechanisms for FedLLMs, highlighting critical limitations, particularly under non-IID text distributions. In particular, current defenses primarily utilize distance-based outlier detection or norm constraints, operating under the assumption that adversarial updates significantly diverge from benign statistics. This assumption can fail when facing adaptive attackers targeting billionparameter LLMs. Next, this article investigates emerging Graph Representation-Based Model Poisoning (GRMP), a novel attack paradigm that leverages higher-order correlations among honest client gradients to synthesize malicious updates indistinguishable from legitimate model updates. GRMP can effectively evade advanced defenses, resulting in substantial accuracy loss and performance degradation. Moreover, this article outlines a research roadmap emphasizing the importance of graph-aware secure aggregation methods, FedLLMs-specific vulnerability metrics, and evaluation frameworks to strengthen the robustness of future federated language model deployments.

http://arxiv.org/abs/2507.01321
ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks. (13%)
Zhiyao Ren; Siyuan Liang; Aishan Liu; Dacheng Tao
In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces a critical vulnerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poisoning a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstrations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability is dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Our method encourages LLMs to select clean demonstrations during the ICL phase by leveraging confidence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive experiments across multiple LLMs and tasks demonstrate that our method achieves state-of-the-art defense effectiveness, significantly outperforming existing approaches (+26.02% on average). Furthermore, our method exhibits exceptional adaptability and defensive performance even for closed-source models (e.g., GPT-4).

http://arxiv.org/abs/2507.01513
SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism. (12%)
Beitao Chen; Xinyu Lyu; Lianli Gao; Jingkuan Song; Heng Tao Shen
By incorporating visual inputs, Multimodal Large Language Models (MLLMs) extend LLMs to support visual reasoning. However, this integration also introduces new vulnerabilities, making MLLMs susceptible to multimodal jailbreak attacks and hindering their safe deployment.Existing defense methods, including Image-to-Text Translation, Safe Prompting, and Multimodal Safety Tuning, attempt to address this by aligning multimodal inputs with LLMs' built-in safeguards.Yet, they fall short in uncovering root causes of multimodal vulnerabilities, particularly how harmful multimodal tokens trigger jailbreak in MLLMs? Consequently, they remain vulnerable to text-driven multimodal jailbreaks, often exhibiting overdefensive behaviors and imposing heavy training overhead.To bridge this gap, we present an comprehensive analysis of where, how and which harmful multimodal tokens bypass safeguards in MLLMs. Surprisingly, we find that less than 1% tokens in early-middle layers are responsible for inducing unsafe behaviors, highlighting the potential of precisely removing a small subset of harmful tokens, without requiring safety tuning, can still effectively improve safety against jailbreaks. Motivated by this, we propose Safe Prune-then-Restore (SafePTR), an training-free defense framework that selectively prunes harmful tokens at vulnerable layers while restoring benign features at subsequent layers.Without incurring additional computational overhead, SafePTR significantly enhances the safety of MLLMs while preserving efficiency. Extensive evaluations across three MLLMs and five benchmarks demonstrate SafePTR's state-of-the-art performance in mitigating jailbreak risks without compromising utility.

http://arxiv.org/abs/2406.15213
Backdooring Bias (B^2) into Stable Diffusion Models. (9%)
Ali Naseh; Jaechul Roh; Eugene Bagdasaryan; Amir Houmansadr
Recent advances in large text-conditional diffusion models have revolutionized image generation by enabling users to create realistic, high-quality images from textual prompts, significantly enhancing artistic creation and visual communication. However, these advancements also introduce an underexplored attack opportunity: the possibility of inducing biases by an adversary into the generated images for malicious intentions, e.g., to influence public opinion and spread propaganda. In this paper, we study an attack vector that allows an adversary to inject arbitrary bias into a target model. The attack leverages low-cost backdooring techniques using a targeted set of natural textual triggers embedded within a small number of malicious data samples produced with public generative models. An adversary could pick common sequences of words that can then be inadvertently activated by benign users during inference. We investigate the feasibility and challenges of such attacks, demonstrating how modern generative models have made this adversarial process both easier and more adaptable. On the other hand, we explore various aspects of the detectability of such attacks and demonstrate that the model's utility remains intact in the absence of the triggers. Our extensive experiments using over 200,000 generated images and against hundreds of fine-tuned models demonstrate the feasibility of the presented backdoor attack. We illustrate how these biases maintain strong text-image alignment, highlighting the challenges in detecting biased images without knowing that bias in advance. Our cost analysis confirms the low financial barrier ($10-$15) to executing such attacks, underscoring the need for robust defensive strategies against such vulnerabilities in diffusion models.

http://arxiv.org/abs/2507.01710
Towards Better Attribute Inference Vulnerability Measures. (1%)
Paul Francis; David Wagner
The purpose of anonymizing structured data is to protect the privacy of individuals in the data while retaining the statistical properties of the data. An important class of attack on anonymized data is attribute inference, where an attacker infers the value of an unknown attribute of a target individual given knowledge of one or more known attributes. A major limitation of recent attribute inference measures is that they do not take recall into account, only precision. It is often the case that attacks target only a fraction of individuals, for instance data outliers. Incorporating recall, however, substantially complicates the measure, because one must determine how to combine recall and precision in a composite measure for both the attack and baseline. This paper presents the design and implementation of an attribute inference measure that incorporates both precision and recall. Our design also improves on how the baseline attribute inference is computed. In experiments using a generic best row match attack on moderately-anonymized microdata, we show that in over 25\% of the attacks, our approach correctly labeled the attack to be at risk while the prior approach incorrectly labeled the attack to be safe.

http://arxiv.org/abs/2507.00577
BadViM: Backdoor Attack against Vision Mamba. (69%)
Yinghao Wu; Liyan Zhang
Vision State Space Models (SSMs), particularly architectures like Vision Mamba (ViM), have emerged as promising alternatives to Vision Transformers (ViTs). However, the security implications of this novel architecture, especially their vulnerability to backdoor attacks, remain critically underexplored. Backdoor attacks aim to embed hidden triggers into victim models, causing the model to misclassify inputs containing these triggers while maintaining normal behavior on clean inputs. This paper investigates the susceptibility of ViM to backdoor attacks by introducing BadViM, a novel backdoor attack framework specifically designed for Vision Mamba. The proposed BadViM leverages a Resonant Frequency Trigger (RFT) that exploits the frequency sensitivity patterns of the victim model to create stealthy, distributed triggers. To maximize attack efficacy, we propose a Hidden State Alignment loss that strategically manipulates the internal representations of model by aligning the hidden states of backdoor images with those of target classes. Extensive experimental results demonstrate that BadViM achieves superior attack success rates while maintaining clean data accuracy. Meanwhile, BadViM exhibits remarkable resilience against common defensive measures, including PatchDrop, PatchShuffle and JPEG compression, which typically neutralize normal backdoor attacks.

http://arxiv.org/abs/2507.00817
CAVALRY-V: A Large-Scale Generator Framework for Adversarial Attacks on Video MLLMs. (47%)
Jiaming Zhang; Rui Hu; Qing Guo; Wei Yang Bryan Lim
Video Multimodal Large Language Models (V-MLLMs) have shown impressive capabilities in temporal reasoning and cross-modal understanding, yet their vulnerability to adversarial attacks remains underexplored due to unique challenges: complex cross-modal reasoning mechanisms, temporal dependencies, and computational constraints. We present CAVALRY-V (Cross-modal Language-Vision Adversarial Yielding for Videos), a novel framework that directly targets the critical interface between visual perception and language generation in V-MLLMs. Our approach introduces two key innovations: (1) a dual-objective semantic-visual loss function that simultaneously disrupts the model's text generation logits and visual representations to undermine cross-modal integration, and (2) a computationally efficient two-stage generator framework that combines large-scale pre-training for cross-model transferability with specialized fine-tuning for spatiotemporal coherence. Empirical evaluation on comprehensive video understanding benchmarks demonstrates that CAVALRY-V significantly outperforms existing attack methods, achieving 22.8% average improvement over the best baseline attacks on both commercial systems (GPT-4.1, Gemini 2.0) and open-source models (QwenVL-2.5, InternVL-2.5, Llava-Video, Aria, MiniCPM-o-2.6). Our framework achieves flexibility through implicit temporal coherence modeling rather than explicit regularization, enabling significant performance improvements even on image understanding (34.4% average gain). This capability demonstrates CAVALRY-V's potential as a foundational approach for adversarial research across multimodal systems.

http://arxiv.org/abs/2507.00423
Find a Scapegoat: Poisoning Membership Inference Attack and Defense to Federated Learning. (41%)
Wenjin Mo; Zhiyuan Li; Minghong Fang; Mingwei Fang
Federated learning (FL) allows multiple clients to collaboratively train a global machine learning model with coordination from a central server, without needing to share their raw data. This approach is particularly appealing in the era of privacy regulations like the GDPR, leading many prominent companies to adopt it. However, FL's distributed nature makes it susceptible to poisoning attacks, where malicious clients, controlled by an attacker, send harmful data to compromise the model. Most existing poisoning attacks in FL aim to degrade the model's integrity, such as reducing its accuracy, with limited attention to privacy concerns from these attacks. In this study, we introduce FedPoisonMIA, a novel poisoning membership inference attack targeting FL. FedPoisonMIA involves malicious clients crafting local model updates to infer membership information. Additionally, we propose a robust defense mechanism to mitigate the impact of FedPoisonMIA attacks. Extensive experiments across various datasets demonstrate the attack's effectiveness, while our defense approach reduces its impact to a degree.

http://arxiv.org/abs/2507.00971
Reasoning as an Adaptive Defense for Safety. (31%)
Taeyoun Kim; Fahim Tajwar; Aditi Raghunathan; Aviral Kumar
Reasoning methods that adaptively allocate test-time compute have advanced LLM performance on easy to verify domains such as math and code. In this work, we study how to utilize this approach to train models that exhibit a degree of robustness to safety vulnerabilities, and show that doing so can provide benefits. We build a recipe called $\textit{TARS}$ (Training Adaptive Reasoners for Safety), a reinforcement learning (RL) approach that trains models to reason about safety using chain-of-thought traces and a reward signal that balances safety with task completion. To build TARS, we identify three critical design choices: (1) a "lightweight" warmstart SFT stage, (2) a mix of harmful, harmless, and ambiguous prompts to prevent shortcut behaviors such as too many refusals, and (3) a reward function to prevent degeneration of reasoning capabilities during training. Models trained with TARS exhibit adaptive behaviors by spending more compute on ambiguous queries, leading to better safety-refusal trade-offs. They also internally learn to better distinguish between safe and unsafe prompts and attain greater robustness to both white-box (e.g., GCG) and black-box attacks (e.g., PAIR). Overall, our work provides an effective, open recipe for training LLMs against jailbreaks and harmful requests by reasoning per prompt.

http://arxiv.org/abs/2507.00690
Cage-Based Deformation for Transferable and Undefendable Point Cloud Attack. (26%)
Keke Tang; Ziyong Du; Weilong Peng; Xiaofei Wang; Peican Zhu; Ligang Liu; Zhihong Tian
Adversarial attacks on point clouds often impose strict geometric constraints to preserve plausibility; however, such constraints inherently limit transferability and undefendability. While deformation offers an alternative, existing unstructured approaches may introduce unnatural distortions, making adversarial point clouds conspicuous and undermining their plausibility. In this paper, we propose CageAttack, a cage-based deformation framework that produces natural adversarial point clouds. It first constructs a cage around the target object, providing a structured basis for smooth, natural-looking deformation. Perturbations are then applied to the cage vertices, which seamlessly propagate to the point cloud, ensuring that the resulting deformations remain intrinsic to the object and preserve plausibility. Extensive experiments on seven 3D deep neural network classifiers across three datasets show that CageAttack achieves a superior balance among transferability, undefendability, and plausibility, outperforming state-of-the-art methods. Codes will be made public upon acceptance.

http://arxiv.org/abs/2507.00506
SCING:Towards More Efficient and Robust Person Re-Identification through Selective Cross-modal Prompt Tuning. (2%)
Yunfei Xie; Yuxuan Cheng; Juncheng Wu; Haoyu Zhang; Yuyin Zhou; Shoudong Han
Recent advancements in adapting vision-language pre-training models like CLIP for person re-identification (ReID) tasks often rely on complex adapter design or modality-specific tuning while neglecting cross-modal interaction, leading to high computational costs or suboptimal alignment. To address these limitations, we propose a simple yet effective framework named Selective Cross-modal Prompt Tuning (SCING) that enhances cross-modal alignment and robustness against real-world perturbations. Our method introduces two key innovations: Firstly, we proposed Selective Visual Prompt Fusion (SVIP), a lightweight module that dynamically injects discriminative visual features into text prompts via a cross-modal gating mechanism. Moreover, the proposed Perturbation-Driven Consistency Alignment (PDCA) is a dual-path training strategy that enforces invariant feature alignment under random image perturbations by regularizing consistency between original and augmented cross-modal embeddings. Extensive experiments are conducted on several popular benchmarks covering Market1501, DukeMTMC-ReID, Occluded-Duke, Occluded-REID, and P-DukeMTMC, which demonstrate the impressive performance of the proposed method. Notably, our framework eliminates heavy adapters while maintaining efficient inference, achieving an optimal trade-off between performance and computational overhead. The code will be released upon acceptance.

http://arxiv.org/abs/2507.00480
Posterior Inference in Latent Space for Scalable Constrained Black-box Optimization. (1%)
Kiyoung Om; Kyuil Sim; Taeyoung Yun; Hyeongyu Kang; Jinkyoo Park
Optimizing high-dimensional black-box functions under black-box constraints is a pervasive task in a wide range of scientific and engineering problems. These problems are typically harder than unconstrained problems due to hard-to-find feasible regions. While Bayesian optimization (BO) methods have been developed to solve such problems, they often struggle with the curse of dimensionality. Recently, generative model-based approaches have emerged as a promising alternative for constrained optimization. However, they suffer from poor scalability and are vulnerable to mode collapse, particularly when the target distribution is highly multi-modal. In this paper, we propose a new framework to overcome these challenges. Our method iterates through two stages. First, we train flow-based models to capture the data distribution and surrogate models that predict both function values and constraint violations with uncertainty quantification. Second, we cast the candidate selection problem as a posterior inference problem to effectively search for promising candidates that have high objective values while not violating the constraints. During posterior inference, we find that the posterior distribution is highly multi-modal and has a large plateau due to constraints, especially when constraint feedback is given as binary indicators of feasibility. To mitigate this issue, we amortize the sampling from the posterior distribution in the latent space of flow-based models, which is much smoother than that in the data space. We empirically demonstrate that our method achieves superior performance on various synthetic and real-world constrained black-box optimization tasks. Our code is publicly available \href{https://github.com/umkiyoung/CiBO}{here}.

http://arxiv.org/abs/2506.23581
PBCAT: Patch-based composite adversarial training against physically realizable attacks on object detection. (99%)
Xiao Li; Yiming Zhu; Yifan Huang; Wei Zhang; Yingzhe He; Jie Shi; Xiaolin Hu
Object detection plays a crucial role in many security-sensitive applications. However, several recent studies have shown that object detectors can be easily fooled by physically realizable attacks, \eg, adversarial patches and recent adversarial textures, which pose realistic and urgent threats. Adversarial Training (AT) has been recognized as the most effective defense against adversarial attacks. While AT has been extensively studied in the $l_\infty$ attack settings on classification models, AT against physically realizable attacks on object detectors has received limited exploration. Early attempts are only performed to defend against adversarial patches, leaving AT against a wider range of physically realizable attacks under-explored. In this work, we consider defending against various physically realizable attacks with a unified AT method. We propose PBCAT, a novel Patch-Based Composite Adversarial Training strategy. PBCAT optimizes the model by incorporating the combination of small-area gradient-guided adversarial patches and imperceptible global adversarial perturbations covering the entire image. With these designs, PBCAT has the potential to defend against not only adversarial patches but also unseen physically realizable attacks such as adversarial textures. Extensive experiments in multiple settings demonstrated that PBCAT significantly improved robustness against various physically realizable attacks over state-of-the-art defense methods. Notably, it improved the detection accuracy by 29.7\% over previous defense methods under one recent adversarial texture attack.

http://arxiv.org/abs/2507.02965
Concept-based Adversarial Attack: a Probabilistic Perspective. (99%)
Andi Zhang; Xuan Ding; Steven McDonagh; Samuel Kaski
We propose a concept-based adversarial attack framework that extends beyond single-image perturbations by adopting a probabilistic perspective. Rather than modifying a single image, our method operates on an entire concept -- represented by a probabilistic generative model or a set of images -- to generate diverse adversarial examples. Preserving the concept is essential, as it ensures that the resulting adversarial images remain identifiable as instances of the original underlying category or identity. By sampling from this concept-based adversarial distribution, we generate images that maintain the original concept but vary in pose, viewpoint, or background, thereby misleading the classifier. Mathematically, this framework remains consistent with traditional adversarial attacks in a principled manner. Our theoretical and empirical results demonstrate that concept-based adversarial attacks yield more diverse adversarial examples and effectively preserve the underlying concept, while achieving higher attack efficiency.

http://arxiv.org/abs/2506.24048
Consensus-based optimization for closed-box adversarial attacks and a connection to evolution strategies. (81%)
Tim Roith; Leon Bungert; Philipp Wacker
Consensus-based optimization (CBO) has established itself as an efficient gradient-free optimization scheme, with attractive mathematical properties, such as mean-field convergence results for non-convex loss functions. In this work, we study CBO in the context of closed-box adversarial attacks, which are imperceptible input perturbations that aim to fool a classifier, without accessing its gradient. Our contribution is to establish a connection between the so-called consensus hopping as introduced by Riedl et al. and natural evolution strategies (NES) commonly applied in the context of adversarial attacks and to rigorously relate both methods to gradient-based optimization schemes. Beyond that, we provide a comprehensive experimental study that shows that despite the conceptual similarities, CBO can outperform NES and other evolutionary strategies in certain scenarios.

http://arxiv.org/abs/2506.23676
A Unified Framework for Stealthy Adversarial Generation via Latent Optimization and Transferability Enhancement. (81%)
Gaozheng Pei; Ke Ma; Dongpeng Zhang; Chengzhi Sun; Qianqian Xu; Qingming Huang
Due to their powerful image generation capabilities, diffusion-based adversarial example generation methods through image editing are rapidly gaining popularity. However, due to reliance on the discriminative capability of the diffusion model, these diffusion-based methods often struggle to generalize beyond conventional image classification tasks, such as in Deepfake detection. Moreover, traditional strategies for enhancing adversarial example transferability are challenging to adapt to these methods. To address these challenges, we propose a unified framework that seamlessly incorporates traditional transferability enhancement strategies into diffusion model-based adversarial example generation via image editing, enabling their application across a wider range of downstream tasks. Our method won first place in the "1st Adversarial Attacks on Deepfake Detectors: A Challenge in the Era of AI-Generated Media" competition at ACM MM25, which validates the effectiveness of our approach.

http://arxiv.org/abs/2506.24033
Poisoning Attacks to Local Differential Privacy for Ranking Estimation. (38%)
Pei Zhan; Peng Tang; Yangzhuo Li; Puwen Wei; Shanqing Guo
Local differential privacy (LDP) involves users perturbing their inputs to provide plausible deniability of their data. However, this also makes LDP vulnerable to poisoning attacks. In this paper, we first introduce novel poisoning attacks for ranking estimation. These attacks are intricate, as fake attackers do not merely adjust the frequency of target items. Instead, they leverage a limited number of fake users to precisely modify frequencies, effectively altering item rankings to maximize gains. To tackle this challenge, we introduce the concepts of attack cost and optimal attack item (set), and propose corresponding strategies for kRR, OUE, and OLH protocols. For kRR, we iteratively select optimal attack items and allocate suitable fake users. For OUE, we iteratively determine optimal attack item sets and consider the incremental changes in item frequencies across different sets. Regarding OLH, we develop a harmonic cost function based on the pre-image of a hash to select that supporting a larger number of effective attack items. Lastly, we present an attack strategy based on confidence levels to quantify the probability of a successful attack and the number of attack iterations more precisely. We demonstrate the effectiveness of our attacks through theoretical and empirical evidence, highlighting the necessity for defenses against these attacks. The source code and data have been made available at https://github.com/LDP-user/LDP-Ranking.git.

http://arxiv.org/abs/2506.23534
Improving vulnerability type prediction and line-level detection via adversarial training-based data augmentation and multi-task learning. (13%)
Siyu Chen; Jiongyi Yang; Xiang Chen; Menglin Zheng; Minnan Wei; Xiaolin Ju
Context: Software vulnerabilities pose a significant threat to modern software systems, as evidenced by the growing number of reported vulnerabilities and cyberattacks. These escalating trends underscore the urgent need for effective approaches that can automatically detect and understand software vulnerabilities. Objective: However, the scarcity of labeled samples and the class imbalance issue in vulnerability datasets present significant challenges for both Vulnerability Type Prediction (VTP) and Line-level Vulnerability Detection (LVD), especially for rare yet critical vulnerability types. Moreover, most existing studies treat VTP and LVD as independent tasks, overlooking their inherent correlation, which limits the potential to leverage shared semantic patterns across tasks. Methods: To address these limitations, we propose a unified approach that integrates Embedding-Layer Driven Adversarial Training (EDAT) with Multi-task Learning (MTL). Specifically, EDAT enhances model robustness by introducing adversarial perturbations to identifier embeddings, guided by semantic importance. Meanwhile, MTL improves overall performance by leveraging shared representations and inter-task correlations between VTP and LVD. Results: Extensive experiments demonstrate that our proposed approach outperforms state-of-the-art baselines on both VTP and LVD tasks. For VTP, it yields notable improvements in accuracy, precision, recall, and F1-score, particularly in identifying rare vulnerability types. Similarly, for LVD, our approach enhances line-level detection accuracy while significantly reducing false positives. Conclusion: Our study demonstrates that combining EDAT with MTL provides a unified solution that improves performance on both tasks and warrants further investigation.

http://arxiv.org/abs/2506.24040
Quickest Detection of Adversarial Attacks Against Correlated Equilibria. (4%)
Kiarash Kazari; Aris Kanellopoulos; György Dán
We consider correlated equilibria in strategic games in an adversarial environment, where an adversary can compromise the public signal used by the players for choosing their strategies, while players aim at detecting a potential attack as soon as possible to avoid loss of utility. We model the interaction between the adversary and the players as a zero-sum game and we derive the maxmin strategies for both the defender and the attacker using the framework of quickest change detection. We define a class of adversarial strategies that achieve the optimal trade-off between attack impact and attack detectability and show that a generalized CUSUM scheme is asymptotically optimal for the detection of the attacks. Our numerical results on the Sioux-Falls benchmark traffic routing game show that the proposed detection scheme can effectively limit the utility loss by a potential adversary.

http://arxiv.org/abs/2506.23735
AutoEvoEval: An Automated Framework for Evolving Close-Ended LLM Evaluation Data. (3%)
JiaRu Wu; Mingwei Liu
Large language models (LLMs) have shown remarkable performance on various tasks, but existing evaluation benchmarks are often static and insufficient to fully assess their robustness and generalization in realistic scenarios. Prior work using evolutionary or adversarial data augmentation has improved evaluation diversity but lacks systematic control over perturbation types and multi-step complexity, limiting comprehensive robustness analysis. To address these gaps, we propose AutoEvoEval, an evolution-based evaluation framework for close-ended tasks such as multi-choice question answering. AutoEvoEval introduces 22 interpretable atomic evolution operations and supports multi-round compositions, enabling controlled generation of diverse, challenging, and realistic test samples. We conduct extensive experiments addressing four research questions on a broad set of open- and closed-source LLMs. Our results show that atomic operations cause an average accuracy drop of 7.283\%, with structure-disrupting or misleading semantic edits causing the largest declines. Model sensitivities vary significantly for the same perturbation, and combining multiple evolution steps amplifies adversarial effects by up to 52.932\%. These findings suggest current benchmarks may overestimate true model generalization and emphasize the need for evolution-aware robustness evaluation. Code and resources are available at: https://github.com/SYSUSELab/AutoEvoEval.

http://arxiv.org/abs/2506.24081
SQUASH: A SWAP-Based Quantum Attack to Sabotage Hybrid Quantum Neural Networks. (2%)
Rahul Kumar; Wenqi Wei; Ying Mao; Junaid Farooq; Ying Wang; Juntao Chen
We propose a circuit-level attack, SQUASH, a SWAP-Based Quantum Attack to sabotage Hybrid Quantum Neural Networks (HQNNs) for classification tasks. SQUASH is executed by inserting SWAP gate(s) into the variational quantum circuit of the victim HQNN. Unlike conventional noise-based or adversarial input attacks, SQUASH directly manipulates the circuit structure, leading to qubit misalignment and disrupting quantum state evolution. This attack is highly stealthy, as it does not require access to training data or introduce detectable perturbations in input states. Our results demonstrate that SQUASH significantly degrades classification performance, with untargeted SWAP attacks reducing accuracy by up to 74.08\% and targeted SWAP attacks reducing target class accuracy by up to 79.78\%. These findings reveal a critical vulnerability in HQNN implementations, underscoring the need for more resilient architectures against circuit-level adversarial interventions.

http://arxiv.org/abs/2506.23881
Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution Detection. (1%)
Reihaneh Zohrabi; Hosein Hasani; Mahdieh Soleymani Baghshah; Anna Rohrbach; Marcus Rohrbach; Mohammad Hossein Rohban
Out-of-distribution (OOD) detection is crucial for ensuring the reliability and safety of machine learning models in real-world applications, where they frequently face data distributions unseen during training. Despite progress, existing methods are often vulnerable to spurious correlations that mislead models and compromise robustness. To address this, we propose SPROD, a novel prototype-based OOD detection approach that explicitly addresses the challenge posed by unknown spurious correlations. Our post-hoc method refines class prototypes to mitigate bias from spurious features without additional data or hyperparameter tuning, and is broadly applicable across diverse backbones and OOD detection settings. We conduct a comprehensive spurious correlation OOD detection benchmarking, comparing our method against existing approaches and demonstrating its superior performance across challenging OOD datasets, such as CelebA, Waterbirds, UrbanCars, Spurious Imagenet, and the newly introduced Animals MetaCoCo. On average, SPROD improves AUROC by 4.7% and FPR@95 by 9.3% over the second best.

http://arxiv.org/abs/2506.23814
Breaking Out from the TESSERACT: Reassessing ML-based Malware Detection under Spatio-Temporal Drift. (1%)
Theo Chow; Mario D'Onghia; Lorenz Linhardt; Zeliang Kan; Daniel Arp; Lorenzo Cavallaro; Fabio Pierazzi
Several recent works focused on the best practices for applying machine learning to cybersecurity. In the context of malware, TESSERACT highlighted the impact of concept drift on detection performance and suggested temporal and spatial constraints to be enforced to ensure realistic time-aware evaluations, which have been adopted by the community. In this paper, we demonstrate striking discrepancies in the performance of learning-based malware detection across the same time frame when evaluated on two representative Android malware datasets used in top-tier security conferences, both adhering to established sampling and evaluation guidelines. This questions our ability to understand how current state-of-the-art approaches would perform in realistic scenarios. To address this, we identify five novel temporal and spatial bias factors that affect realistic evaluations. We thoroughly evaluate the impact of these factors in the Android malware domain on two representative datasets and five Android malware classifiers used or proposed in top-tier security conferences. For each factor, we provide practical and actionable recommendations that the community should integrate in their methodology for more realistic and reproducible settings.

http://arxiv.org/abs/2506.23977
A Scalable Approach for Safe and Robust Learning via Lipschitz-Constrained Networks. (1%)
Zain ul Abdeen; Vassilis Kekatos; Ming Jin
Certified robustness is a critical property for deploying neural networks (NN) in safety-critical applications. A principle approach to achieving such guarantees is to constrain the global Lipschitz constant of the network. However, accurate methods for Lipschitz-constrained training often suffer from non-convex formulations and poor scalability due to reliance on global semidefinite programs (SDPs). In this letter, we propose a convex training framework that enforces global Lipschitz constraints via semidefinite relaxation. By reparameterizing the NN using loop transformation, we derive a convex admissibility condition that enables tractable and certifiable training. While the resulting formulation guarantees robustness, its scalability is limited by the size of global SDP. To overcome this, we develop a randomized subspace linear matrix inequalities (RS-LMI) approach that decomposes the global constraints into sketched layerwise constraints projected onto low-dimensional subspaces, yielding a smooth and memory-efficient training objective. Empirical results on MNIST, CIFAR-10, and ImageNet demonstrate that the proposed framework achieves competitive accuracy with significantly improved Lipschitz bounds and runtime performance.

http://arxiv.org/abs/2506.20702
The Singapore Consensus on Global AI Safety Research Priorities. (1%)
Yoshua Bengio; Tegan Maharaj; Luke Ong; Stuart Russell; Dawn Song; Max Tegmark; Lan Xue; Ya-Qin Zhang; Stephen Casper; Wan Sie Lee; Sören Mindermann; Vanessa Wilfred; Vidhisha Balachandran; Fazl Barez; Michael Belinsky; Imane Bello; Malo Bourgon; Mark Brakel; Siméon Campos; Duncan Cass-Beggs; Jiahao Chen; Rumman Chowdhury; Kuan Chua Seah; Jeff Clune; Juntao Dai; Agnes Delaborde; Nouha Dziri; Francisco Eiras; Joshua Engels; Jinyu Fan; Adam Gleave; Noah Goodman; Fynn Heide; Johannes Heidecke; Dan Hendrycks; Cyrus Hodes; Bryan Low Kian Hsiang; Minlie Huang; Sami Jawhar; Wang Jingyu; Adam Tauman Kalai; Meindert Kamphuis; Mohan Kankanhalli; Subhash Kantamneni; Mathias Bonde Kirk; Thomas Kwa; Jeffrey Ladish; Kwok-Yan Lam; Wan Lee Sie; Taewhi Lee; Xiaojian Li; Jiajun Liu; Chaochao Lu; Yifan Mai; Richard Mallah; Julian Michael; Nick Moës; Simon Möller; Kihyuk Nam; Kwan Yee Ng; Mark Nitzberg; Besmira Nushi; Seán O hÉigeartaigh; Alejandro Ortega; Pierre Peigné; James Petrie; Benjamin Prud'Homme; Reihaneh Rabbany; Nayat Sanchez-Pi; Sarah Schwettmann; Buck Shlegeris; Saad Siddiqui; Aradhana Sinha; Martín Soto; Cheston Tan; Dong Ting; William Tjhi; Robert Trager; Brian Tse; Anthony Tung K. H.; Vanessa Wilfred; John Willes; Denise Wong; Wei Xu; Rongwu Xu; Yi Zeng; HongJiang Zhang; Djordje Žikelić
Rapidly improving AI capabilities and autonomy hold significant promise of transformation, but are also driving vigorous debate on how to ensure that AI is safe, i.e., trustworthy, reliable, and secure. Building a trusted ecosystem is therefore essential -- it helps people embrace AI with confidence and gives maximal space for innovation while avoiding backlash.
  The "2025 Singapore Conference on AI (SCAI): International Scientific Exchange on AI Safety" aimed to support research in this space by bringing together AI scientists across geographies to identify and synthesise research priorities in AI safety. This resulting report builds on the International AI Safety Report chaired by Yoshua Bengio and backed by 33 governments. By adopting a defence-in-depth model, this report organises AI safety research domains into three types: challenges with creating trustworthy AI systems (Development), challenges with evaluating their risks (Assessment), and challenges with monitoring and intervening after deployment (Control).

http://arxiv.org/abs/2506.23260
From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows. (16%)
Mohamed Amine Ferrag; Norbert Tihanyi; Djallel Hamouda; Leandros Maglaras; Merouane Debbah
Autonomous AI agents powered by large language models (LLMs) with structured function-calling interfaces have dramatically expanded capabilities for real-time data retrieval, complex computation, and multi-step orchestration. Yet, the explosive proliferation of plugins, connectors, and inter-agent protocols has outpaced discovery mechanisms and security practices, resulting in brittle integrations vulnerable to diverse threats. In this survey, we introduce the first unified, end-to-end threat model for LLM-agent ecosystems, spanning host-to-tool and agent-to-agent communications, formalize adversary capabilities and attacker objectives, and catalog over thirty attack techniques. Specifically, we organized the threat model into four domains: Input Manipulation (e.g., prompt injections, long-context hijacks, multimodal adversarial inputs), Model Compromise (e.g., prompt- and parameter-level backdoors, composite and encrypted multi-backdoors, poisoning strategies), System and Privacy Attacks (e.g., speculative side-channels, membership inference, retrieval poisoning, social-engineering simulations), and Protocol Vulnerabilities (e.g., exploits in Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent Network Protocol (ANP), and Agent-to-Agent (A2A) protocol). For each category, we review representative scenarios, assess real-world feasibility, and evaluate existing defenses. Building on our threat taxonomy, we identify key open challenges and future research directions, such as securing MCP deployments through dynamic trust management and cryptographic provenance tracking; designing and hardening Agentic Web Interfaces; and achieving resilience in multi-agent and federated environments. Our work provides a comprehensive reference to guide the design of robust defense mechanisms and establish best practices for resilient LLM-agent workflows.

http://arxiv.org/abs/2407.09972
MedLeak: Multimodal Medical Data Leakage in Secure Federated Learning with Crafted Models. (12%)
Shanghao Shi; Md Shahedul Haque; Abhijeet Parida; Chaoyu Zhang; Marius George Linguraru; Y. Thomas Hou; Syed Muhammad Anwar; Wenjing Lou
Federated learning (FL) allows participants to collaboratively train machine learning models while keeping their data local, making it ideal for collaborations among healthcare institutions on sensitive data. However, in this paper, we propose a novel privacy attack called MedLeak, which allows a malicious FL server to recover high-quality site-specific private medical data from the client model updates. MedLeak works by introducing an adversarially crafted model during the FL training process. Honest clients, unaware of the insidious changes in the published models, continue to send back their updates as per the standard FL protocol. Leveraging a novel analytical method, MedLeak can efficiently recover private client data from the aggregated parameter updates, eliminating costly optimization. In addition, the scheme relies solely on the aggregated updates, thus rendering secure aggregation protocols ineffective, as they depend on the randomization of intermediate results for security while leaving the final aggregated results unaltered.
  We implement MedLeak on medical image datasets (MedMNIST, COVIDx CXR-4, and Kaggle Brain Tumor MRI), as well as a medical text dataset (MedAbstract). The results demonstrate that our attack achieves high recovery rates and strong quantitative scores on both image and text datasets. We also thoroughly evaluate MedLeak across different attack parameters, providing insights into key factors that influence attack performance and potential defenses. Furthermore, we demonstrate that the recovered data can support downstream tasks such as disease classification with minimal performance loss. Our findings validate the need for enhanced privacy measures in FL systems, particularly for safeguarding sensitive medical data against powerful model inversion attacks.

http://arxiv.org/abs/2506.23189
Trident: Detecting Face Forgeries with Adversarial Triplet Learning. (1%)
Mustafa Hakan Kara; Aysegul Dundar; Uğur Güdükbay
As face forgeries generated by deep neural networks become increasingly sophisticated, detecting face manipulations in digital media has posed a significant challenge, underscoring the importance of maintaining digital media integrity and combating visual disinformation. Current detection models, predominantly based on supervised training with domain-specific data, often falter against forgeries generated by unencountered techniques. In response to this challenge, we introduce \textit{Trident}, a face forgery detection framework that employs triplet learning with a Siamese network architecture for enhanced adaptability across diverse forgery methods. \textit{Trident} is trained on curated triplets to isolate nuanced differences of forgeries, capturing fine-grained features that distinguish pristine samples from manipulated ones while controlling for other variables. To further enhance generalizability, we incorporate domain-adversarial training with a forgery discriminator. This adversarial component guides our embedding model towards forgery-agnostic representations, improving its robustness to unseen manipulations. In addition, we prevent gradient flow from the classifier head to the embedding model, avoiding overfitting induced by artifacts peculiar to certain forgeries. Comprehensive evaluations across multiple benchmarks and ablation studies demonstrate the effectiveness of our framework. We will release our code in a GitHub repository.

http://arxiv.org/abs/2506.23280
BAPE: Learning an Explicit Bayes Classifier for Long-tailed Visual Recognition. (1%)
Chaoqun Du; Yulin Wang; Shiji Song; Gao Huang
Bayesian decision theory advocates the Bayes classifier as the optimal approach for minimizing the risk in machine learning problems. Current deep learning algorithms usually solve for the optimal classifier by \emph{implicitly} estimating the posterior probabilities, \emph{e.g.}, by minimizing the Softmax cross-entropy loss. This simple methodology has been proven effective for meticulously balanced academic benchmark datasets. However, it is not applicable to the long-tailed data distributions in the real world, where it leads to the gradient imbalance issue and fails to ensure the Bayes optimal decision rule. To address these challenges, this paper presents a novel approach (BAPE) that provides a more precise theoretical estimation of the data distributions by \emph{explicitly} modeling the parameters of the posterior probabilities and solving them with point estimation. Consequently, our method directly learns the Bayes classifier without gradient descent based on Bayes' theorem, simultaneously alleviating the gradient imbalance and ensuring the Bayes optimal decision rule. Furthermore, we propose a straightforward yet effective \emph{distribution adjustment} technique. This method enables the Bayes classifier trained from the long-tailed training set to effectively adapt to the test data distribution with an arbitrary imbalance factor, thereby enhancing performance without incurring additional computational costs. In addition, we demonstrate the gains of our method are orthogonal to existing learning approaches for long-tailed scenarios, as they are mostly designed under the principle of \emph{implicitly} estimating the posterior probabilities. Extensive empirical evaluations on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist demonstrate that our method significantly improves the generalization performance of popular deep networks, despite its simplicity.

http://arxiv.org/abs/2506.22982
Revisiting CroPA: A Reproducibility Study and Enhancements for Cross-Prompt Adversarial Transferability in Vision-Language Models. (99%)
Atharv Mittal; Agam Pandey; Amritanshu Tiwari; Sukrit Jindal; Swadesh Swain
Large Vision-Language Models (VLMs) have revolutionized computer vision, enabling tasks such as image classification, captioning, and visual question answering. However, they remain highly vulnerable to adversarial attacks, particularly in scenarios where both visual and textual modalities can be manipulated. In this study, we conduct a comprehensive reproducibility study of "An Image is Worth 1000 Lies: Adversarial Transferability Across Prompts on Vision-Language Models" validating the Cross-Prompt Attack (CroPA) and confirming its superior cross-prompt transferability compared to existing baselines. Beyond replication we propose several key improvements: (1) A novel initialization strategy that significantly improves Attack Success Rate (ASR). (2) Investigate cross-image transferability by learning universal perturbations. (3) A novel loss function targeting vision encoder attention mechanisms to improve generalization. Our evaluation across prominent VLMs -- including Flamingo, BLIP-2, and InstructBLIP as well as extended experiments on LLaVA validates the original results and demonstrates that our improvements consistently boost adversarial effectiveness. Our work reinforces the importance of studying adversarial vulnerabilities in VLMs and provides a more robust framework for generating transferable adversarial examples, with significant implications for understanding the security of VLMs in real-world applications.

http://arxiv.org/abs/2506.22722
Kill Two Birds with One Stone! Trajectory enabled Unified Online Detection of Adversarial Examples and Backdoor Attacks. (98%)
Anmin Fu; Fanyu Meng; Huaibing Peng; Hua Ma; Zhi Zhang; Yifeng Zheng; Willy Susilo; Yansong Gao
The proposed UniGuard is the first unified online detection framework capable of simultaneously addressing adversarial examples and backdoor attacks. UniGuard builds upon two key insights: first, both AE and backdoor attacks have to compromise the inference phase, making it possible to tackle them simultaneously during run-time via online detection. Second, an adversarial input, whether a perturbed sample in AE attacks or a trigger-carrying sample in backdoor attacks, exhibits distinctive trajectory signatures from a benign sample as it propagates through the layers of a DL model in forward inference. The propagation trajectory of the adversarial sample must deviate from that of its benign counterpart; otherwise, the adversarial objective cannot be fulfilled. Detecting these trajectory signatures is inherently challenging due to their subtlety; UniGuard overcomes this by treating the propagation trajectory as a time-series signal, leveraging LSTM and spectrum transformation to amplify differences between adversarial and benign trajectories that are subtle in the time domain. UniGuard exceptional efficiency and effectiveness have been extensively validated across various modalities (image, text, and audio) and tasks (classification and regression), ranging from diverse model architectures against a wide range of AE attacks and backdoor attacks, including challenging partial backdoors and dynamic triggers. When compared to SOTA methods, including ContraNet (NDSS 22) specific for AE detection and TED (IEEE SP 24) specific for backdoor detection, UniGuard consistently demonstrates superior performance, even when matched against each method's strengths in addressing their respective threats-each SOTA fails to parts of attack strategies while UniGuard succeeds for all.

http://arxiv.org/abs/2506.22776
Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation. (22%)
Sen Fang; Weiyuan Ding; Antonio Mastropaolo; Bowen Xu
Quantization has emerged as a mainstream method for compressing Large Language Models (LLMs), reducing memory requirements and accelerating inference without architectural modifications. While existing research primarily focuses on evaluating the effectiveness of quantized LLMs compared to their original counterparts, the impact on robustness remains largely unexplored.In this paper, we present the first systematic investigation of how quantization affects the robustness of LLMs in code generation tasks. Through extensive experiments across four prominent LLM families (LLaMA, DeepSeek, CodeGen, and StarCoder) with parameter scales ranging from 350M to 33B, we evaluate robustness from dual perspectives: adversarial attacks on input prompts and noise perturbations on model architecture. Our findings challenge conventional wisdom by demonstrating that quantized LLMs often exhibit superior robustness compared to their full-precision counterparts, with 51.59% versus 42.86% of our adversarial experiments showing better resilience in quantized LLMs. Similarly, our noise perturbation experiments also confirm that LLMs after quantitation generally withstand higher levels of weight disturbances. These results suggest that quantization not only reduces computational requirements but can actually enhance LLMs' reliability in code generation tasks, providing valuable insights for developing more robust and efficient LLM deployment strategies.

http://arxiv.org/abs/2506.22806
Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate. (3%)
Byung Hyun Lee; Sungjin Lim; Seunggyu Lee; Dong Un Kang; Se Young Chun
Remarkable progress in text-to-image diffusion models has brought a major concern about potentially generating images on inappropriate or trademarked concepts. Concept erasing has been investigated with the goals of deleting target concepts in diffusion models while preserving other concepts with minimal distortion. To achieve these goals, recent concept erasing methods usually fine-tune the cross-attention layers of diffusion models. In this work, we first show that merely updating the cross-attention layers in diffusion models, which is mathematically equivalent to adding \emph{linear} modules to weights, may not be able to preserve diverse remaining concepts. Then, we propose a novel framework, dubbed Concept Pinpoint Eraser (CPE), by adding \emph{nonlinear} Residual Attention Gates (ResAGs) that selectively erase (or cut) target concepts while safeguarding remaining concepts from broad distributions by employing an attention anchoring loss to prevent the forgetting. Moreover, we adversarially train CPE with ResAG and learnable text embeddings in an iterative manner to maximize erasing performance and enhance robustness against adversarial attacks. Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. Code is available at https://github.com/Hyun1A/CPE

http://arxiv.org/abs/2506.21874
On the Feasibility of Poisoning Text-to-Image AI Models via Adversarial Mislabeling. (99%)
Stanley Wu; Ronik Bhaskar; Anna Yoo Jeong Ha; Shawn Shan; Haitao Zheng; Ben Y. Zhao
Today's text-to-image generative models are trained on millions of images sourced from the Internet, each paired with a detailed caption produced by Vision-Language Models (VLMs). This part of the training pipeline is critical for supplying the models with large volumes of high-quality image-caption pairs during training. However, recent work suggests that VLMs are vulnerable to stealthy adversarial attacks, where adversarial perturbations are added to images to mislead the VLMs into producing incorrect captions.
  In this paper, we explore the feasibility of adversarial mislabeling attacks on VLMs as a mechanism to poisoning training pipelines for text-to-image models. Our experiments demonstrate that VLMs are highly vulnerable to adversarial perturbations, allowing attackers to produce benign-looking images that are consistently miscaptioned by the VLM models. This has the effect of injecting strong "dirty-label" poison samples into the training pipeline for text-to-image models, successfully altering their behavior with a small number of poisoned samples. We find that while potential defenses can be effective, they can be targeted and circumvented by adaptive attackers. This suggests a cat-and-mouse game that is likely to reduce the quality of training data and increase the cost of text-to-image model development. Finally, we demonstrate the real-world effectiveness of these attacks, achieving high attack success (over 73%) even in black-box scenarios against commercial VLMs (Google Vertex AI and Microsoft Azure).

http://arxiv.org/abs/2506.22602
Are Fast Methods Stable in Adversarially Robust Transfer Learning? (96%)
Joshua C. Zhao; Saurabh Bagchi
Transfer learning is often used to decrease the computational cost of model training, as fine-tuning a model allows a downstream task to leverage the features learned from the pre-training dataset and quickly adapt them to a new task. This is particularly useful for achieving adversarial robustness, as adversarially training models from scratch is very computationally expensive. However, high robustness in transfer learning still requires adversarial training during the fine-tuning phase, which requires up to an order of magnitude more time than standard fine-tuning. In this work, we revisit the use of the fast gradient sign method (FGSM) in robust transfer learning to improve the computational cost of adversarial fine-tuning. We surprisingly find that FGSM is much more stable in adversarial fine-tuning than when training from scratch. In particular, FGSM fine-tuning does not suffer from any issues with catastrophic overfitting at standard perturbation budgets of $\varepsilon=4$ or $\varepsilon=8$. This stability is further enhanced with parameter-efficient fine-tuning methods, where FGSM remains stable even up to $\varepsilon=32$ for linear probing. We demonstrate how this stability translates into performance across multiple datasets. Compared to fine-tuning with the more commonly used method of projected gradient descent (PGD), on average, FGSM only loses 0.39% and 1.39% test robustness for $\varepsilon=4$ and $\varepsilon=8$ while using $4\times$ less training time. Surprisingly, FGSM may not only be a significantly more efficient alternative to PGD in adversarially robust transfer learning but also a well-performing one.

http://arxiv.org/abs/2506.21972
Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses. (81%)
Mohamed Ahmed; Mohamed Abdelmouty; Mingyu Kim; Gunvanth Kandula; Alex Park; James C. Davis
The advancement of Pre-Trained Language Models (PTLMs) and Large Language Models (LLMs) has led to their widespread adoption across diverse applications. Despite their success, these models remain vulnerable to attacks that exploit their inherent weaknesses to bypass safety measures. Two primary inference-phase threats are token-level and prompt-level jailbreaks. Token-level attacks embed adversarial sequences that transfer well to black-box models like GPT but leave detectable patterns and rely on gradient-based token optimization, whereas prompt-level attacks use semantically structured inputs to elicit harmful responses yet depend on iterative feedback that can be unreliable. To address the complementary limitations of these methods, we propose two hybrid approaches that integrate token- and prompt-level techniques to enhance jailbreak effectiveness across diverse PTLMs. GCG + PAIR and the newly explored GCG + WordGame hybrids were evaluated across multiple Vicuna and Llama models. GCG + PAIR consistently raised attack-success rates over its constituent techniques on undefended models; for instance, on Llama-3, its Attack Success Rate (ASR) reached 91.6%, a substantial increase from PAIR's 58.4% baseline. Meanwhile, GCG + WordGame matched the raw performance of WordGame maintaining a high ASR of over 80% even under stricter evaluators like Mistral-Sorry-Bench. Crucially, both hybrids retained transferability and reliably pierced advanced defenses such as Gradient Cuff and JBShield, which fully blocked single-mode attacks. These findings expose previously unreported vulnerabilities in current safety stacks, highlight trade-offs between raw success and defensive robustness, and underscore the need for holistic safeguards against adaptive adversaries.

http://arxiv.org/abs/2506.21842
Adversarial Threats in Quantum Machine Learning: A Survey of Attacks and Defenses. (13%)
Archisman Ghosh; Satwik Kundu; Swaroop Ghosh
Quantum Machine Learning (QML) integrates quantum computing with classical machine learning, primarily to solve classification, regression and generative tasks. However, its rapid development raises critical security challenges in the Noisy Intermediate-Scale Quantum (NISQ) era. This chapter examines adversarial threats unique to QML systems, focusing on vulnerabilities in cloud-based deployments, hybrid architectures, and quantum generative models. Key attack vectors include model stealing via transpilation or output extraction, data poisoning through quantum-specific perturbations, reverse engineering of proprietary variational quantum circuits, and backdoor attacks. Adversaries exploit noise-prone quantum hardware and insufficiently secured QML-as-a-Service (QMLaaS) workflows to compromise model integrity, ownership, and functionality. Defense mechanisms leverage quantum properties to counter these threats. Noise signatures from training hardware act as non-invasive watermarks, while hardware-aware obfuscation techniques and ensemble strategies disrupt cloning attempts. Emerging solutions also adapt classical adversarial training and differential privacy to quantum settings, addressing vulnerabilities in quantum neural networks and generative architectures. However, securing QML requires addressing open challenges such as balancing noise levels for reliability and security, mitigating cross-platform attacks, and developing quantum-classical trust frameworks. This chapter summarizes recent advances in attacks and defenses, offering a roadmap for researchers and practitioners to build robust, trustworthy QML systems resilient to evolving adversarial landscapes.

http://arxiv.org/abs/2506.21883
GRASP-PsONet: Gradient-based Removal of Spurious Patterns for PsOriasis Severity Classification. (8%)
Basudha Pal; Sharif Amit Kamran; Brendon Lutnick; Molly Lucas; Chaitanya Parmar; Asha Patel Shah; David Apfel; Steven Fakharzadeh; Lloyd Miller; Gabriela Cula; Kristopher Standish
Psoriasis (PsO) severity scoring is important for clinical trials but is hindered by inter-rater variability and the burden of in person clinical evaluation. Remote imaging using patient captured mobile photos offers scalability but introduces challenges, such as variation in lighting, background, and device quality that are often imperceptible to humans but can impact model performance. These factors, along with inconsistencies in dermatologist annotations, reduce the reliability of automated severity scoring. We propose a framework to automatically flag problematic training images that introduce spurious correlations which degrade model generalization, using a gradient based interpretability approach. By tracing the gradients of misclassified validation images, we detect training samples where model errors align with inconsistently rated examples or are affected by subtle, nonclinical artifacts. We apply this method to a ConvNeXT based weakly supervised model designed to classify PsO severity from phone images. Removing 8.2% of flagged images improves model AUC-ROC by 5% (85% to 90%) on a held out test set. Commonly, multiple annotators and an adjudication process ensure annotation accuracy, which is expensive and time consuming. Our method detects training images with annotation inconsistencies, potentially removing the need for manual review. When applied to a subset of training data rated by two dermatologists, the method identifies over 90% of cases with inter-rater disagreement by reviewing only the top 30% of samples. This improves automated scoring for remote assessments, ensuring robustness despite data collection variability.

http://arxiv.org/abs/2403.03508
EXPRTS: Exploring and Probing the Robustness of Time Series Forecasting Models. (1%)
Håkon Hanisch Kjærnli; Lluis Mas-Ribas; Hans Jakob Håland; Vegard Sjåvik; Aida Ashrafi; Helge Langseth; Odd Erik Gundersen
When deploying time series forecasting models based on machine learning to real world settings, one often encounter situations where the data distribution drifts. Such drifts expose the forecasting models to out-of-distribution (OOD) data, and machine learning models lack robustness in these settings. Robustness can be improved by using deep generative models or genetic algorithms to augment time series datasets, but these approaches lack interpretability and are computationally expensive. In this work, we develop an interpretable and simple framework for generating time series. Our method combines time-series decompositions with analytic functions, and is able to generate time series with characteristics matching both in- and out-of-distribution data. This approach allows users to generate new time series in an interpretable fashion, which can be used to augment the dataset and improve forecasting robustness. We demonstrate our framework through EXPRTS, a visual analytics tool designed for univariate time series forecasting models and datasets. Different visualizations of the data distribution, forecasting errors and single time series instances enable users to explore time series datasets, apply transformations, and evaluate forecasting model robustness across diverse scenarios. We show how our framework can generate meaningful OOD time series that improve model robustness, and we validate EXPRTS effectiveness and usability through three use-cases and a user study.

http://arxiv.org/abs/2506.21046
Boosting Generative Adversarial Transferability with Self-supervised Vision Transformer Features. (98%)
Shangbo Wu; Yu-an Tan; Ruinan Ma; Wencong Ma; Dehua Zhu; Yuanzhang Li
The ability of deep neural networks (DNNs) come from extracting and interpreting features from the data provided. By exploiting intermediate features in DNNs instead of relying on hard labels, we craft adversarial perturbation that generalize more effectively, boosting black-box transferability. These features ubiquitously come from supervised learning in previous work. Inspired by the exceptional synergy between self-supervised learning and the Transformer architecture, this paper explores whether exploiting self-supervised Vision Transformer (ViT) representations can improve adversarial transferability. We present dSVA -- a generative dual self-supervised ViT features attack, that exploits both global structural features from contrastive learning (CL) and local textural features from masked image modeling (MIM), the self-supervised learning paradigm duo for ViTs. We design a novel generative training framework that incorporates a generator to create black-box adversarial examples, and strategies to train the generator by exploiting joint features and the attention mechanism of self-supervised ViTs. Our findings show that CL and MIM enable ViTs to attend to distinct feature tendencies, which, when exploited in tandem, boast great adversarial generalizability. By disrupting dual deep features distilled by self-supervised ViTs, we are rewarded with remarkable black-box transferability to models of various architectures that outperform state-of-the-arts. Code available at https://github.com/spencerwooo/dSVA.

http://arxiv.org/abs/2506.21142
Generative Adversarial Evasion and Out-of-Distribution Detection for UAV Cyber-Attacks. (98%)
Deepak Kumar Panda; Weisi Guo
The growing integration of UAVs into civilian airspace underscores the need for resilient and intelligent intrusion detection systems (IDS), as traditional anomaly detection methods often fail to identify novel threats. A common approach treats unfamiliar attacks as out-of-distribution (OOD) samples; however, this leaves systems vulnerable when mitigation is inadequate. Moreover, conventional OOD detectors struggle to distinguish stealthy adversarial attacks from genuine OOD events. This paper introduces a conditional generative adversarial network (cGAN)-based framework for crafting stealthy adversarial attacks that evade IDS mechanisms. We first design a robust multi-class IDS classifier trained on benign UAV telemetry and known cyber-attacks, including Denial of Service (DoS), false data injection (FDI), man-in-the-middle (MiTM), and replay attacks. Using this classifier, our cGAN perturbs known attacks to generate adversarial samples that misclassify as benign while retaining statistical resemblance to OOD distributions. These adversarial samples are iteratively refined to achieve high stealth and success rates. To detect such perturbations, we implement a conditional variational autoencoder (CVAE), leveraging negative log-likelihood to separate adversarial inputs from authentic OOD samples. Comparative evaluation shows that CVAE-based regret scores significantly outperform traditional Mahalanobis distance-based detectors in identifying stealthy adversarial threats. Our findings emphasize the importance of advanced probabilistic modeling to strengthen IDS capabilities against adaptive, generative-model-based cyber intrusions.

http://arxiv.org/abs/2506.21129
Curriculum-Guided Antifragile Reinforcement Learning for Secure UAV Deconfliction under Observation-Space Attacks. (89%)
Deepak Kumar Panda; Adolfo Perrusquia; Weisi Guo
Reinforcement learning (RL) policies deployed in safety-critical systems, such as unmanned aerial vehicle (UAV) navigation in dynamic airspace, are vulnerable to out-ofdistribution (OOD) adversarial attacks in the observation space. These attacks induce distributional shifts that significantly degrade value estimation, leading to unsafe or suboptimal decision making rendering the existing policy fragile. To address this vulnerability, we propose an antifragile RL framework designed to adapt against curriculum of incremental adversarial perturbations. The framework introduces a simulated attacker which incrementally increases the strength of observation-space perturbations which enables the RL agent to adapt and generalize across a wider range of OOD observations and anticipate previously unseen attacks. We begin with a theoretical characterization of fragility, formally defining catastrophic forgetting as a monotonic divergence in value function distributions with increasing perturbation strength. Building on this, we define antifragility as the boundedness of such value shifts and derive adaptation conditions under which forgetting is stabilized. Our method enforces these bounds through iterative expert-guided critic alignment using Wasserstein distance minimization across incrementally perturbed observations. We empirically evaluate the approach in a UAV deconfliction scenario involving dynamic 3D obstacles. Results show that the antifragile policy consistently outperforms standard and robust RL baselines when subjected to both projected gradient descent (PGD) and GPS spoofing attacks, achieving up to 15% higher cumulative reward and over 30% fewer conflict events. These findings demonstrate the practical and theoretical viability of antifragile reinforcement learning for secure and resilient decision-making in environments with evolving threat scenarios.

http://arxiv.org/abs/2506.21127
Robust Policy Switching for Antifragile Reinforcement Learning for UAV Deconfliction in Adversarial Environments. (82%)
Deepak Kumar Panda; Weisi Guo
The increasing automation of navigation for unmanned aerial vehicles (UAVs) has exposed them to adversarial attacks that exploit vulnerabilities in reinforcement learning (RL) through sensor manipulation. Although existing robust RL methods aim to mitigate such threats, their effectiveness has limited generalization to out-of-distribution shifts from the optimal value distribution, as they are primarily designed to handle fixed perturbation. To address this limitation, this paper introduces an antifragile RL framework that enhances adaptability to broader distributional shifts by incorporating a switching mechanism based on discounted Thompson sampling (DTS). This mechanism dynamically selects among multiple robust policies to minimize adversarially induced state-action-value distribution shifts. The proposed approach first derives a diverse ensemble of action robust policies by accounting for a range of perturbations in the policy space. These policies are then modeled as a multiarmed bandit (MAB) problem, where DTS optimally selects policies in response to nonstationary Bernoulli rewards, effectively adapting to evolving adversarial strategies. Theoretical framework has also been provided where by optimizing the DTS to minimize the overall regrets due to distributional shift, results in effective adaptation against unseen adversarial attacks thus inducing antifragility. Extensive numerical simulations validate the effectiveness of the proposed framework in complex navigation environments with multiple dynamic three-dimensional obstacles and with stronger projected gradient descent (PGD) and spoofing attacks. Compared to conventional robust, non-adaptive RL methods, the antifragile approach achieves superior performance, demonstrating shorter navigation path lengths and a higher rate of conflict-free navigation trajectories compared to existing robust RL techniques

http://arxiv.org/abs/2506.20931
SPA: Towards More Stealth and Persistent Backdoor Attacks in Federated Learning. (75%)
Chengcheng Zhu; Ye Li; Bosen Rao; Jiale Zhang; Yunlong Mao; Sheng Zhong
Federated Learning (FL) has emerged as a leading paradigm for privacy-preserving distributed machine learning, yet the distributed nature of FL introduces unique security challenges, notably the threat of backdoor attacks. Existing backdoor strategies predominantly rely on end-to-end label supervision, which, despite their efficacy, often results in detectable feature disentanglement and limited persistence. In this work, we propose a novel and stealthy backdoor attack framework, named SPA, which fundamentally departs from traditional approaches by leveraging feature-space alignment rather than direct trigger-label association. Specifically, SPA reduces representational distances between backdoor trigger features and target class features, enabling the global model to misclassify trigger-embedded inputs with high stealth and persistence. We further introduce an adaptive, adversarial trigger optimization mechanism, utilizing boundary-search in the feature space to enhance attack longevity and effectiveness, even against defensive FL scenarios and non-IID data distributions. Extensive experiments on various FL benchmarks demonstrate that SPA consistently achieves high attack success rates with minimal impact on model utility, maintains robustness under challenging participation and data heterogeneity conditions, and exhibits persistent backdoor effects far exceeding those of conventional techniques. Our results call urgent attention to the evolving sophistication of backdoor threats in FL and emphasize the pressing need for advanced, feature-level defense techniques.

http://arxiv.org/abs/2506.09803
Devil's Hand: Data Poisoning Attacks to Locally Private Graph Learning Protocols. (68%)
Longzhu He; Chaozhuo Li; Peng Tang; Li Sun; Sen Su; Philip S. Yu
Graph neural networks (GNNs) have achieved significant success in graph representation learning and have been applied to various domains. However, many real-world graphs contain sensitive personal information, such as user profiles in social networks, raising serious privacy concerns when graph learning is performed using GNNs. To address this issue, locally private graph learning protocols have gained considerable attention. These protocols leverage the privacy advantages of local differential privacy (LDP) and the effectiveness of GNN's message-passing in calibrating noisy data, offering strict privacy guarantees for users' local data while maintaining high utility (e.g., node classification accuracy) for graph learning. Despite these advantages, such protocols may be vulnerable to data poisoning attacks, a threat that has not been considered in previous research. Identifying and addressing these threats is crucial for ensuring the robustness and security of privacy-preserving graph learning frameworks. This work introduces the first data poisoning attack targeting locally private graph learning protocols. The attacker injects fake users into the protocol, manipulates these fake users to establish links with genuine users, and sends carefully crafted data to the server, ultimately compromising the utility of private graph learning. The effectiveness of the attack is demonstrated both theoretically and empirically. In addition, several defense strategies have also been explored, but their limited effectiveness highlights the need for more robust defenses.

http://arxiv.org/abs/2506.22521
A Survey on Model Extraction Attacks and Defenses for Large Language Models. (10%)
Kaixiang Zhao; Lincan Li; Kaize Ding; Neil Zhenqiang Gong; Yue Zhao; Yushun Dong
Model extraction attacks pose significant security threats to deployed language models, potentially compromising intellectual property and user privacy. This survey provides a comprehensive taxonomy of LLM-specific extraction attacks and defenses, categorizing attacks into functionality extraction, training data extraction, and prompt-targeted attacks. We analyze various attack methodologies including API-based knowledge distillation, direct querying, parameter recovery, and prompt stealing techniques that exploit transformer architectures. We then examine defense mechanisms organized into model protection, data privacy protection, and prompt-targeted strategies, evaluating their effectiveness across different deployment scenarios. We propose specialized metrics for evaluating both attack effectiveness and defense performance, addressing the specific challenges of generative language models. Through our analysis, we identify critical limitations in current approaches and propose promising research directions, including integrated attack methodologies and adaptive defense mechanisms that balance security with model utility. This work serves NLP researchers, ML engineers, and security professionals seeking to protect language models in production environments.

http://arxiv.org/abs/2506.21106
PhishKey: A Novel Centroid-Based Approach for Enhanced Phishing Detection Using Adaptive HTML Component Extraction. (1%)
Felipe Castaño; Eduardo Fidalgo; Enrique Alegre; Rocio Alaiz-Rodríguez; Raul Orduna; Francesco Zola
Phishing attacks pose a significant cybersecurity threat, evolving rapidly to bypass detection mechanisms and exploit human vulnerabilities. This paper introduces PhishKey to address the challenges of adaptability, robustness, and efficiency. PhishKey is a novel phishing detection method using automatic feature extraction from hybrid sources. PhishKey combines character-level processing with Convolutional Neural Networks (CNN) for URL classification, and a Centroid-Based Key Component Phishing Extractor (CAPE) for HTML content at the word level. CAPE reduces noise and ensures complete sample processing avoiding crop operations on the input data. The predictions from both modules are integrated using a soft-voting ensemble to achieve more accurate and reliable classifications. Experimental evaluations on four state-of-the-art datasets demonstrate the effectiveness of PhishKey. It achieves up to 98.70% F1 Score and shows strong resistance to adversarial manipulations such as injection attacks with minimal performance degradation.

http://arxiv.org/abs/2506.20816
Universal and Efficient Detection of Adversarial Data through Nonuniform Impact on Network Layers. (99%)
Furkan Mumcu; Yasin Yilmaz
Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial input designs with limited noise budgets. While numerous successful attacks with subtle modifications to original input have been proposed, defense techniques against these attacks are relatively understudied. Existing defense approaches either focus on improving DNN robustness by negating the effects of perturbations or use a secondary model to detect adversarial data. Although equally important, the attack detection approach, which is studied in this work, provides a more practical defense compared to the robustness approach. We show that the existing detection methods are either ineffective against the state-of-the-art attack techniques or computationally inefficient for real-time processing. We propose a novel universal and efficient method to detect adversarial examples by analyzing the varying degrees of impact of attacks on different DNN layers. {Our method trains a lightweight regression model that predicts deeper-layer features from early-layer features, and uses the prediction error to detect adversarial samples.} Through theoretical arguments and extensive experiments, we demonstrate that our detection method is highly effective, computationally efficient for real-time processing, compatible with any DNN architecture, and applicable across different domains, such as image, video, and audio.

http://arxiv.org/abs/2506.20806
Poster: Enhancing GNN Robustness for Network Intrusion Detection via Agent-based Analysis. (98%)
Zhonghao Zhan; Huichi Zhou; Hamed Haddadi
Graph Neural Networks (GNNs) show great promise for Network Intrusion Detection Systems (NIDS), particularly in IoT environments, but suffer performance degradation due to distribution drift and lack robustness against realistic adversarial attacks. Current robustness evaluations often rely on unrealistic synthetic perturbations and lack demonstrations on systematic analysis of different kinds of adversarial attack, which encompass both black-box and white-box scenarios. This work proposes a novel approach to enhance GNN robustness and generalization by employing Large Language Models (LLMs) in an agentic pipeline as simulated cybersecurity expert agents. These agents scrutinize graph structures derived from network flow data, identifying and potentially mitigating suspicious or adversarially perturbed elements before GNN processing. Our experiments, using a framework designed for realistic evaluation and testing with a variety of adversarial attacks including a dataset collected from physical testbed experiments, demonstrate that integrating LLM analysis can significantly improve the resilience of GNN-based NIDS against challenges, showcasing the potential of LLM agent as a complementary layer in intrusion detection architectures.

http://arxiv.org/abs/2506.20576
Vulnerability Disclosure through Adaptive Black-Box Adversarial Attacks on NIDS. (97%)
Sabrine Ennaji; Elhadj Benkhelifa; Luigi V. Mancini
Adversarial attacks, wherein slight inputs are carefully crafted to mislead intelligent models, have attracted increasing attention. However, a critical gap persists between theoretical advancements and practical application, particularly in structured data like network traffic, where interdependent features complicate effective adversarial manipulations. Moreover, ambiguity in current approaches restricts reproducibility and limits progress in this field. Hence, existing defenses often fail to handle evolving adversarial attacks. This paper proposes a novel approach for black-box adversarial attacks, that addresses these limitations. Unlike prior work, which often assumes system access or relies on repeated probing, our method strictly respect black-box constraints, reducing interaction to avoid detection and better reflect real-world scenarios. We present an adaptive feature selection strategy using change-point detection and causality analysis to identify and target sensitive features to perturbations. This lightweight design ensures low computational cost and high deployability. Our comprehensive experiments show the attack's effectiveness in evading detection with minimal interaction, enhancing its adaptability and applicability in real-world scenarios. By advancing the understanding of adversarial attacks in network traffic, this work lays a foundation for developing robust defenses.

http://arxiv.org/abs/2506.22506
SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning. (68%)
Momin Ahmad Khan; Yasra Chandio; Fatima Muhammad Anwar
Federated Prompt Learning has emerged as a communication-efficient and privacy-preserving paradigm for adapting large vision-language models like CLIP across decentralized clients. However, the security implications of this setup remain underexplored. In this work, we present the first study of backdoor attacks in Federated Prompt Learning. We show that when malicious clients inject visually imperceptible, learnable noise triggers into input images, the global prompt learner becomes vulnerable to targeted misclassification while still maintaining high accuracy on clean inputs. Motivated by this vulnerability, we propose SABRE-FL, a lightweight, modular defense that filters poisoned prompt updates using an embedding-space anomaly detector trained offline on out-of-distribution data. SABRE-FL requires no access to raw client data or labels and generalizes across diverse datasets. We show, both theoretically and empirically, that malicious clients can be reliably identified and filtered using an embedding-based detector. Across five diverse datasets and four baseline defenses, SABRE-FL outperforms all baselines by significantly reducing backdoor accuracy while preserving clean accuracy, demonstrating strong empirical performance and underscoring the need for robust prompt learning in future federated systems.

http://arxiv.org/abs/2506.20705
On Convolutions, Intrinsic Dimension, and Diffusion Models. (50%)
Kin Kwan Leung; Rasa Hosseinzadeh; Gabriel Loaiza-Ganem
The manifold hypothesis asserts that data of interest in high-dimensional ambient spaces, such as image data, lies on unknown low-dimensional submanifolds. Diffusion models (DMs) -- which operate by convolving data with progressively larger amounts of Gaussian noise and then learning to revert this process -- have risen to prominence as the most performant generative models, and are known to be able to learn distributions with low-dimensional support. For a given datum in one of these submanifolds, we should thus intuitively expect DMs to have implicitly learned its corresponding local intrinsic dimension (LID), i.e. the dimension of the submanifold it belongs to. Kamkari et al. (2024b) recently showed that this is indeed the case by linking this LID to the rate of change of the log marginal densities of the DM with respect to the amount of added noise, resulting in an LID estimator known as FLIPD. LID estimators such as FLIPD have a plethora of uses, among others they quantify the complexity of a given datum, and can be used to detect outliers, adversarial examples and AI-generated text. FLIPD achieves state-of-the-art performance at LID estimation, yet its theoretical underpinnings are incomplete since Kamkari et al. (2024b) only proved its correctness under the highly unrealistic assumption of affine submanifolds. In this work we bridge this gap by formally proving the correctness of FLIPD under realistic assumptions. Additionally, we show that an analogous result holds when Gaussian convolutions are replaced with uniform ones, and discuss the relevance of this result.

http://arxiv.org/abs/2506.20102
Autonomous Cyber Resilience via a Co-Evolutionary Arms Race within a Fortified Digital Twin Sandbox. (11%)
Malikussaid; Sutiyo
The convergence of IT and OT has created hyper-connected ICS, exposing critical infrastructure to a new class of adaptive, intelligent adversaries that render static defenses obsolete. Existing security paradigms often fail to address a foundational "Trinity of Trust," comprising the fidelity of the system model, the integrity of synchronizing data, and the resilience of the analytical engine against sophisticated evasion. This paper introduces the ARC framework, a method for achieving analytical resilience through an autonomous, closed-loop hardening process. ARC establishes a perpetual co-evolutionary arms race within the high-fidelity sandbox of a F-SCDT. A DRL agent, the "Red Agent," is formalized and incentivized to autonomously discover stealthy, physically-plausible attack paths that maximize process disruption while evading detection. Concurrently, an ensemble-based "Blue Agent" defender is continuously hardened via adversarial training against the evolving threats discovered by its adversary. This co-evolutionary dynamic forces both agents to become progressively more sophisticated, enabling the system to autonomously probe and patch its own vulnerabilities. Experimental validation on both the TEP and the SWaT testbeds demonstrates the framework's superior performance. A comprehensive ablation study, supported by extensive visualizations including ROC curves and SHAP plots, reveals that the co-evolutionary process itself is responsible for a significant performance increase in detecting novel attacks. By integrating XAI to ensure operator trust and proposing a scalable F-ARC architecture, this work presents ARC not merely as an improvement, but as a necessary paradigm shift toward dynamic, self-improving security for the future of critical infrastructure.

http://arxiv.org/abs/2506.20651
Hear No Evil: Detecting Gradient Leakage by Malicious Servers in Federated Learning. (8%)
Fei Wang; Baochun Li
Recent work has shown that gradient updates in federated learning (FL) can unintentionally reveal sensitive information about a client's local data. This risk becomes significantly greater when a malicious server manipulates the global model to provoke information-rich updates from clients. In this paper, we adopt a defender's perspective to provide the first comprehensive analysis of malicious gradient leakage attacks and the model manipulation techniques that enable them. Our investigation reveals a core trade-off: these attacks cannot be both highly effective in reconstructing private data and sufficiently stealthy to evade detection -- especially in realistic FL settings that incorporate common normalization techniques and federated averaging.
  Building on this insight, we argue that malicious gradient leakage attacks, while theoretically concerning, are inherently limited in practice and often detectable through basic monitoring. As a complementary contribution, we propose a simple, lightweight, and broadly applicable client-side detection mechanism that flags suspicious model updates before local training begins, despite the fact that such detection may not be strictly necessary in realistic FL settings. This mechanism further underscores the feasibility of defending against these attacks with minimal overhead, offering a deployable safeguard for privacy-conscious federated learning systems.

http://arxiv.org/abs/2411.13047
Bounding-box Watermarking: Defense against Model Extraction Attacks on Object Detectors. (5%)
Satoru Koda; Ikuya Morikawa
Deep neural networks (DNNs) deployed in a cloud often allow users to query models via the APIs. However, these APIs expose the models to model extraction attacks (MEAs). In this attack, the attacker attempts to duplicate the target model by abusing the responses from the API. Backdoor-based DNN watermarking is known as a promising defense against MEAs, wherein the defender injects a backdoor into extracted models via API responses. The backdoor is used as a watermark of the model; if a suspicious model has the watermark (i.e., backdoor), it is verified as an extracted model. This work focuses on object detection (OD) models. Existing backdoor attacks on OD models are not applicable for model watermarking as the defense against MEAs on a realistic threat model. Our proposed approach involves inserting a backdoor into extracted models via APIs by stealthily modifying the bounding-boxes (BBs) of objects detected in queries while keeping the OD capability. In our experiments on three OD datasets, the proposed approach succeeded in identifying the extracted models with 100% accuracy in a wide variety of experimental scenarios.

http://arxiv.org/abs/2506.20494
Multimodal Representation Learning and Fusion. (4%)
Qihang Jin; Enze Ge; Yuhang Xie; Hongying Luo; Junhao Song; Ziqian Bi; Chia Xin Liang; Jibin Guan; Joe Yeong; Junfeng Hao
Multi-modal learning is a fast growing area in artificial intelligence. It tries to help machines understand complex things by combining information from different sources, like images, text, and audio. By using the strengths of each modality, multi-modal learning allows AI systems to build stronger and richer internal representations. These help machines better interpretation, reasoning, and making decisions in real-life situations. This field includes core techniques such as representation learning (to get shared features from different data types), alignment methods (to match information across modalities), and fusion strategies (to combine them by deep learning models). Although there has been good progress, some major problems still remain. Like dealing with different data formats, missing or incomplete inputs, and defending against adversarial attacks. Researchers now are exploring new methods, such as unsupervised or semi-supervised learning, AutoML tools, to make models more efficient and easier to scale. And also more attention on designing better evaluation metrics or building shared benchmarks, make it easier to compare model performance across tasks and domains. As the field continues to grow, multi-modal learning is expected to improve many areas: computer vision, natural language processing, speech recognition, and healthcare. In the future, it may help to build AI systems that can understand the world in a way more like humans, flexible, context aware, and able to deal with real-world complexity.

http://arxiv.org/abs/2502.01633
Adversarial Reasoning at Jailbreaking Time. (3%)
Mahdi Sabbaghi; Paul Kassianik; George Pappas; Yaron Singer; Amin Karbasi; Hamed Hassani
As large language models (LLMs) are becoming more capable and widespread, the study of their failure cases is becoming increasingly important. Recent advances in standardizing, measuring, and scaling test-time compute suggest new methodologies for optimizing models to achieve high performance on hard tasks. In this paper, we apply these advances to the task of model jailbreaking: eliciting harmful responses from aligned LLMs. We develop an adversarial reasoning approach to automatic jailbreaking that leverages a loss signal to guide the test-time compute, achieving SOTA attack success rates against many aligned LLMs, even those that aim to trade inference-time compute for adversarial robustness. Our approach introduces a new paradigm in understanding LLM vulnerabilities, laying the foundation for the development of more robust and trustworthy AI systems.

http://arxiv.org/abs/2507.22063
RedCoder: Automated Multi-Turn Red Teaming for Code LLMs. (1%)
Wenjie Jacky Mo; Qin Liu; Xiaofei Wen; Dongwon Jung; Hadi Askari; Wenxuan Zhou; Zhe Zhao; Muhao Chen
Large Language Models (LLMs) for code generation (i.e., Code LLMs) have demonstrated impressive capabilities in AI-assisted software development and testing. However, recent studies have shown that these models are prone to generating vulnerable or even malicious code under adversarial settings. Existing red-teaming approaches rely on extensive human effort, limiting their scalability and practicality, and generally overlook the interactive nature of real-world AI-assisted programming, which often unfolds over multiple turns. To bridge these gaps, we present RedCoder, a red-teaming agent that engages victim models in multi-turn conversation to elicit vulnerable code. The pipeline to construct RedCoder begins with a multi-agent gaming process that simulates adversarial interactions, yielding a set of prototype conversations and an arsenal of reusable attack strategies. We then fine-tune an LLM on these prototype conversations to serve as the backbone of RedCoder. Once deployed, RedCoder autonomously engages Code LLMs in multi-turn conversations, dynamically retrieving relevant strategies from the arsenal to steer the dialogue toward vulnerability-inducing outputs. Experiments across multiple Code LLMs show that our approach outperforms prior single-turn and multi-turn red-team methods in inducing vulnerabilities in code generation, offering a scalable and effective tool for evaluating the security boundaries of modern code-generation systems.

http://arxiv.org/abs/2506.19302
Adversarial Attacks on Deep Learning-Based False Data Injection Detection in Differential Relays. (99%)
Ahmad Mohammad Saber; Aditi Maheshwari; Amr Youssef; Deepa Kundur
The application of Deep Learning-based Schemes (DLSs) for detecting False Data Injection Attacks (FDIAs) in smart grids has attracted significant attention. This paper demonstrates that adversarial attacks, carefully crafted FDIAs, can evade existing DLSs used for FDIA detection in Line Current Differential Relays (LCDRs). We propose a novel adversarial attack framework, utilizing the Fast Gradient Sign Method, which exploits DLS vulnerabilities by introducing small perturbations to LCDR remote measurements, leading to misclassification of the FDIA as a legitimate fault while also triggering the LCDR to trip. We evaluate the robustness of multiple deep learning models, including multi-layer perceptrons, convolutional neural networks, long short-term memory networks, and residual networks, under adversarial conditions. Our experimental results demonstrate that while these models perform well, they exhibit high degrees of vulnerability to adversarial attacks. For some models, the adversarial attack success rate exceeds 99.7%. To address this threat, we introduce adversarial training as a proactive defense mechanism, significantly enhancing the models' ability to withstand adversarial FDIAs without compromising fault detection accuracy. Our results highlight the significant threat posed by adversarial attacks to DLS-based FDIA detection, underscore the necessity for robust cybersecurity measures in smart grids, and demonstrate the effectiveness of adversarial training in enhancing model robustness against adversarial FDIAs.

http://arxiv.org/abs/2405.14781
Unified Neural Backdoor Removal with Only Few Clean Samples through Unlearning and Relearning. (76%)
Nay Myat Min; Long H. Pham; Jun Sun
Deep neural networks have achieved remarkable success across various applications; however, their vulnerability to backdoor attacks poses severe security risks -- especially in situations where only a limited set of clean samples is available for defense. In this work, we address this critical challenge by proposing ULRL (UnLearn and ReLearn for backdoor removal), a novel two-phase approach for comprehensive backdoor removal. Our method first employs an unlearning phase, in which the network's loss is intentionally maximized on a small clean dataset to expose neurons that are excessively sensitive to backdoor triggers. Subsequently, in the relearning phase, these suspicious neurons are recalibrated using targeted reinitialization and cosine similarity regularization, effectively neutralizing backdoor influences while preserving the model's performance on benign data. Extensive experiments with 12 backdoor types on multiple datasets (CIFAR-10, CIFAR-100, GTSRB, and Tiny-ImageNet) and architectures (PreAct-ResNet18, VGG19-BN, and ViT-B-16) demonstrate that ULRL significantly reduces the attack success rate without compromising clean accuracy -- even when only 1% of clean data is used for defense.

http://arxiv.org/abs/2408.00523
Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models. (73%)
Yingkai Dong; Xiangtao Meng; Ning Yu; Zheng Li; Shanqing Guo
Text-to-image (T2I) generative models have revolutionized content creation by transforming textual descriptions into high-quality images. However, these models are vulnerable to jailbreaking attacks, where carefully crafted prompts bypass safety mechanisms to produce unsafe content. While researchers have developed various jailbreak attacks to expose this risk, these methods face significant limitations, including impractical access requirements, easily detectable unnatural prompts, restricted search spaces, and high query demands on the target system. In this paper, we propose JailFuzzer, a novel fuzzing framework driven by large language model (LLM) agents, designed to efficiently generate natural and semantically meaningful jailbreak prompts in a black-box setting. Specifically, JailFuzzer employs fuzz-testing principles with three components: a seed pool for initial and jailbreak prompts, a guided mutation engine for generating meaningful variations, and an oracle function to evaluate jailbreak success. Furthermore, we construct the guided mutation engine and oracle function by LLM-based agents, which further ensures efficiency and adaptability in black-box settings. Extensive experiments demonstrate that JailFuzzer has significant advantages in jailbreaking T2I models. It generates natural and semantically coherent prompts, reducing the likelihood of detection by traditional defenses. Additionally, it achieves a high success rate in jailbreak attacks with minimal query overhead, outperforming existing methods across all key metrics. This study underscores the need for stronger safety mechanisms in generative models and provides a foundation for future research on defending against sophisticated jailbreaking attacks. JailFuzzer is open-source and available at this repository: https://github.com/YingkaiD/JailFuzzer.

http://arxiv.org/abs/2506.19889
Retrieval-Confused Generation is a Good Defender for Privacy Violation Attack of Large Language Models. (67%)
Wanli Peng; Xin Chen; Hang Fu; XinYu He; Xue Yiming; Juan Wen
Recent advances in large language models (LLMs) have made a profound impact on our society and also raised new security concerns. Particularly, due to the remarkable inference ability of LLMs, the privacy violation attack (PVA), revealed by Staab et al., introduces serious personal privacy issues. Existing defense methods mainly leverage LLMs to anonymize the input query, which requires costly inference time and cannot gain satisfactory defense performance. Moreover, directly rejecting the PVA query seems like an effective defense method, while the defense method is exposed, promoting the evolution of PVA. In this paper, we propose a novel defense paradigm based on retrieval-confused generation (RCG) of LLMs, which can efficiently and covertly defend the PVA. We first design a paraphrasing prompt to induce the LLM to rewrite the "user comments" of the attack query to construct a disturbed database. Then, we propose the most irrelevant retrieval strategy to retrieve the desired user data from the disturbed database. Finally, the "data comments" are replaced with the retrieved user data to form a defended query, leading to responding to the adversary with some wrong personal attributes, i.e., the attack fails. Extensive experiments are conducted on two datasets and eight popular LLMs to comprehensively evaluate the feasibility and the superiority of the proposed defense method.

http://arxiv.org/abs/2501.16490
Towards Robust Stability Prediction in Smart Grids: GAN-based Approach under Data Constraints and Adversarial Challenges. (62%)
Emad Efatinasab; Alessandro Brighente; Denis Donadel; Mauro Conti; Mirco Rampazzo
Smart grids are crucial for meeting rising energy demands driven by global population growth and urbanization. By integrating renewable energy sources, they enhance efficiency, reliability, and sustainability. However, ensuring their availability and security requires advanced operational control and safety measures. Although artificial intelligence and machine learning can help assess grid stability, challenges such as data scarcity and cybersecurity threats, particularly adversarial attacks, remain. Data scarcity is a major issue, as obtaining real-world instances of grid instability requires significant expertise, resources, and time. Yet, these instances are critical for testing new research advancements and security mitigations. This paper introduces a novel framework for detecting instability in smart grids using only stable data. It employs a Generative Adversarial Network (GAN) where the generator is designed not to produce near-realistic data but instead to generate Out-Of-Distribution (OOD) samples with respect to the stable class. These OOD samples represent unstable behavior, anomalies, or disturbances that deviate from the stable data distribution. By training exclusively on stable data and exposing the discriminator to OOD samples, our framework learns a robust decision boundary to distinguish stable conditions from any unstable behavior, without requiring unstable data during training. Furthermore, we incorporate an adversarial training layer to enhance resilience against attacks. Evaluated on a real-world dataset, our solution achieves up to 98.1\% accuracy in predicting grid stability and 98.9\% in detecting adversarial attacks. Implemented on a single-board computer, it enables real-time decision-making with an average response time of under 7ms.

http://arxiv.org/abs/2506.19464
Assessing Risk of Stealing Proprietary Models for Medical Imaging Tasks. (22%)
Ankita Raj; Harsh Swaika; Deepankar Varma; Chetan Arora
The success of deep learning in medical imaging applications has led several companies to deploy proprietary models in diagnostic workflows, offering monetized services. Even though model weights are hidden to protect the intellectual property of the service provider, these models are exposed to model stealing (MS) attacks, where adversaries can clone the model's functionality by querying it with a proxy dataset and training a thief model on the acquired predictions. While extensively studied on general vision tasks, the susceptibility of medical imaging models to MS attacks remains inadequately explored. This paper investigates the vulnerability of black-box medical imaging models to MS attacks under realistic conditions where the adversary lacks access to the victim model's training data and operates with limited query budgets. We demonstrate that adversaries can effectively execute MS attacks by using publicly available datasets. To further enhance MS capabilities with limited query budgets, we propose a two-step model stealing approach termed QueryWise. This method capitalizes on unlabeled data obtained from a proxy distribution to train the thief model without incurring additional queries. Evaluation on two medical imaging models for Gallbladder Cancer and COVID-19 classification substantiates the effectiveness of the proposed attack. The source code is available at https://github.com/rajankita/QueryWise.

http://arxiv.org/abs/2507.00724
Holmes: Towards Effective and Harmless Model Ownership Verification to Personalized Large Vision Models via Decoupling Common Features. (11%)
Linghui Zhu; Yiming Li; Haiqin Weng; Yan Liu; Tianwei Zhang; Shu-Tao Xia; Zhi Wang
Large vision models achieve remarkable performance in various downstream tasks, primarily by personalizing pre-trained models through fine-tuning with private and valuable local data, which makes the personalized model a valuable intellectual property for its owner. Similar to the era of traditional DNNs, model stealing attacks also pose significant risks to these personalized models. However, in this paper, we reveal that most existing defense methods (developed for traditional DNNs), typically designed for models trained from scratch, either introduce additional security risks, are prone to misjudgment, or are even ineffective for fine-tuned models. To alleviate these problems, this paper proposes a harmless model ownership verification method for personalized models by decoupling similar common features. In general, our method consists of three main stages. In the first stage, we create shadow models that retain common features of the victim model while disrupting dataset-specific features. We represent the dataset-specific features of the victim model by the output differences between the shadow and victim models. After that, a meta-classifier is trained to identify stolen models by determining whether suspicious models contain the dataset-specific features of the victim. In the third stage, we conduct model ownership verification by hypothesis test to mitigate randomness and enhance robustness. Extensive experiments on benchmark datasets verify the effectiveness of the proposed method in detecting different types of model stealing simultaneously.

http://arxiv.org/abs/2506.19250
Robust Behavior Cloning Via Global Lipschitz Regularization. (5%)
Shili Wu; Yizhao Jin; Puhua Niu; Aniruddha Datta; Sean B. Andersson
Behavior Cloning (BC) is an effective imitation learning technique and has even been adopted in some safety-critical domains such as autonomous vehicles. BC trains a policy to mimic the behavior of an expert by using a dataset composed of only state-action pairs demonstrated by the expert, without any additional interaction with the environment. However, During deployment, the policy observations may contain measurement errors or adversarial disturbances. Since the observations may deviate from the true states, they can mislead the agent into making sub-optimal actions. In this work, we use a global Lipschitz regularization approach to enhance the robustness of the learned policy network. We then show that the resulting global Lipschitz property provides a robustness certificate to the policy with respect to different bounded norm perturbations. Then, we propose a way to construct a Lipschitz neural network that ensures the policy robustness. We empirically validate our theory across various environments in Gymnasium. Keywords: Robust Reinforcement Learning; Behavior Cloning; Lipschitz Neural Network

http://arxiv.org/abs/2506.19260
Network Structures as an Attack Surface: Topology-Based Privacy Leakage in Federated Learning. (2%)
Murtaza Rangwala; Richard O. Sinnott; Rajkumar Buyya
Federated learning systems increasingly rely on diverse network topologies to address scalability and organizational constraints. While existing privacy research focuses on gradient-based attacks, the privacy implications of network topology knowledge remain critically understudied. We conduct the first comprehensive analysis of topology-based privacy leakage across realistic adversarial knowledge scenarios, demonstrating that adversaries with varying degrees of structural knowledge can infer sensitive data distribution patterns even under strong differential privacy guarantees. Through systematic evaluation of 4,720 attack instances, we analyze six distinct adversarial knowledge scenarios: complete topology knowledge and five partial knowledge configurations reflecting real-world deployment constraints. We propose three complementary attack vectors: communication pattern analysis, parameter magnitude profiling, and structural position correlation, achieving success rates of 84.1%, 65.0%, and 47.2% under complete knowledge conditions. Critically, we find that 80% of realistic partial knowledge scenarios maintain attack effectiveness above security thresholds, with certain partial knowledge configurations achieving performance superior to the baseline complete knowledge scenario. To address these vulnerabilities, we propose and empirically validate structural noise injection as a complementary defense mechanism across 808 configurations, demonstrating up to 51.4% additional attack reduction when properly layered with existing privacy techniques. These results establish that network topology represents a fundamental privacy vulnerability in federated learning systems while providing practical pathways for mitigation through topology-aware defense mechanisms.

http://arxiv.org/abs/2506.19680
Model Guidance via Robust Feature Attribution. (1%)
Mihnea Ghitu; Matthew Wicker; Vihari Piratla
Controlling the patterns a model learns is essential to preventing reliance on irrelevant or misleading features. Such reliance on irrelevant features, often called shortcut features, has been observed across domains, including medical imaging and natural language processing, where it may lead to real-world harms. A common mitigation strategy leverages annotations (provided by humans or machines) indicating which features are relevant or irrelevant. These annotations are compared to model explanations, typically in the form of feature salience, and used to guide the loss function during training. Unfortunately, recent works have demonstrated that feature salience methods are unreliable and therefore offer a poor signal to optimize. In this work, we propose a simplified objective that simultaneously optimizes for explanation robustness and mitigation of shortcut learning. Unlike prior objectives with similar aims, we demonstrate theoretically why our approach ought to be more effective. Across a comprehensive series of experiments, we show that our approach consistently reduces test-time misclassifications by 20% compared to state-of-the-art methods. We also extend prior experimental settings to include natural language processing tasks. Additionally, we conduct novel ablations that yield practical insights, including the relative importance of annotation quality over quantity. Code for our method and experiments is available at: https://github.com/Mihneaghitu/ModelGuidanceViaRobustFeatureAttribution.

http://arxiv.org/abs/2506.18870
Amplifying Machine Learning Attacks Through Strategic Compositions. (99%)
Yugeng Liu; Zheng Li; Hai Huang; Michael Backes; Yang Zhang
Machine learning (ML) models are proving to be vulnerable to a variety of attacks that allow the adversary to learn sensitive information, cause mispredictions, and more. While these attacks have been extensively studied, current research predominantly focuses on analyzing each attack type individually. In practice, however, adversaries may employ multiple attack strategies simultaneously rather than relying on a single approach. This prompts a crucial yet underexplored question: When the adversary has multiple attacks at their disposal, are they able to mount or amplify the effect of one attack with another? In this paper, we take the first step in studying the strategic interactions among different attacks, which we define as attack compositions. Specifically, we focus on four well-studied attacks during the model's inference phase: adversarial examples, attribute inference, membership inference, and property inference. To facilitate the study of their interactions, we propose a taxonomy based on three stages of the attack pipeline: preparation, execution, and evaluation. Using this taxonomy, we identify four effective attack compositions, such as property inference assisting attribute inference at its preparation level and adversarial examples assisting property inference at its execution level. We conduct extensive experiments on the attack compositions using three ML model architectures and three benchmark image datasets. Empirical results demonstrate the effectiveness of these four attack compositions. We implement and release a modular reusable toolkit, COAT. Arguably, our work serves as a call for researchers and practitioners to consider advanced adversarial settings involving multiple attack strategies, aiming to strengthen the security and robustness of AI systems.

http://arxiv.org/abs/2506.18516
DUMB and DUMBer: Is Adversarial Training Worth It in the Real World? (99%)
Francesco Marchiori; Marco Alecci; Luca Pajola; Mauro Conti
Adversarial examples are small and often imperceptible perturbations crafted to fool machine learning models. These attacks seriously threaten the reliability of deep neural networks, especially in security-sensitive domains. Evasion attacks, a form of adversarial attack where input is modified at test time to cause misclassification, are particularly insidious due to their transferability: adversarial examples crafted against one model often fool other models as well. This property, known as adversarial transferability, complicates defense strategies since it enables black-box attacks to succeed without direct access to the victim model. While adversarial training is one of the most widely adopted defense mechanisms, its effectiveness is typically evaluated on a narrow and homogeneous population of models. This limitation hinders the generalizability of empirical findings and restricts practical adoption.
  In this work, we introduce DUMBer, an attack framework built on the foundation of the DUMB (Dataset soUrces, Model architecture, and Balance) methodology, to systematically evaluate the resilience of adversarially trained models. Our testbed spans multiple adversarial training techniques evaluated across three diverse computer vision tasks, using a heterogeneous population of uniquely trained models to reflect real-world deployment variability. Our experimental pipeline comprises over 130k evaluations spanning 13 state-of-the-art attack algorithms, allowing us to capture nuanced behaviors of adversarial training under varying threat models and dataset conditions. Our findings offer practical, actionable insights for AI practitioners, identifying which defenses are most effective based on the model, dataset, and attacker setup.

http://arxiv.org/abs/2506.18304
Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policies. (98%)
Junchao Fan; Xuyang Lei; Xiaolin Chang
Deep reinforcement learning (DRL) has emerged as a promising paradigm for autonomous driving. However, despite their advanced capabilities, DRL-based policies remain highly vulnerable to adversarial attacks, posing serious safety risks in real-world deployments. Investigating such attacks is crucial for revealing policy vulnerabilities and guiding the development of more robust autonomous systems. While prior attack methods have made notable progress, they still face several challenges: 1) they often rely on high-frequency attacks, yet critical attack opportunities are typically context-dependent and temporally sparse, resulting in inefficient attack patterns; 2) restricting attack frequency can improve efficiency but often results in unstable training due to the adversary's limited exploration. To address these challenges, we propose an adaptive expert-guided adversarial attack method that enhances both the stability and efficiency of attack policy training. Our method first derives an expert policy from successful attack demonstrations using imitation learning, strengthened by an ensemble Mixture-of-Experts architecture for robust generalization across scenarios. This expert policy then guides a DRL-based adversary through a KL-divergence regularization term. Due to the diversity of scenarios, expert policies may be imperfect. To address this, we further introduce a performance-aware annealing strategy that gradually reduces reliance on the expert as the adversary improves. Extensive experiments demonstrate that our method achieves outperforms existing approaches in terms of collision rate, attack efficiency, and training stability, especially in cases where the expert policy is sub-optimal.

http://arxiv.org/abs/2506.18591
SpaNN: Detecting Multiple Adversarial Patches on CNNs by Spanning Saliency Thresholds. (98%)
Mauricio Byrd Victorica; György Dán; Henrik Sandberg
State-of-the-art convolutional neural network models for object detection and image classification are vulnerable to physically realizable adversarial perturbations, such as patch attacks. Existing defenses have focused, implicitly or explicitly, on single-patch attacks, leaving their sensitivity to the number of patches as an open question or rendering them computationally infeasible or inefficient against attacks consisting of multiple patches in the worst cases. In this work, we propose SpaNN, an attack detector whose computational complexity is independent of the expected number of adversarial patches. The key novelty of the proposed detector is that it builds an ensemble of binarized feature maps by applying a set of saliency thresholds to the neural activations of the first convolutional layer of the victim model. It then performs clustering on the ensemble and uses the cluster features as the input to a classifier for attack detection. Contrary to existing detectors, SpaNN does not rely on a fixed saliency threshold for identifying adversarial regions, which makes it robust against white box adversarial attacks. We evaluate SpaNN on four widely used data sets for object detection and classification, and our results show that SpaNN outperforms state-of-the-art defenses by up to 11 and 27 percentage points in the case of object detection and the case of image classification, respectively. Our code is available at https://github.com/gerkbyrd/SpaNN.

http://arxiv.org/abs/2506.18543
Security Assessment of DeepSeek and GPT Series Models against Jailbreak Attacks. (22%)
Xiaodong Wu; Xiangman Li; Jianbing Ni
The widespread deployment of large language models (LLMs) has raised critical concerns over their vulnerability to jailbreak attacks, i.e., adversarial prompts that bypass alignment mechanisms and elicit harmful or policy-violating outputs. While proprietary models like GPT-4 have undergone extensive evaluation, the robustness of emerging open-source alternatives such as DeepSeek remains largely underexplored, despite their growing adoption in real-world applications. In this paper, we present the first systematic jailbreak evaluation of DeepSeek-series models, comparing them with GPT-3.5 and GPT-4 using the HarmBench benchmark. We evaluate seven representative attack strategies across 510 harmful behaviors categorized by both function and semantic domain. Our analysis reveals that DeepSeek's Mixture-of-Experts (MoE) architecture introduces routing sparsity that offers selective robustness against optimization-based attacks such as TAP-T, but leads to significantly higher vulnerability under prompt-based and manually engineered attacks. In contrast, GPT-4 Turbo demonstrates stronger and more consistent safety alignment across diverse behaviors, likely due to its dense Transformer design and reinforcement learning from human feedback. Fine-grained behavioral analysis and case studies further show that DeepSeek often routes adversarial prompts to under-aligned expert modules, resulting in inconsistent refusal behaviors. These findings highlight a fundamental trade-off between architectural efficiency and alignment generalization, emphasizing the need for targeted safety tuning and modular alignment strategies to ensure secure deployment of open-source LLMs.

http://arxiv.org/abs/2506.19051
NIC-RobustBench: A Comprehensive Open-Source Toolkit for Neural Image Compression and Robustness Analysis. (8%)
Georgii Bychkov; Khaled Abud; Egor Kovalev; Alexander Gushchin; Dmitriy Vatolin; Anastasia Antsiferova
Adversarial robustness of neural networks is an increasingly important area of research, combining studies on computer vision models, large language models (LLMs), and others. With the release of JPEG AI -- the first standard for end-to-end neural image compression (NIC) methods -- the question of evaluating NIC robustness has become critically significant. However, previous research has been limited to a narrow range of codecs and attacks. To address this, we present \textbf{NIC-RobustBench}, the first open-source framework to evaluate NIC robustness and adversarial defenses' efficiency, in addition to comparing Rate-Distortion (RD) performance. The framework includes the largest number of codecs among all known NIC libraries and is easily scalable. The paper demonstrates a comprehensive overview of the NIC-RobustBench framework and employs it to analyze NIC robustness. Our code is available online at https://github.com/msu-video-group/NIC-RobustBench.

http://arxiv.org/abs/2504.16359
VideoMark: A Distortion-Free Robust Watermarking Framework for Video Diffusion Models. (2%)
Xuming Hu; Hanqian Li; Jungang Li; Yu Huang; Aiwei Liu
This work introduces \textbf{VideoMark}, a distortion-free robust watermarking framework for video diffusion models. As diffusion models excel in generating realistic videos, reliable content attribution is increasingly critical. However, existing video watermarking methods often introduce distortion by altering the initial distribution of diffusion variables and are vulnerable to temporal attacks, such as frame deletion, due to variable video lengths. VideoMark addresses these challenges by employing a \textbf{pure pseudorandom initialization} to embed watermarks, avoiding distortion while ensuring uniform noise distribution in the latent space to preserve generation quality. To enhance robustness, we adopt a frame-wise watermarking strategy with pseudorandom error correction (PRC) codes, using a fixed watermark sequence with randomly selected starting indices for each video. For watermark extraction, we propose a Temporal Matching Module (TMM) that leverages edit distance to align decoded messages with the original watermark sequence, ensuring resilience against temporal attacks. Experimental results show that VideoMark achieves higher decoding accuracy than existing methods while maintaining video quality comparable to watermark-free generation. The watermark remains imperceptible to attackers without the secret key, offering superior invisibility compared to other frameworks. VideoMark provides a practical, training-free solution for content attribution in diffusion-based video generation. Code and data are available at \href{https://github.com/KYRIE-LI11/VideoMark}{https://github.com/KYRIE-LI11/VideoMark}{Project Page}.

http://arxiv.org/abs/2506.19109
Enhancing Security in LLM Applications: A Performance Evaluation of Early Detection Systems. (1%)
Valerii Gakh; Hayretdin Bahsi
Prompt injection threatens novel applications that emerge from adapting LLMs for various user tasks. The newly developed LLM-based software applications become more ubiquitous and diverse. However, the threat of prompt injection attacks undermines the security of these systems as the mitigation and defenses against them, proposed so far, are insufficient. We investigated the capabilities of early prompt injection detection systems, focusing specifically on the detection performance of techniques implemented in various open-source solutions. These solutions are supposed to detect certain types of prompt injection attacks, including the prompt leak. In prompt leakage attacks, an attacker maliciously manipulates the LLM into outputting its system instructions, violating the system's confidentiality. Our study presents analyzes of distinct prompt leakage detection techniques, and a comparative analysis of several detection solutions, which implement those techniques. We identify the strengths and weaknesses of these techniques and elaborate on their optimal configuration and usage in high-stake deployments. In one of the first studies on existing prompt leak detection solutions, we compared the performances of LLM Guard, Vigil, and Rebuff. We concluded that the implementations of canary word checks in Vigil and Rebuff were not effective at detecting prompt leak attacks, and we proposed improvements for them. We also found an evasion weakness in Rebuff's secondary model-based technique and proposed a mitigation. Then, the result of the comparison of LLM Guard, Vigil, and Rebuff at their peak performance revealed that Vigil is optimal for cases when minimal false positive rate is required, and Rebuff is the most optimal for average needs.

http://arxiv.org/abs/2506.17874
DRO-Augment Framework: Robustness by Synergizing Wasserstein Distributionally Robust Optimization and Data Augmentation. (98%)
Jiaming Hu; Debarghya Mukherjee; Ioannis Ch. Paschalidis
In many real-world applications, ensuring the robustness and stability of deep neural networks (DNNs) is crucial, particularly for image classification tasks that encounter various input perturbations. While data augmentation techniques have been widely adopted to enhance the resilience of a trained model against such perturbations, there remains significant room for improvement in robustness against corrupted data and adversarial attacks simultaneously. To address this challenge, we introduce DRO-Augment, a novel framework that integrates Wasserstein Distributionally Robust Optimization (W-DRO) with various data augmentation strategies to improve the robustness of the models significantly across a broad spectrum of corruptions. Our method outperforms existing augmentation methods under severe data perturbations and adversarial attack scenarios while maintaining the accuracy on the clean datasets on a range of benchmark datasets, including but not limited to CIFAR-10-C, CIFAR-100-C, MNIST, and Fashion-MNIST. On the theoretical side, we establish novel generalization error bounds for neural networks trained using a computationally efficient, variation-regularized loss function closely related to the W-DRO problem.

http://arxiv.org/abs/2506.19871
An Attack Method for Medical Insurance Claim Fraud Detection based on Generative Adversarial Network. (75%)
Yining Pang; Chenghan Li
Insurance fraud detection represents a pivotal advancement in modern insurance service, providing intelligent and digitalized monitoring to enhance management and prevent fraud. It is crucial for ensuring the security and efficiency of insurance systems. Although AI and machine learning algorithms have demonstrated strong performance in detecting fraudulent claims, the absence of standardized defense mechanisms renders current systems vulnerable to emerging adversarial threats. In this paper, we propose a GAN-based approach to conduct adversarial attacks on fraud detection systems. Our results indicate that an attacker, without knowledge of the training data or internal model details, can generate fraudulent cases that are classified as legitimate with a 99\% attack success rate (ASR). By subtly modifying real insurance records and claims, adversaries can significantly increase the fraud risk, potentially bypassing compromised detection systems. These findings underscore the urgent need to enhance the robustness of insurance fraud detection models against adversarial manipulation, thereby ensuring the stability and reliability of different insurance systems.

http://arxiv.org/abs/2506.09600
Effective Red-Teaming of Policy-Adherent Agents. (22%)
Itay Nakash; George Kour; Koren Lazar; Matan Vetzler; Guy Uziel; Ateret Anaby-Tavor
Task-oriented LLM-based agents are increasingly used in domains with strict policies, such as refund eligibility or cancellation rules. The challenge lies in ensuring that the agent consistently adheres to these rules and policies, appropriately refusing any request that would violate them, while still maintaining a helpful and natural interaction. This calls for the development of tailored design and evaluation methodologies to ensure agent resilience against malicious user behavior. We propose a novel threat model that focuses on adversarial users aiming to exploit policy-adherent agents for personal benefit. To address this, we present CRAFT, a multi-agent red-teaming system that leverages policy-aware persuasive strategies to undermine a policy-adherent agent in a customer-service scenario, outperforming conventional jailbreak methods such as DAN prompts, emotional manipulation, and coercive. Building upon the existing tau-bench benchmark, we introduce tau-break, a complementary benchmark designed to rigorously assess the agent's robustness against manipulative user behavior. Finally, we evaluate several straightforward yet effective defense strategies. While these measures provide some protection, they fall short, highlighting the need for stronger, research-driven safeguards to protect policy-adherent agents from adversarial attacks

http://arxiv.org/abs/2506.17632
Optimization-Free Patch Attack on Stereo Depth Estimation. (93%)
Hangcheng Liu; Xu Kuang; Xingshuo Han; Xingwan Wu; Haoran Ou; Shangwei Guo; Xingyi Huang; Tao Xiang; Tianwei Zhang
Stereo Depth Estimation (SDE) is essential for scene understanding in vision-based systems like autonomous driving. However, recent studies show that SDE models are vulnerable to adversarial attacks, which are often limited to unrealistic settings, e.g., digital perturbations on separate stereo views in static scenes, restricting their real-world applicability. This raises a critical question: how can we design physically realizable, scene-adaptive, and transferable attacks against SDE under realistic constraints?
  To answer this, we make two key contributions. First, we propose a unified attack framework that extends optimization-based techniques to four core stages of stereo matching: feature extraction, cost-volume construction, cost aggregation, and disparity regression. A comprehensive stage-wise evaluation across 9 mainstream SDE models, under constraints like photometric consistency, reveals that optimization-based patches suffer from poor transferability. Interestingly, partially transferable patches suggest that patterns, rather than pixel-level perturbations, may be key to generalizable attacks. Motivated by this, we present PatchHunter, the first optimization-free adversarial patch attack against SDE. PatchHunter formulates patch generation as a reinforcement learning-driven search over a structured space of visual patterns crafted to disrupt SDE assumptions.
  We validate PatchHunter across three levels: the KITTI dataset, the CARLA simulator, and real-world vehicle deployment. PatchHunter not only surpasses optimization-based methods in effectiveness but also achieves significantly better black-box transferability. Even under challenging physical conditions like low light, PatchHunter maintains high attack success (e.g., D1-all > 0.4), whereas optimization-based methods fail.

http://arxiv.org/abs/2506.17621
Exploiting Efficiency Vulnerabilities in Dynamic Deep Learning Systems. (41%)
Ravishka Rathnasuriya; Wei Yang
The growing deployment of deep learning models in real-world environments has intensified the need for efficient inference under strict latency and resource constraints. To meet these demands, dynamic deep learning systems (DDLSs) have emerged, offering input-adaptive computation to optimize runtime efficiency. While these systems succeed in reducing cost, their dynamic nature introduces subtle and underexplored security risks. In particular, input-dependent execution pathways create opportunities for adversaries to degrade efficiency, resulting in excessive latency, energy usage, and potential denial-of-service in time-sensitive deployments. This work investigates the security implications of dynamic behaviors in DDLSs and reveals how current systems expose efficiency vulnerabilities exploitable by adversarial inputs. Through a survey of existing attack strategies, we identify gaps in the coverage of emerging model architectures and limitations in current defense mechanisms. Building on these insights, we propose to examine the feasibility of efficiency attacks on modern DDLSs and develop targeted defenses to preserve robustness under adversarial conditions.

http://arxiv.org/abs/2506.17162
Analyzing PDFs like Binaries: Adversarially Robust PDF Malware Analysis via Intermediate Representation and Language Model. (87%)
Side Liu; Jiang Ming; Guodong Zhou; Xinyi Liu; Jianming Fu; Guojun Peng
Malicious PDF files have emerged as a persistent threat and become a popular attack vector in web-based attacks. While machine learning-based PDF malware classifiers have shown promise, these classifiers are often susceptible to adversarial attacks, undermining their reliability. To address this issue, recent studies have aimed to enhance the robustness of PDF classifiers. Despite these efforts, the feature engineering underlying these studies remains outdated. Consequently, even with the application of cutting-edge machine learning techniques, these approaches fail to fundamentally resolve the issue of feature instability.
  To tackle this, we propose a novel approach for PDF feature extraction and PDF malware detection. We introduce the PDFObj IR (PDF Object Intermediate Representation), an assembly-like language framework for PDF objects, from which we extract semantic features using a pretrained language model. Additionally, we construct an Object Reference Graph to capture structural features, drawing inspiration from program analysis. This dual approach enables us to analyze and detect PDF malware based on both semantic and structural features. Experimental results demonstrate that our proposed classifier achieves strong adversarial robustness while maintaining an exceptionally low false positive rate of only 0.07% on baseline dataset compared to state-of-the-art PDF malware classifiers.

http://arxiv.org/abs/2506.17133
Robust Training with Data Augmentation for Medical Imaging Classification. (83%)
Josué Martínez-Martínez; Olivia Brown; Mostafa Karami; Sheida Nabavi
Deep neural networks are increasingly being used to detect and diagnose medical conditions using medical imaging. Despite their utility, these models are highly vulnerable to adversarial attacks and distribution shifts, which can affect diagnostic reliability and undermine trust among healthcare professionals. In this study, we propose a robust training algorithm with data augmentation (RTDA) to mitigate these vulnerabilities in medical image classification. We benchmark classifier robustness against adversarial perturbations and natural variations of RTDA and six competing baseline techniques, including adversarial training and data augmentation approaches in isolation and combination, using experimental data sets with three different imaging technologies (mammograms, X-rays, and ultrasound). We demonstrate that RTDA achieves superior robustness against adversarial attacks and improved generalization performance in the presence of distribution shift in each image classification task while maintaining high clean accuracy.

http://arxiv.org/abs/2506.17350
CUBA: Controlled Untargeted Backdoor Attack against Deep Neural Networks. (75%)
Yinghao Wu; Liyan Zhang
Backdoor attacks have emerged as a critical security threat against deep neural networks in recent years. The majority of existing backdoor attacks focus on targeted backdoor attacks, where trigger is strongly associated to specific malicious behavior. Various backdoor detection methods depend on this inherent property and shows effective results in identifying and mitigating such targeted attacks. However, a purely untargeted attack in backdoor scenarios is, in some sense, self-weakening, since the target nature is what makes backdoor attacks so powerful. In light of this, we introduce a novel Constrained Untargeted Backdoor Attack (CUBA), which combines the flexibility of untargeted attacks with the intentionality of targeted attacks. The compromised model, when presented with backdoor images, will classify them into random classes within a constrained range of target classes selected by the attacker. This combination of randomness and determinedness enables the proposed untargeted backdoor attack to natively circumvent existing backdoor defense methods. To implement the untargeted backdoor attack under controlled flexibility, we propose to apply logit normalization on cross-entropy loss with flipped one-hot labels. By constraining the logit during training, the compromised model will show a uniform distribution across selected target classes, resulting in controlled untargeted attack. Extensive experiments demonstrate the effectiveness of the proposed CUBA on different datasets.

http://arxiv.org/abs/2506.16760
Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Models. (68%)
Lei Jiang; Zixun Zhang; Zizhou Wang; Xiaobing Sun; Zhen Li; Liangli Zhen; Xiaohua Xu
Large Vision-Language Models (LVLMs) demonstrate exceptional performance across multimodal tasks, yet remain vulnerable to jailbreak attacks that bypass built-in safety mechanisms to elicit restricted content generation. Existing black-box jailbreak methods primarily rely on adversarial textual prompts or image perturbations, yet these approaches are highly detectable by standard content filtering systems and exhibit low query and computational efficiency. In this work, we present Cross-modal Adversarial Multimodal Obfuscation (CAMO), a novel black-box jailbreak attack framework that decomposes malicious prompts into semantically benign visual and textual fragments. By leveraging LVLMs' cross-modal reasoning abilities, CAMO covertly reconstructs harmful instructions through multi-step reasoning, evading conventional detection mechanisms. Our approach supports adjustable reasoning complexity and requires significantly fewer queries than prior attacks, enabling both stealth and efficiency. Comprehensive evaluations conducted on leading LVLMs validate CAMO's effectiveness, showcasing robust performance and strong cross-model transferability. These results underscore significant vulnerabilities in current built-in safety mechanisms, emphasizing an urgent need for advanced, alignment-aware security and safety solutions in vision-language systems.

http://arxiv.org/abs/2501.16029
FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting. (61%)
Zhiyuan Fu; Junfan Chen; Lan Zhang; Ting Yang; Jun Niu; Hongyu Sun; Ruidong Li; Peng Liu; Jice Wang; Fannv He; Yuqing Zhang
Large Language Models (LLMs) are rapidly transforming the landscape of digital content creation. However, the prevalent black-box Application Programming Interface (API) access to many LLMs introduces significant challenges in accountability, governance, and security. LLM fingerprinting, which aims to identify the source model by analyzing statistical and stylistic features of generated text, offers a potential solution. Current progress in this area is hindered by a lack of dedicated datasets and the need for efficient, practical methods that are robust against adversarial manipulations. To address these challenges, we introduce FD-Dataset, a comprehensive bilingual fingerprinting benchmark comprising 90,000 text samples from 20 famous proprietary and open-source LLMs. Furthermore, we present FDLLM, a novel fingerprinting method that leverages parameter-efficient Low-Rank Adaptation (LoRA) to fine-tune a foundation model. This approach enables LoRA to extract deep, persistent features that characterize each source LLM. Through our analysis, we find that LoRA adaptation promotes the aggregation of outputs from the same LLM in representation space while enhancing the separation between different LLMs. This mechanism explains why LoRA proves particularly effective for LLM fingerprinting. Extensive empirical evaluations on FD-Dataset demonstrate FDLLM's superiority, achieving a Macro F1 score 22.1% higher than the strongest baseline. FDLLM also exhibits strong generalization to newly released models, achieving an average accuracy of 95% on unseen models. Notably, FDLLM remains consistently robust under various adversarial attacks, including polishing, translation, and synonym substitution. Experimental results show that FDLLM reduces the average attack success rate from 49.2% (LM-D) to 23.9%.

http://arxiv.org/abs/2506.17040
Stretching Beyond the Obvious: A Gradient-Free Framework to Unveil the Hidden Landscape of Visual Invariance. (56%)
Lorenzo Tausani; Paolo Muratore; Morgan B. Talbot; Giacomo Amerio; Gabriel Kreiman; Davide Zoccolan
Uncovering which features' combinations high-level visual units encode is critical to understand how images are transformed into representations that support recognition. While existing feature visualization approaches typically infer a unit's most exciting images, this is insufficient to reveal the manifold of transformations under which responses remain invariant, which is key to generalization in vision. Here we introduce Stretch-and-Squeeze (SnS), an unbiased, model-agnostic, and gradient-free framework to systematically characterize a unit's invariance landscape and its vulnerability to adversarial perturbations in both biological and artificial visual systems. SnS frames these transformations as bi-objective optimization problems. To probe invariance, SnS seeks image perturbations that maximally alter the representation of a reference stimulus in a given processing stage while preserving unit activation. To probe adversarial sensitivity, SnS seeks perturbations that minimally alter the stimulus while suppressing unit activation. Applied to convolutional neural networks (CNNs), SnS revealed image variations that were further from a reference image in pixel-space than those produced by affine transformations, while more strongly preserving the target unit's response. The discovered invariant images differed dramatically depending on the choice of image representation used for optimization: pixel-level changes primarily affected luminance and contrast, while stretching mid- and late-layer CNN representations altered texture and pose respectively. Notably, the invariant images from robust networks were more recognizable by human subjects than those from standard networks, supporting the higher fidelity of robust CNNs as models of the visual system.

http://arxiv.org/abs/2506.16792
MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuning. (47%)
Muyang Zheng; Yuanzhi Yao; Changting Lin; Rui Wang; Meng Han
Despite efforts to align large language models (LLMs) with societal and moral values, these models remain susceptible to jailbreak attacks--methods designed to elicit harmful responses. Jailbreaking black-box LLMs is considered challenging due to the discrete nature of token inputs, restricted access to the target LLM, and limited query budget. To address the issues above, we propose an effective method for jailbreaking black-box large language Models via Iterative Semantic Tuning, named MIST. MIST enables attackers to iteratively refine prompts that preserve the original semantic intent while inducing harmful content. Specifically, to balance semantic similarity with computational efficiency, MIST incorporates two key strategies: sequential synonym search, and its advanced version--order-determining optimization. Extensive experiments across two open-source models and four closed-source models demonstrate that MIST achieves competitive attack success rates and attack transferability compared with other state-of-the-art white-box and black-box jailbreak methods. Additionally, we conduct experiments on computational efficiency to validate the practical viability of MIST.

http://arxiv.org/abs/2506.16690
DepthVanish: Optimizing Adversarial Interval Structures for Stereo-Depth-Invisible Patches. (47%)
Yun Xing; Yue Cao; Nhat Chung; Jie Zhang; Ivor Tsang; Ming-Ming Cheng; Yang Liu; Lei Ma; Qing Guo
Stereo Depth estimation is a critical task in autonomous driving and robotics, where inaccuracies (such as misidentifying nearby objects as distant) can lead to dangerous situations. Adversarial attacks against stereo depth estimation can help reveal vulnerabilities before deployment. Previous work has shown that repeating optimized textures can effectively mislead stereo depth estimation in digital settings. However, our research reveals that these naively repeated texture structures perform poorly in physical-world implementations, i.e., when deployed as patches, limiting their practical utility for testing stereo depth estimation systems. In this work, for the first time, we discover that introducing regular intervals between repeated textures, creating a striped structure, significantly enhances the patch attack effectiveness. Through extensive experimentation, we analyze how variations of this novel structure influence the performance. Based on these insights, we develop a novel stereo depth attack that jointly optimizes both the striped structure and texture elements. Our generated adversarial patches can be inserted into any scenes and successfully attack state-of-the-art stereo depth estimation methods, i.e., RAFT-Stereo and STTR. Most critically, our patch can also attack commercial RGB-D cameras (Intel RealSense) in real-world conditions, demonstrating their practical relevance for security assessment of stereo systems.

http://arxiv.org/abs/2506.16950
LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models. (4%)
Fanfei Li; Thomas Klein; Wieland Brendel; Robert Geirhos; Roland S. Zimmermann
Out-of-distribution (OOD) robustness is a desired property of computer vision models. Improving model robustness requires high-quality signals from robustness benchmarks to quantify progress. While various benchmark datasets such as ImageNet-C were proposed in the ImageNet era, most ImageNet-C corruption types are no longer OOD relative to today's large, web-scraped datasets, which already contain common corruptions such as blur or JPEG compression artifacts. Consequently, these benchmarks are no longer well-suited for evaluating OOD robustness in the era of web-scale datasets. Indeed, recent models show saturating scores on ImageNet-era OOD benchmarks, indicating that it is unclear whether models trained on web-scale datasets truly become better at OOD generalization or whether they have simply been exposed to the test distortions during training. To address this, we introduce LAION-C as a benchmark alternative for ImageNet-C. LAION-C consists of six novel distortion types specifically designed to be OOD, even for web-scale datasets such as LAION. In a comprehensive evaluation of state-of-the-art models, we find that the LAION-C dataset poses significant challenges to contemporary models, including MLLMs such as Gemini and GPT-4o. We additionally conducted a psychophysical experiment to evaluate the difficulty of our corruptions for human observers, enabling a comparison of models to lab-quality human robustness data. We observe a paradigm shift in OOD generalization: from humans outperforming models, to the best models now matching or outperforming the best human observers.

http://arxiv.org/abs/2506.17047
Navigating the Deep: Signature Extraction on Deep Neural Networks. (2%)
Haolin Liu; Adrien Siproudhis; Samuel Experton; Peter Lorenz; Christina Boura; Thomas Peyrin
Neural network model extraction has emerged in recent years as an important security concern, as adversaries attempt to recover a network's parameters via black-box queries. A key step in this process is signature extraction, which aims to recover the absolute values of the network's weights layer by layer. Prior work, notably by Carlini et al. (2020), introduced a technique inspired by differential cryptanalysis to extract neural network parameters. However, their method suffers from several limitations that restrict its applicability to networks with a few layers only. Later works focused on improving sign extraction, but largely relied on the assumption that signature extraction itself was feasible.
  In this work, we revisit and refine the signature extraction process by systematically identifying and addressing for the first time critical limitations of Carlini et al.'s signature extraction method. These limitations include rank deficiency and noise propagation from deeper layers. To overcome these challenges, we propose efficient algorithmic solutions for each of the identified issues, greatly improving the efficiency of signature extraction. Our approach permits the extraction of much deeper networks than was previously possible. We validate our method through extensive experiments on ReLU-based neural networks, demonstrating significant improvements in extraction depth and accuracy. For instance, our extracted network matches the target network on at least 95% of the input space for each of the eight layers of a neural network trained on the CIFAR-10 dataset, while previous works could barely extract the first three layers. Our results represent a crucial step toward practical attacks on larger and more complex neural network architectures.

http://arxiv.org/abs/2506.16819
Loupe: A Generalizable and Adaptive Framework for Image Forgery Detection. (1%)
Yuchu Jiang; Jiaming Chu; Jian Zhao; Xin Zhang; Xu Yang; Lei Jin; Chi Zhang; Xuelong Li
The proliferation of generative models has raised serious concerns about visual content forgery. Existing deepfake detection methods primarily target either image-level classification or pixel-wise localization. While some achieve high accuracy, they often suffer from limited generalization across manipulation types or rely on complex architectures. In this paper, we propose Loupe, a lightweight yet effective framework for joint deepfake detection and localization. Loupe integrates a patch-aware classifier and a segmentation module with conditional queries, allowing simultaneous global authenticity classification and fine-grained mask prediction. To enhance robustness against distribution shifts of test set, Loupe introduces a pseudo-label-guided test-time adaptation mechanism by leveraging patch-level predictions to supervise the segmentation head. Extensive experiments on the DDL dataset demonstrate that Loupe achieves state-of-the-art performance, securing the first place in the IJCAI 2025 Deepfake Detection and Localization Challenge with an overall score of 0.846. Our results validate the effectiveness of the proposed patch-level fusion and conditional query design in improving both classification accuracy and spatial localization under diverse forgery patterns. The code is available at https://github.com/Kamichanw/Loupe.

http://arxiv.org/abs/2506.17125
Large Language Model Unlearning for Source Code. (1%)
Xue Jiang; Yihong Dong; Zheng Fang; Yingwei Ma; Tangxinyu Wang; Rongyu Cao; Binhua Li; Zhi Jin; Wenpin Jiao; Yongbin Li; Ge Li
LLM4SE has demonstrated significant success, but LLMs' potential memorization of sensitive or outdated training data introduces critical risks to legal compliance, software security, and code quality. LLM unlearning techniques, which can eliminate the influence of undesired data from LLMs in a post-training way, present a promising solution to address these concerns. While recent efforts in LLM unlearning show effectiveness in natural language, their applicability to source code remains underexplored. Our empirical study reveals that existing LLM unlearning approaches, when applied to source code, cause severe model utility degradation, rendering models practically unusable for code generation. In this paper, we propose PROD, a novel unlearning approach that enables LLMs to forget undesired code content while effectively preserving their code generation capabilities. PROD suppresses the probability of forget data in LLMs' output distribution while promoting candidate distributional components, enabling the model to jointly learn to forget specific content and retain its general capabilities. To facilitate this study, we establish a benchmark for code unlearning evaluation, which includes three critical downstream tasks: copyrighted code unlearning, insecure code unlearning, and deprecated API unlearning. Our evaluation demonstrates that PROD achieves superior balance between forget quality and model utility compared to existing unlearning approaches across three downstream tasks, while consistently exhibiting improvements when applied to LLMs of varying series. PROD also exhibits superior robustness against adversarial attacks without generating or exposing the data to be forgotten. The results underscore that our approach not only extends the application boundary of unlearning techniques to source code, but also holds significant implications for advancing reliable code generation.

http://arxiv.org/abs/2506.15988
Adversarial Attacks and Detection in Visual Place Recognition for Safer Robot Navigation. (99%)
Connor Malone; Owen Claxton; Iman Shames; Michael Milford
Stand-alone Visual Place Recognition (VPR) systems have little defence against a well-designed adversarial attack, which can lead to disastrous consequences when deployed for robot navigation. This paper extensively analyzes the effect of four adversarial attacks common in other perception tasks and four novel VPR-specific attacks on VPR localization performance. We then propose how to close the loop between VPR, an Adversarial Attack Detector (AAD), and active navigation decisions by demonstrating the performance benefit of simulated AADs in a novel experiment paradigm -- which we detail for the robotics community to use as a system framework. In the proposed experiment paradigm, we see the addition of AADs across a range of detection accuracies can improve performance over baseline; demonstrating a significant improvement -- such as a ~50% reduction in the mean along-track localization error -- can be achieved with True Positive and False Positive detection rates of only 75% and up to 25% respectively. We examine a variety of metrics including: Along-Track Error, Percentage of Time Attacked, Percentage of Time in an `Unsafe' State, and Longest Continuous Time Under Attack. Expanding further on these results, we provide the first investigation into the efficacy of the Fast Gradient Sign Method (FGSM) adversarial attack for VPR. The analysis in this work highlights the need for AADs in real-world systems for trustworthy navigation, and informs quantitative requirements for system design.

http://arxiv.org/abs/2506.16157
MBA: Multimodal Bidirectional Attack for Referring Expression Segmentation Models. (99%)
Xingbai Chen; Tingchao Fu; Renyang Liu; Wei Zhou; Chao Yi
Referring Expression Segmentation (RES) enables precise object segmentation in images based on natural language descriptions, offering high flexibility and broad applicability in real-world vision tasks. Despite its impressive performance, the robustness of RES models against adversarial examples remains largely unexplored. While prior adversarial attack methods have explored adversarial robustness on conventional segmentation models, they perform poorly when directly applied to RES, failing to expose vulnerabilities in its multimodal structure. Moreover, in practical open-world scenarios, users typically issue multiple, diverse referring expressions to interact with the same image, highlighting the need for adversarial examples that generalize across varied textual inputs. To address these multimodal challenges, we propose a novel adversarial attack strategy termed \textbf{Multimodal Bidirectional Attack}, tailored for RES models. Our method introduces learnable proxy textual embedding perturbation and jointly performs visual-aligned optimization on the image modality and textual-adversarial optimization on the textual modality during attack generation. This dual optimization framework encourages adversarial images to actively adapt to more challenging text embedding during optimization, thereby enhancing their cross-text transferability, which refers to the ability of adversarial examples to remain effective under a variety of unseen or semantically diverse textual inputs. Extensive experiments conducted on multiple RES models and benchmark datasets demonstrate the superior effectiveness of our method compared to existing methods.

http://arxiv.org/abs/2506.11611
KCES: Training-Free Defense for Robust Graph Neural Networks via Kernel Complexity. (98%)
Yaning Jia; Shenyang Deng; Chiyu Ma; Yaoqing Yang; Soroush Vosoughi
Graph Neural Networks (GNNs) have achieved impressive success across a wide range of graph-based tasks, yet they remain highly vulnerable to small, imperceptible perturbations and adversarial attacks. Although numerous defense methods have been proposed to address these vulnerabilities, many rely on heuristic metrics, overfit to specific attack patterns, and suffer from high computational complexity. In this paper, we propose Kernel Complexity-Based Edge Sanitization (KCES), a training-free, model-agnostic defense framework. KCES leverages Graph Kernel Complexity (GKC), a novel metric derived from the graph's Gram matrix that characterizes GNN generalization via its test error bound. Building on GKC, we define a KC score for each edge, measuring the change in GKC when the edge is removed. Edges with high KC scores, typically introduced by adversarial perturbations, are pruned to mitigate their harmful effects, thereby enhancing GNNs' robustness. KCES can also be seamlessly integrated with existing defense strategies as a plug-and-play module without requiring training. Theoretical analysis and extensive experiments demonstrate that KCES consistently enhances GNN robustness, outperforms state-of-the-art baselines, and amplifies the effectiveness of existing defenses, offering a principled and efficient solution for securing GNNs.

http://arxiv.org/abs/2406.03458
Distributional Adversarial Loss. (96%)
Saba Ahmadi; Siddharth Bhandari; Avrim Blum; Chen Dan; Prabhav Jain
We initiate the study of a new notion of adversarial loss which we call distributional adversarial loss. In this notion, we assume for each original example, the allowed adversarial perturbation set is a family of distributions, and the adversarial loss over each example is the maximum loss over all the associated distributions. The goal is to minimize the overall adversarial loss. We show sample complexity bounds in the PAC-learning setting for our notion of adversarial loss. Our notion of adversarial loss contrasts the prior work on robust learning that considers a set of points, not distributions, as the perturbation set of each clean example. As an application of our approach, we show how to unify the two lines of work on randomized smoothing and robust learning in the PAC-learning setting and derive sample complexity bounds for randomized smoothing methods.
  Furthermore, we investigate the role of randomness in achieving robustness against adversarial attacks. We show a general derandomization technique that preserves the extent of a randomized classifier's robustness against adversarial attacks and show its effectiveness empirically.

http://arxiv.org/abs/2503.23804
DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents. (81%)
Shiyi Yang; Zhibo Hu; Xinshu Li; Chen Wang; Tong Yu; Xiwei Xu; Liming Zhu; Lina Yao
Large language model (LLM)-powered agents are increasingly used in recommender systems (RSs) to achieve personalized behavior modeling, where the memory mechanism plays a pivotal role in enabling the agents to autonomously explore, learn and self-evolve from real-world interactions. However, this very mechanism, serving as a contextual repository, inherently exposes an attack surface for potential adversarial manipulations. Despite its central role, the robustness of agentic RSs in the face of such threats remains largely underexplored. Previous works suffer from semantic mismatches or rely on static embeddings or pre-defined prompts, all of which hinder their applicability to systems with dynamic memory states. This challenge is exacerbated by the black-box nature of commercial RSs.
  To tackle the above problems, in this paper, we present the first systematic investigation of memory-based vulnerabilities in LLM-powered recommender agents, revealing their security limitations and guiding efforts to strengthen system resilience and trustworthiness. Specifically, we propose a novel black-box attack framework named DrunkAgent. DrunkAgent crafts semantically meaningful adversarial textual triggers for target item promotions and introduces a series of strategies to maximize the trigger effect by corrupting the memory updates during the interactions. The triggers and strategies are optimized on a surrogate model, enabling DrunkAgent transferable and stealthy. Extensive experiments on real-world datasets across diverse agentic RSs, including collaborative filtering, retrieval augmentation and sequential recommendations, demonstrate the generalizability, transferability and stealthiness of DrunkAgent.

http://arxiv.org/abs/2506.16447
Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models. (69%)
Biao Yi; Tiansheng Huang; Sishuo Chen; Tong Li; Zheli Liu; Zhixuan Chu; Yiming Li
Backdoor unalignment attacks against Large Language Models (LLMs) enable the stealthy compromise of safety alignment using a hidden trigger while evading normal safety auditing. These attacks pose significant threats to the applications of LLMs in the real-world Large Language Model as a Service (LLMaaS) setting, where the deployed model is a fully black-box system that can only interact through text. Furthermore, the sample-dependent nature of the attack target exacerbates the threat. Instead of outputting a fixed label, the backdoored LLM follows the semantics of any malicious command with the hidden trigger, significantly expanding the target space. In this paper, we introduce BEAT, a black-box defense that detects triggered samples during inference to deactivate the backdoor. It is motivated by an intriguing observation (dubbed the probe concatenate effect), where concatenated triggered samples significantly reduce the refusal rate of the backdoored LLM towards a malicious probe, while non-triggered samples have little effect. Specifically, BEAT identifies whether an input is triggered by measuring the degree of distortion in the output distribution of the probe before and after concatenation with the input. Our method addresses the challenges of sample-dependent targets from an opposite perspective. It captures the impact of the trigger on the refusal signal (which is sample-independent) instead of sample-specific successful attack behaviors. It overcomes black-box access limitations by using multiple sampling to approximate the output distribution. Extensive experiments are conducted on various backdoor attacks and LLMs (including the closed-source GPT-3.5-turbo), verifying the effectiveness and efficiency of our defense. Besides, we also preliminarily verify that BEAT can effectively defend against popular jailbreak attacks, as they can be regarded as 'natural backdoors'.

http://arxiv.org/abs/2506.16407
Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attacks. (54%)
Dong Nguyen Tien; Dung D. Le
Visual Document Understanding (VDU) systems have achieved strong performance in information extraction by integrating textual, layout, and visual signals. However, their robustness under realistic adversarial perturbations remains insufficiently explored. We introduce the first unified framework for generating and evaluating multi-modal adversarial attacks on OCR-based VDU models. Our method covers six gradient-based layout attack scenarios, incorporating manipulations of OCR bounding boxes, pixels, and texts across both word and line granularities, with constraints on layout perturbation budget (e.g., IoU >= 0.6) to preserve plausibility.
  Experimental results across four datasets (FUNSD, CORD, SROIE, DocVQA) and six model families demonstrate that line-level attacks and compound perturbations (BBox + Pixel + Text) yield the most severe performance degradation. Projected Gradient Descent (PGD)-based BBox perturbations outperform random-shift baselines in all investigated models. Ablation studies further validate the impact of layout budget, text modification, and adversarial transferability.

http://arxiv.org/abs/2506.01444
Variance-Based Defense Against Blended Backdoor Attacks. (54%)
Sujeevan Aseervatham; Achraf Kerzazi; Younès Bennani
Backdoor attacks represent a subtle yet effective class of cyberattacks targeting AI models, primarily due to their stealthy nature. The model behaves normally on clean data but exhibits malicious behavior only when the attacker embeds a specific trigger into the input. This attack is performed during the training phase, where the adversary corrupts a small subset of the training data by embedding a pattern and modifying the labels to a chosen target. The objective is to make the model associate the pattern with the target label while maintaining normal performance on unaltered data. Several defense mechanisms have been proposed to sanitize training data-sets. However, these methods often rely on the availability of a clean dataset to compute statistical anomalies, which may not always be feasible in real-world scenarios where datasets can be unavailable or compromised. To address this limitation, we propose a novel defense method that trains a model on the given dataset, detects poisoned classes, and extracts the critical part of the attack trigger before identifying the poisoned instances. This approach enhances explainability by explicitly revealing the harmful part of the trigger. The effectiveness of our method is demonstrated through experimental evaluations on well-known image datasets and comparative analysis against three state-of-the-art algorithms: SCAn, ABL, and AGPD.

http://arxiv.org/abs/2506.16078
Probing the Robustness of Large Language Models Safety to Latent Perturbations. (13%)
Tianle Gu; Kexin Huang; Zongqi Wang; Yixu Wang; Jie Li; Yuanqi Yao; Yang Yao; Yujiu Yang; Yan Teng; Yingchun Wang
Safety alignment is a key requirement for building reliable Artificial General Intelligence. Despite significant advances in safety alignment, we observe that minor latent shifts can still trigger unsafe responses in aligned models. We argue that this stems from the shallow nature of existing alignment methods, which focus on surface-level refusal behaviors without sufficiently altering internal representations. Consequently, small shifts in hidden activations can re-trigger harmful behaviors embedded in the latent space. To explore the robustness of safety alignment to latent perturbations, we introduce a probing method that measures the Negative Log-Likelihood of the original response generated by the model. This probe quantifies local sensitivity in the latent space, serving as a diagnostic tool for identifying vulnerable directions. Based on this signal, we construct effective jailbreak trajectories, giving rise to the Activation Steering Attack (ASA). More importantly, these insights offer a principled foundation for improving alignment robustness. To this end, we introduce Layer-wise Adversarial Patch Training~(LAPT), a fine-tuning strategy that inject controlled perturbations into hidden representations during training. Experimental results highlight that LAPT strengthen alignment robustness without compromising general capabilities. Our findings reveal fundamental flaws in current alignment paradigms and call for representation-level training strategies that move beyond surface-level behavior supervision. Codes and results are available at https://github.com/Carol-gutianle/LatentSafety.

http://arxiv.org/abs/2506.16460
Black-Box Privacy Attacks on Shared Representations in Multitask Learning. (13%)
John Abascal; Nicolás Berrios; Alina Oprea; Jonathan Ullman; Adam Smith; Matthew Jagielski
Multitask learning (MTL) has emerged as a powerful paradigm that leverages similarities among multiple learning tasks, each with insufficient samples to train a standalone model, to solve them simultaneously while minimizing data sharing across users and organizations. MTL typically accomplishes this goal by learning a shared representation that captures common structure among the tasks by embedding data from all tasks into a common feature space. Despite being designed to be the smallest unit of shared information necessary to effectively learn patterns across multiple tasks, these shared representations can inadvertently leak sensitive information about the particular tasks they were trained on.
  In this work, we investigate what information is revealed by the shared representations through the lens of inference attacks. Towards this, we propose a novel, black-box task-inference threat model where the adversary, given the embedding vectors produced by querying the shared representation on samples from a particular task, aims to determine whether that task was present when training the shared representation. We develop efficient, purely black-box attacks on machine learning models that exploit the dependencies between embeddings from the same task without requiring shadow models or labeled reference data. We evaluate our attacks across vision and language domains for multiple use cases of MTL and demonstrate that even with access only to fresh task samples rather than training data, a black-box adversary can successfully infer a task's inclusion in training. To complement our experiments, we provide theoretical analysis of a simplified learning setting and show a strict separation between adversaries with training samples and fresh samples from the target task's distribution.

http://arxiv.org/abs/2506.16458
SecureFed: A Two-Phase Framework for Detecting Malicious Clients in Federated Learning. (8%)
Likhitha Annapurna Kavuri; Akshay Mhatre; Akarsh K Nair; Deepti Gupta
Federated Learning (FL) protects data privacy while providing a decentralized method for training models. However, because of the distributed schema, it is susceptible to adversarial clients that could alter results or sabotage model performance. This study presents SecureFed, a two-phase FL framework for identifying and reducing the impact of such attackers. Phase 1 involves collecting model updates from participating clients and applying a dimensionality reduction approach to identify outlier patterns frequently associated with malicious behavior. Temporary models constructed from the client updates are evaluated on synthetic datasets to compute validation losses and support anomaly scoring. The idea of learning zones is presented in Phase 2, where weights are dynamically routed according to their contribution scores and gradient magnitudes. High-value gradient zones are given greater weight in aggregation and contribute more significantly to the global model, while lower-value gradient zones, which may indicate possible adversarial activity, are gradually removed from training. Until the model converges and a strong defense against poisoning attacks is possible, this training cycle continues Based on the experimental findings, SecureFed considerably improves model resilience without compromising model performance.

http://arxiv.org/abs/2506.15755
VLMInferSlow: Evaluating the Efficiency Robustness of Large Vision-Language Models as a Service. (96%)
Xiasi Wang; Tianliang Yao; Simin Chen; Runqi Wang; Lei YE; Kuofeng Gao; Yi Huang; Yuan Yao
Vision-Language Models (VLMs) have demonstrated great potential in real-world applications. While existing research primarily focuses on improving their accuracy, the efficiency remains underexplored. Given the real-time demands of many applications and the high inference overhead of VLMs, efficiency robustness is a critical issue. However, previous studies evaluate efficiency robustness under unrealistic assumptions, requiring access to the model architecture and parameters -- an impractical scenario in ML-as-a-service settings, where VLMs are deployed via inference APIs. To address this gap, we propose VLMInferSlow, a novel approach for evaluating VLM efficiency robustness in a realistic black-box setting. VLMInferSlow incorporates fine-grained efficiency modeling tailored to VLM inference and leverages zero-order optimization to search for adversarial examples. Experimental results show that VLMInferSlow generates adversarial images with imperceptible perturbations, increasing the computational cost by up to 128.47%. We hope this research raises the community's awareness about the efficiency robustness of VLMs.

http://arxiv.org/abs/2506.15499
Pixel-level Certified Explanations via Randomized Smoothing. (82%)
Alaa Anani; Tobias Lorenz; Mario Fritz; Bernt Schiele
Post-hoc attribution methods aim to explain deep learning predictions by highlighting influential input pixels. However, these explanations are highly non-robust: small, imperceptible input perturbations can drastically alter the attribution map while maintaining the same prediction. This vulnerability undermines their trustworthiness and calls for rigorous robustness guarantees of pixel-level attribution scores. We introduce the first certification framework that guarantees pixel-level robustness for any black-box attribution method using randomized smoothing. By sparsifying and smoothing attribution maps, we reformulate the task as a segmentation problem and certify each pixel's importance against $\ell_2$-bounded perturbations. We further propose three evaluation metrics to assess certified robustness, localization, and faithfulness. An extensive evaluation of 12 attribution methods across 5 ImageNet models shows that our certified attributions are robust, interpretable, and faithful, enabling reliable use in downstream tasks. Our code is at https://github.com/AlaaAnani/certified-attributions.

http://arxiv.org/abs/2506.19054
PolyGuard: Massive Multi-Domain Safety Policy-Grounded Guardrail Dataset. (22%)
Mintong Kang; Zhaorun Chen; Chejian Xu; Jiawei Zhang; Chengquan Guo; Minzhou Pan; Ivan Revilla; Yu Sun; Bo Li
As LLMs become widespread across diverse applications, concerns about the security and safety of LLM interactions have intensified. Numerous guardrail models and benchmarks have been developed to ensure LLM content safety. However, existing guardrail benchmarks are often built upon ad hoc risk taxonomies that lack a principled grounding in standardized safety policies, limiting their alignment with real-world operational requirements. Moreover, they tend to overlook domain-specific risks, while the same risk category can carry different implications across different domains. To bridge these gaps, we introduce PolyGuard, the first massive multi-domain safety policy-grounded guardrail dataset. PolyGuard offers: (1) broad domain coverage across eight safety-critical domains, such as finance, law, and codeGen; (2) policy-grounded risk construction based on authentic, domain-specific safety guidelines; (3) diverse interaction formats, encompassing declarative statements, questions, instructions, and multi-turn conversations; (4) advanced benign data curation via detoxification prompting to challenge over-refusal behaviors; and (5) \textbf{attack-enhanced instances} that simulate adversarial inputs designed to bypass guardrails. Based on PolyGuard, we benchmark 19 advanced guardrail models and uncover a series of findings, such as: (1) All models achieve varied F1 scores, with many demonstrating high variance across risk categories, highlighting their limited domain coverage and insufficient handling of domain-specific safety concerns; (2) As models evolve, their coverage of safety risks broadens, but performance on common risk categories may decrease; (3) All models remain vulnerable to optimized adversarial attacks. We believe that \dataset and the unique insights derived from our evaluations will advance the development of policy-aligned and resilient guardrail systems.

http://arxiv.org/abs/2506.15751
Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts. (10%)
Kartik Sharma; Yiqiao Jin; Vineeth Rakesh; Yingtong Dou; Menghai Pan; Mahashweta Das; Srijan Kumar
As large language models (LLMs) are deployed in safety-critical settings, it is essential to ensure that their responses comply with safety standards. Prior research has revealed that LLMs often fail to grasp the notion of safe behaviors, resulting in either unjustified refusals to harmless prompts or the generation of harmful content. While substantial efforts have been made to improve their robustness, existing defenses often rely on costly fine-tuning of model parameters or employ suboptimal heuristic techniques. In this work, we take a novel approach to safeguard LLMs by learning to adapt the system prompts in instruction-tuned LLMs. While LLMs are typically pre-trained to follow a fixed system prompt, we investigate the impact of tailoring the system prompt to each specific user input on the safety of the responses. To this end, we propose $\textbf{Sysformer}$, a trans$\textbf{former}$ model that updates an initial $\textbf{sys}$tem prompt to a more robust system prompt in the LLM input embedding space while attending to the user prompt. While keeping the LLM parameters frozen, the Sysformer is trained to refuse to respond to a set of harmful prompts while responding ideally to a set of safe ones. Through extensive experiments on $5$ LLMs from different families and $2$ recent benchmarks, we demonstrate that Sysformer can significantly enhance the robustness of LLMs, leading to upto $80\%$ gain in the refusal rate on harmful prompts while enhancing the compliance with the safe prompts by upto $90\%$. Results also generalize well to sophisticated jailbreaking attacks, making LLMs upto $100\%$ more robust against different attack strategies. We hope our findings lead to cheaper safeguarding of LLMs and motivate future investigations into designing variable system prompts.

http://arxiv.org/abs/2506.17318
Context manipulation attacks : Web agents are susceptible to corrupted memory. (9%)
Atharv Singh Patlan; Ashwin Hebbar; Pramod Viswanath; Prateek Mittal
Autonomous web navigation agents, which translate natural language instructions into sequences of browser actions, are increasingly deployed for complex tasks across e-commerce, information retrieval, and content discovery. Due to the stateless nature of large language models (LLMs), these agents rely heavily on external memory systems to maintain context across interactions. Unlike centralized systems where context is securely stored server-side, agent memory is often managed client-side or by third-party applications, creating significant security vulnerabilities. This was recently exploited to attack production systems.
  We introduce and formalize "plan injection," a novel context manipulation attack that corrupts these agents' internal task representations by targeting this vulnerable context. Through systematic evaluation of two popular web agents, Browser-use and Agent-E, we show that plan injections bypass robust prompt injection defenses, achieving up to 3x higher attack success rates than comparable prompt-based attacks. Furthermore, "context-chained injections," which craft logical bridges between legitimate user goals and attacker objectives, lead to a 17.7% increase in success rate for privacy exfiltration tasks. Our findings highlight that secure memory handling must be a first-class concern in agentic systems.

http://arxiv.org/abs/2506.15343
Offensive Robot Cybersecurity. (2%)
Víctor Mayoral-Vilches
Offensive Robot Cybersecurity introduces a groundbreaking approach by advocating for offensive security methods empowered by means of automation. It emphasizes the necessity of understanding attackers' tactics and identifying vulnerabilities in advance to develop effective defenses, thereby improving robots' security posture. This thesis leverages a decade of robotics experience, employing Machine Learning and Game Theory to streamline the vulnerability identification and exploitation process. Intrinsically, the thesis uncovers a profound connection between robotic architecture and cybersecurity, highlighting that the design and creation aspect of robotics deeply intertwines with its protection against attacks. This duality -- whereby the architecture that shapes robot behavior and capabilities also necessitates a defense mechanism through offensive and defensive cybersecurity strategies -- creates a unique equilibrium. Approaching cybersecurity with a dual perspective of defense and attack, rooted in an understanding of systems architecture, has been pivotal. Through comprehensive analysis, including ethical considerations, the development of security tools, and executing cyber attacks on robot software, hardware, and industry deployments, this thesis proposes a novel architecture for cybersecurity cognitive engines. These engines, powered by advanced game theory and machine learning, pave the way for autonomous offensive cybersecurity strategies for robots, marking a significant shift towards self-defending robotic systems. This research not only underscores the importance of offensive measures in enhancing robot cybersecurity but also sets the stage for future advancements where robots are not just resilient to cyber threats but are equipped to autonomously safeguard themselves.

http://arxiv.org/abs/2506.14582
Busting the Paper Ballot: Voting Meets Adversarial Machine Learning. (99%)
Kaleel Mahmood; Caleb Manicke; Ethan Rathbun; Aayushi Verma; Sohaib Ahmad; Nicholas Stamatakis; Laurent Michel; Benjamin Fuller
  We show the security risk associated with using machine learning classifiers
in United States election tabulators. The central classification task in
election tabulation is deciding whether a mark does or does not appear on a
bubble associated to an alternative in a contest on the ballot. Barretto et al.
(E-Vote-ID 2021) reported that convolutional neural networks are a viable
option in this field, as they outperform simple feature-based classifiers.
  Our contributions to election security can be divided into four parts. To
demonstrate and analyze the hypothetical vulnerability of machine learning
models on election tabulators, we first introduce four new ballot datasets.
Second, we train and test a variety of different models on our new datasets.
These models include support vector machines, convolutional neural networks (a
basic CNN, VGG and ResNet), and vision transformers (Twins and CaiT). Third,
using our new datasets and trained models, we demonstrate that traditional
white box attacks are ineffective in the voting domain due to gradient masking.
Our analyses further reveal that gradient masking is a product of numerical
instability. We use a modified difference of logits ratio loss to overcome this
issue (Croce and Hein, ICML 2020). Fourth, in the physical world, we conduct
attacks with the adversarial examples generated using our new methods. In
traditional adversarial machine learning, a high (50% or greater) attack
success rate is ideal. However, for certain elections, even a 5% attack success
rate can flip the outcome of a race. We show such an impact is possible in the
physical domain. We thoroughly discuss attack realism, and the challenges and
practicality associated with printing and scanning ballot adversarial examples.


http://arxiv.org/abs/2506.14539
Doppelg\"anger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attack. (86%)
Daewon Kang; YeongHwan Shin; Doyeon Kim; Kyu-Hwan Jung; Meong Hi Son
  Since the advent of large language models, prompt engineering now enables the
rapid, low-effort creation of diverse autonomous agents that are already in
widespread use. Yet this convenience raises urgent concerns about the safety,
robustness, and behavioral consistency of the underlying prompts, along with
the pressing challenge of preventing those prompts from being exposed to user's
attempts. In this paper, we propose the ''Doppelg\"anger method'' to
demonstrate the risk of an agent being hijacked, thereby exposing system
instructions and internal information. Next, we define the ''Prompt Alignment
Collapse under Adversarial Transfer (PACAT)'' level to evaluate the
vulnerability to this adversarial transfer attack. We also propose a ''Caution
for Adversarial Transfer (CAT)'' prompt to counter the Doppelg\"anger method.
The experimental results demonstrate that the Doppelg\"anger method can
compromise the agent's consistency and expose its internal information. In
contrast, CAT prompts enable effective defense against this adversarial attack.


http://arxiv.org/abs/2506.14374
Excessive Reasoning Attack on Reasoning LLMs. (33%)
Wai Man Si; Mingjie Li; Michael Backes; Yang Zhang
  Recent reasoning large language models (LLMs), such as OpenAI o1 and
DeepSeek-R1, exhibit strong performance on complex tasks through test-time
inference scaling. However, prior studies have shown that these models often
incur significant computational costs due to excessive reasoning, such as
frequent switching between reasoning trajectories (e.g., underthinking) or
redundant reasoning on simple questions (e.g., overthinking). In this work, we
expose a novel threat: adversarial inputs can be crafted to exploit excessive
reasoning behaviors and substantially increase computational overhead without
compromising model utility. Therefore, we propose a novel loss framework
consisting of three components: (1) Priority Cross-Entropy Loss, a modification
of the standard cross-entropy objective that emphasizes key tokens by
leveraging the autoregressive nature of LMs; (2) Excessive Reasoning Loss,
which encourages the model to initiate additional reasoning paths during
inference; and (3) Delayed Termination Loss, which is designed to extend the
reasoning process and defer the generation of final outputs. We optimize and
evaluate our attack for the GSM8K and ORCA datasets on
DeepSeek-R1-Distill-LLaMA and DeepSeek-R1-Distill-Qwen. Empirical results
demonstrate a 3x to 9x increase in reasoning length with comparable utility
performance. Furthermore, our crafted adversarial inputs exhibit
transferability, inducing computational overhead in o3-mini, o1-mini,
DeepSeek-R1, and QWQ models.


http://arxiv.org/abs/2506.14866
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. (3%)
Thomas Kuntz; Agatha Duzan; Hao Zhao; Francesco Croce; Zico Kolter; Nicolas Flammarion; Maksym Andriushchenko
Computer use agents are LLM-based agents that can directly interact with a graphical user interface, by processing screenshots or accessibility trees. While these systems are gaining popularity, their safety has been largely overlooked, despite the fact that evaluating and understanding their potential for harmful behavior is essential for widespread adoption. To address this gap, we introduce OS-Harm, a new benchmark for measuring safety of computer use agents. OS-Harm is built on top of the OSWorld environment and aims to test models across three categories of harm: deliberate user misuse, prompt injection attacks, and model misbehavior. To cover these cases, we create 150 tasks that span several types of safety violations (harassment, copyright infringement, disinformation, data exfiltration, etc.) and require the agent to interact with a variety of OS applications (email client, code editor, browser, etc.). Moreover, we propose an automated judge to evaluate both accuracy and safety of agents that achieves high agreement with human annotations (0.76 and 0.79 F1 score). We evaluate computer use agents based on a range of frontier models - such as o4-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro - and provide insights into their safety. In particular, all models tend to directly comply with many deliberate misuse queries, are relatively vulnerable to static prompt injections, and occasionally perform unsafe actions. The OS-Harm benchmark is available at https://github.com/tml-epfl/os-harm.

http://arxiv.org/abs/2506.17299
LLM Jailbreak Oracle. (1%)
Shuyi Lin; Anshuman Suri; Alina Oprea; Cheng Tan
As large language models (LLMs) become increasingly deployed in safety-critical applications, the lack of systematic methods to assess their vulnerability to jailbreak attacks presents a critical security gap. We introduce the jailbreak oracle problem: given a model, prompt, and decoding strategy, determine whether a jailbreak response can be generated with likelihood exceeding a specified threshold. This formalization enables a principled study of jailbreak vulnerabilities. Answering the jailbreak oracle problem poses significant computational challenges -- the search space grows exponentially with the length of the response tokens. We present Boa, the first efficient algorithm for solving the jailbreak oracle problem. Boa employs a three-phase search strategy: (1) constructing block lists to identify refusal patterns, (2) breadth-first sampling to identify easily accessible jailbreaks, and (3) depth-first priority search guided by fine-grained safety scores to systematically explore promising low-probability paths. Boa enables rigorous security assessments including systematic defense evaluation, standardized comparison of red team attacks, and model certification under extreme adversarial conditions.

http://arxiv.org/abs/2506.13276
Navigating the Black Box: Leveraging LLMs for Effective Text-Level Graph Injection Attacks. (69%)
Yuefei Lyu; Chaozhuo Li; Xi Zhang; Tianle Zhang
  Text-attributed graphs (TAGs) integrate textual data with graph structures,
providing valuable insights in applications such as social network analysis and
recommendation systems. Graph Neural Networks (GNNs) effectively capture both
topological structure and textual information in TAGs but are vulnerable to
adversarial attacks. Existing graph injection attack (GIA) methods assume that
attackers can directly manipulate the embedding layer, producing
non-explainable node embeddings. Furthermore, the effectiveness of these
attacks often relies on surrogate models with high training costs. Thus, this
paper introduces ATAG-LLM, a novel black-box GIA framework tailored for TAGs.
Our approach leverages large language models (LLMs) to generate interpretable
text-level node attributes directly, ensuring attacks remain feasible in
real-world scenarios. We design strategies for LLM prompting that balance
exploration and reliability to guide text generation, and propose a similarity
assessment method to evaluate attack text effectiveness in disrupting graph
homophily. This method efficiently perturbs the target node with minimal
training costs in a strict black-box setting, ensuring a text-level graph
injection attack for TAGs. Experiments on real-world TAG datasets validate the
superior performance of ATAG-LLM compared to state-of-the-art embedding-level
and text-level attack methods.


http://arxiv.org/abs/2506.13160
CertDW: Towards Certified Dataset Ownership Verification via Conformal Prediction. (22%)
Ting Qiao; Yiming Li; Jianbin Li; Yingjia Wang; Leyi Qi; Junfeng Guo; Ruili Feng; Dacheng Tao
  Deep neural networks (DNNs) rely heavily on high-quality open-source datasets
(e.g., ImageNet) for their success, making dataset ownership verification (DOV)
crucial for protecting public dataset copyrights. In this paper, we find
existing DOV methods (implicitly) assume that the verification process is
faithful, where the suspicious model will directly verify ownership by using
the verification samples as input and returning their results. However, this
assumption may not necessarily hold in practice and their performance may
degrade sharply when subjected to intentional or unintentional perturbations.
To address this limitation, we propose the first certified dataset watermark
(i.e., CertDW) and CertDW-based certified dataset ownership verification method
that ensures reliable verification even under malicious attacks, under certain
conditions (e.g., constrained pixel-level perturbation). Specifically, inspired
by conformal prediction, we introduce two statistical measures, including
principal probability (PP) and watermark robustness (WR), to assess model
prediction stability on benign and watermarked samples under noise
perturbations. We prove there exists a provable lower bound between PP and WR,
enabling ownership verification when a suspicious model's WR value
significantly exceeds the PP values of multiple benign models trained on
watermark-free datasets. If the number of PP values smaller than WR exceeds a
threshold, the suspicious model is regarded as having been trained on the
protected dataset. Extensive experiments on benchmark datasets verify the
effectiveness of our CertDW method and its resistance to potential adaptive
attacks. Our codes are at
\href{https://github.com/NcepuQiaoTing/CertDW}{GitHub}.


http://arxiv.org/abs/2506.13563
Unlearning-Enhanced Website Fingerprinting Attack: Against Backdoor Poisoning in Anonymous Networks. (15%)
Yali Yuan; Kai Xu; Ruolin Ma; Yuchen Zhang
  Website Fingerprinting (WF) is an effective tool for regulating and governing
the dark web. However, its performance can be significantly degraded by
backdoor poisoning attacks in practical deployments. This paper aims to address
the problem of hidden backdoor poisoning attacks faced by Website
Fingerprinting attack, and designs a feasible mothed that integrates unlearning
technology to realize detection of automatic poisoned points and complete
removal of its destructive effects, requiring only a small number of known
poisoned test points. Taking Tor onion routing as an example, our method
evaluates the influence value of each training sample on these known poisoned
test points as the basis for judgment. We optimize the use of influence scores
to identify poisoned samples within the training dataset. Furthermore, by
quantifying the difference between the contribution of model parameters on the
taining data and the clean data, the target parameters are dynamically adjusted
to eliminate the impact of the backdoor attacks. Experiments on public datasets
under the assumptions of closed-world (CW) and open-world (OW) verify the
effectiveness of the proposed method. In complex scenes containing both clean
website fingerprinting features and backdoor triggers, the accuracy of the
model on the poisoned dataset and the test dataset is stable at about 80%,
significantly outperforming the traditional WF attack models. In addition, the
proposed method achieves a 2-3 times speedup in runtime efficiency compared to
baseline methods. By incorporating machine unlearning, we realize a WF attack
model that exhibits enhanced resistance to backdoor poisoning and faster
execution speeds in adversarial settings.


http://arxiv.org/abs/2506.13726
Weakest Link in the Chain: Security Vulnerabilities in Advanced Reasoning Models. (8%)
Arjun Krishna; Aaditya Rastogi; Erick Galinkin
  The introduction of advanced reasoning capabilities have improved the
problem-solving performance of large language models, particularly on math and
coding benchmarks. However, it remains unclear whether these reasoning models
are more or less vulnerable to adversarial prompt attacks than their
non-reasoning counterparts. In this work, we present a systematic evaluation of
weaknesses in advanced reasoning models compared to similar non-reasoning
models across a diverse set of prompt-based attack categories. Using
experimental data, we find that on average the reasoning-augmented models are
\emph{slightly more robust} than non-reasoning models (42.51\% vs 45.53\%
attack success rate, lower is better). However, this overall trend masks
significant category-specific differences: for certain attack types the
reasoning models are substantially \emph{more vulnerable} (e.g., up to 32
percentage points worse on a tree-of-attacks prompt), while for others they are
markedly \emph{more robust} (e.g., 29.8 points better on cross-site scripting
injection). Our findings highlight the nuanced security implications of
advanced reasoning in language models and emphasize the importance of
stress-testing safety across diverse adversarial techniques.


http://arxiv.org/abs/2506.12871
Active Adversarial Noise Suppression for Image Forgery Localization. (99%)
Rongxuan Peng; Shunquan Tan; Xianbo Mo; Alex C. Kot; Jiwu Huang
  Recent advances in deep learning have significantly propelled the development
of image forgery localization. However, existing models remain highly
vulnerable to adversarial attacks: imperceptible noise added to forged images
can severely mislead these models. In this paper, we address this challenge
with an Adversarial Noise Suppression Module (ANSM) that generate a defensive
perturbation to suppress the attack effect of adversarial noise. We observe
that forgery-relevant features extracted from adversarial and original forged
images exhibit distinct distributions. To bridge this gap, we introduce
Forgery-relevant Features Alignment (FFA) as a first-stage training strategy,
which reduces distributional discrepancies by minimizing the channel-wise
Kullback-Leibler divergence between these features. To further refine the
defensive perturbation, we design a second-stage training strategy, termed
Mask-guided Refinement (MgR), which incorporates a dual-mask constraint. MgR
ensures that the perturbation remains effective for both adversarial and
original forged images, recovering forgery localization accuracy to their
original level. Extensive experiments across various attack algorithms
demonstrate that our method significantly restores the forgery localization
model's performance on adversarial images. Notably, when ANSM is applied to
original forged images, the performance remains nearly unaffected. To our best
knowledge, this is the first report of adversarial defense in image forgery
localization tasks. We have released the source code and anti-forensics
dataset.


http://arxiv.org/abs/2506.12875
Intriguing Frequency Interpretation of Adversarial Robustness for CNNs and ViTs. (93%)
Lu Chen; Han Yang; Hu Wang; Yuxin Cao; Shaofeng Li; Yuan Luo
  Adversarial examples have attracted significant attention over the years, yet
understanding their frequency-based characteristics remains insufficient. In
this paper, we investigate the intriguing properties of adversarial examples in
the frequency domain for the image classification task, with the following key
findings. (1) As the high-frequency components increase, the performance gap
between adversarial and natural examples becomes increasingly pronounced. (2)
The model performance against filtered adversarial examples initially increases
to a peak and declines to its inherent robustness. (3) In Convolutional Neural
Networks, mid- and high-frequency components of adversarial examples exhibit
their attack capabilities, while in Transformers, low- and mid-frequency
components of adversarial examples are particularly effective. These results
suggest that different network architectures have different frequency
preferences and that differences in frequency components between adversarial
and natural examples may directly influence model robustness. Based on our
findings, we further conclude with three useful proposals that serve as a
valuable reference to the AI model security community.


http://arxiv.org/abs/2506.13024
Position: Certified Robustness Does Not (Yet) Imply Model Security. (73%)
Andrew C. Cullen; Paul Montague; Sarah M. Erfani; Benjamin I. P. Rubinstein
  While certified robustness is widely promoted as a solution to adversarial
examples in Artificial Intelligence systems, significant challenges remain
before these techniques can be meaningfully deployed in real-world
applications. We identify critical gaps in current research, including the
paradox of detection without distinction, the lack of clear criteria for
practitioners to evaluate certification schemes, and the potential security
risks arising from users' expectations surrounding ``guaranteed" robustness
claims. This position paper is a call to arms for the certification research
community, proposing concrete steps to address these fundamental challenges and
advance the field toward practical applicability.


http://arxiv.org/abs/2506.12911
Constraint-Guided Prediction Refinement via Deterministic Diffusion Trajectories. (33%)
Pantelis Dogoulis; Fabien Bernier; Félix Fourreau; Karim Tit; Maxime Cordy
  Many real-world machine learning tasks require outputs that satisfy hard
constraints, such as physical conservation laws, structured dependencies in
graphs, or column-level relationships in tabular data. Existing approaches rely
either on domain-specific architectures and losses or on strong assumptions on
the constraint space, restricting their applicability to linear or convex
constraints. We propose a general-purpose framework for constraint-aware
refinement that leverages denoising diffusion implicit models (DDIMs). Starting
from a coarse prediction, our method iteratively refines it through a
deterministic diffusion trajectory guided by a learned prior and augmented by
constraint gradient corrections. The approach accommodates a wide class of
non-convex and nonlinear equality constraints and can be applied post hoc to
any base model. We demonstrate the method in two representative domains:
constrained adversarial attack generation on tabular data with column-level
dependencies and in AC power flow prediction under Kirchhoff's laws. Across
both settings, our diffusion-guided refinement improves both constraint
satisfaction and performance while remaining lightweight and model-agnostic.


http://arxiv.org/abs/2506.15734
The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models. (10%)
Peiyuan Tang; Haojie Xin; Xiaodong Zhang; Jun Sun; Qin Xia; Zijiang Yang
As Vision-Language Models (VLMs) demonstrate increasing capabilities across real-world applications such as code generation and chatbot assistance, ensuring their safety has become paramount. Unlike traditional Large Language Models (LLMs), VLMs face unique vulnerabilities due to their multimodal nature, allowing adversaries to modify visual or textual inputs to bypass safety guardrails and trigger the generation of harmful content. Through systematic analysis of VLM behavior under attack, we identify a novel phenomenon termed ``delayed safety awareness''. Specifically, we observe that safety-aligned VLMs may initially be compromised to produce harmful content, but eventually recognize the associated risks and attempt to self-correct. This pattern suggests that VLMs retain their underlying safety awareness but experience a temporal delay in their activation. Building on this insight, we hypothesize that VLMs' safety awareness can be proactively reactivated through carefully designed prompts. To this end, we introduce ``The Safety Reminder'', a soft prompt tuning approach that optimizes learnable prompt tokens, which are periodically injected during the text generation process to enhance safety awareness, effectively preventing harmful content generation. Additionally, our safety reminder only activates when harmful content is detected, leaving normal conversations unaffected and preserving the model's performance on benign tasks. Through comprehensive evaluation across three established safety benchmarks and one adversarial attacks, we demonstrate that our approach significantly reduces attack success rates while maintaining model utility, offering a practical solution for deploying safer VLMs in real-world applications.

http://arxiv.org/abs/2506.13060
Rethinking Explainability in the Era of Multimodal AI. (1%)
Chirag Agarwal
  While multimodal AI systems (models jointly trained on heterogeneous data
types such as text, time series, graphs, and images) have become ubiquitous and
achieved remarkable performance across high-stakes applications, transparent
and accurate explanation algorithms are crucial for their safe deployment and
ensure user trust. However, most existing explainability techniques remain
unimodal, generating modality-specific feature attributions, concepts, or
circuit traces in isolation and thus failing to capture cross-modal
interactions. This paper argues that such unimodal explanations systematically
misrepresent and fail to capture the cross-modal influence that drives
multimodal model decisions, and the community should stop relying on them for
interpreting multimodal models. To support our position, we outline key
principles for multimodal explanations grounded in modality: Granger-style
modality influence (controlled ablations to quantify how removing one modality
changes the explanation for another), Synergistic faithfulness (explanations
capture the model's predictive power when modalities are combined), and Unified
stability (explanations remain consistent under small, cross-modal
perturbations). This targeted shift to multimodal explanations will help the
community uncover hidden shortcuts, mitigate modality bias, improve model
reliability, and enhance safety in high-stakes settings where incomplete
explanations can have serious consequences.


http://arxiv.org/abs/2506.12685
Alphabet Index Mapping: Jailbreaking LLMs through Semantic Dissimilarity. (92%)
Bilal Saleh Husain
  Large Language Models (LLMs) have demonstrated remarkable capabilities, yet
their susceptibility to adversarial attacks, particularly jailbreaking, poses
significant safety and ethical concerns. While numerous jailbreak methods
exist, many suffer from computational expense, high token usage, or complex
decoding schemes. Liu et al. (2024) introduced FlipAttack, a black-box method
that achieves high attack success rates (ASR) through simple prompt
manipulation. This paper investigates the underlying mechanisms of FlipAttack's
effectiveness by analyzing the semantic changes induced by its flipping modes.
We hypothesize that semantic dissimilarity between original and manipulated
prompts is inversely correlated with ASR. To test this, we examine embedding
space visualizations (UMAP, KDE) and cosine similarities for FlipAttack's
modes. Furthermore, we introduce a novel adversarial attack, Alphabet Index
Mapping (AIM), designed to maximize semantic dissimilarity while maintaining
simple decodability. Experiments on GPT-4 using a subset of AdvBench show AIM
and its variant AIM+FWO achieve a 94% ASR, outperforming FlipAttack and other
methods on this subset. Our findings suggest that while high semantic
dissimilarity is crucial, a balance with decoding simplicity is key for
successful jailbreaking. This work contributes to a deeper understanding of
adversarial prompt mechanics and offers a new, effective jailbreak technique.


http://arxiv.org/abs/2506.12454
On the existence of consistent adversarial attacks in high-dimensional linear classification. (91%)
Matteo Vilucchio; Lenka Zdeborová; Bruno Loureiro
  What fundamentally distinguishes an adversarial attack from a
misclassification due to limited model expressivity or finite data? In this
work, we investigate this question in the setting of high-dimensional binary
classification, where statistical effects due to limited data availability play
a central role. We introduce a new error metric that precisely capture this
distinction, quantifying model vulnerability to consistent adversarial attacks
-- perturbations that preserve the ground-truth labels. Our main technical
contribution is an exact and rigorous asymptotic characterization of these
metrics in both well-specified models and latent space models, revealing
different vulnerability patterns compared to standard robust error measures.
The theoretical results demonstrate that as models become more
overparameterized, their vulnerability to label-preserving perturbations grows,
offering theoretical insight into the mechanisms underlying model sensitivity
to adversarial attacks.


http://arxiv.org/abs/2506.17283
Second Order State Hallucinations for Adversarial Attack Mitigation in Formation Control of Multi-Agent Systems. (86%)
Laksh Patel; Akhilesh Raj
The increasing deployment of multi-agent systems (MAS) in critical infrastructures such as autonomous transportation, disaster relief, and smart cities demands robust formation control mechanisms resilient to adversarial attacks. Traditional consensus-based controllers, while effective under nominal conditions, are highly vulnerable to data manipulation, sensor spoofing, and communication failures. To address this challenge, we propose Second-Order State Hallucination (SOSH), a novel framework that detects compromised agents through distributed residual monitoring and maintains formation stability by replacing attacked states with predictive second-order approximations. Unlike existing mitigation strategies that require significant restructuring or induce long transients, SOSH offers a lightweight, decentralized correction mechanism based on second-order Taylor expansions, enabling rapid and scalable resilience. We establish rigorous Lyapunov-based stability guarantees, proving that formation errors remain exponentially bounded even under persistent attacks, provided the hallucination parameters satisfy explicit conditions. Comprehensive Monte Carlo experiments on a 5-agent complete graph formation demonstrate that SOSH outperforms established robust control schemes, including W-MSR and Huber-based consensus filters, achieving faster convergence rates, lower steady-state error, and superior transient recovery. Our results confirm that SOSH combines theoretical robustness with practical deployability, offering a promising direction for securing MAS formations against sophisticated adversarial threats.

http://arxiv.org/abs/2506.12706
NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models. (82%)
Jiaming Zhang; Xin Wang; Xingjun Ma; Lingyu Qiu; Yu-Gang Jiang; Jitao Sang
  Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable
capabilities in understanding relationships between visual and textual data
through joint embedding spaces. Despite their effectiveness, these models
remain vulnerable to adversarial attacks, particularly in the image modality,
posing significant security concerns. Building upon our previous work on
Adversarial Prompt Tuning (AdvPT), which introduced learnable text prompts to
enhance adversarial robustness in VLMs without extensive parameter training, we
present a significant extension by introducing the Neural Augmentor framework
for Multi-modal Adversarial Prompt Tuning (NAP-Tuning).Our key innovations
include: (1) extending AdvPT from text-only to multi-modal prompting across
both text and visual modalities, (2) expanding from single-layer to multi-layer
prompt architectures, and (3) proposing a novel architecture-level redesign
through our Neural Augmentor approach, which implements feature purification to
directly address the distortions introduced by adversarial attacks in feature
space. Our NAP-Tuning approach incorporates token refiners that learn to
reconstruct purified features through residual connections, allowing for
modality-specific and layer-specific feature correction.Comprehensive
experiments demonstrate that NAP-Tuning significantly outperforms existing
methods across various datasets and attack types. Notably, our approach shows
significant improvements over the strongest baselines under the challenging
AutoAttack benchmark, outperforming them by 33.5% on ViT-B16 and 33.0% on
ViT-B32 architectures while maintaining competitive clean accuracy.


http://arxiv.org/abs/2506.12411
InverTune: Removing Backdoors from Multimodal Contrastive Learning Models via Trigger Inversion and Activation Tuning. (80%)
Mengyuan Sun; Yu Li; Yuchen Liu; Bo Du; Yunjie Ge
  Multimodal contrastive learning models like CLIP have demonstrated remarkable
vision-language alignment capabilities, yet their vulnerability to backdoor
attacks poses critical security risks. Attackers can implant latent triggers
that persist through downstream tasks, enabling malicious control of model
behavior upon trigger presentation. Despite great success in recent defense
mechanisms, they remain impractical due to strong assumptions about attacker
knowledge or excessive clean data requirements. In this paper, we introduce
InverTune, the first backdoor defense framework for multimodal models under
minimal attacker assumptions, requiring neither prior knowledge of attack
targets nor access to the poisoned dataset. Unlike existing defense methods
that rely on the same dataset used in the poisoning stage, InverTune
effectively identifies and removes backdoor artifacts through three key
components, achieving robust protection against backdoor attacks. Specifically,
InverTune first exposes attack signatures through adversarial simulation,
probabilistically identifying the target label by analyzing model response
patterns. Building on this, we develop a gradient inversion technique to
reconstruct latent triggers through activation pattern analysis. Finally, a
clustering-guided fine-tuning strategy is employed to erase the backdoor
function with only a small amount of arbitrary clean data, while preserving the
original model capabilities. Experimental results show that InverTune reduces
the average attack success rate (ASR) by 97.87% against the state-of-the-art
(SOTA) attacks while limiting clean accuracy (CA) degradation to just 3.07%.
This work establishes a new paradigm for securing multimodal systems, advancing
security in foundation model deployment without compromising performance.


http://arxiv.org/abs/2506.12707
SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression. (76%)
Yucheng Li; Surin Ahn; Huiqiang Jiang; Amir H. Abdi; Yuqing Yang; Lili Qiu
  Large language models (LLMs) have achieved widespread adoption across
numerous applications. However, many LLMs are vulnerable to malicious attacks
even after safety alignment. These attacks typically bypass LLMs' safety
guardrails by wrapping the original malicious instructions inside adversarial
jailbreaks prompts. Previous research has proposed methods such as adversarial
training and prompt rephrasing to mitigate these safety vulnerabilities, but
these methods often reduce the utility of LLMs or lead to significant
computational overhead and online latency. In this paper, we propose
SecurityLingua, an effective and efficient approach to defend LLMs against
jailbreak attacks via security-oriented prompt compression. Specifically, we
train a prompt compressor designed to discern the "true intention" of the input
prompt, with a particular focus on detecting the malicious intentions of
adversarial prompts. Then, in addition to the original prompt, the intention is
passed via the system prompt to the target LLM to help it identify the true
intention of the request. SecurityLingua ensures a consistent user experience
by leaving the original input prompt intact while revealing the user's
potentially malicious intention and stimulating the built-in safety guardrails
of the LLM. Moreover, thanks to prompt compression, SecurityLingua incurs only
a negligible overhead and extra token cost compared to all existing defense
methods, making it an especially practical solution for LLM defense.
Experimental results demonstrate that SecurityLingua can effectively defend
against malicious attacks and maintain utility of the LLM with negligible
compute and latency overhead. Our code is available at
https://aka.ms/SecurityLingua.


http://arxiv.org/abs/2506.12430
Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025. (70%)
Zonghao Ying; Siyang Wu; Run Hao; Peng Ying; Shixuan Sun; Pengyu Chen; Junze Chen; Hao Du; Kaiwen Shen; Shangkun Wu; Jiwei Wei; Shiyuan He; Yang Yang; Xiaohai Xu; Ke Ma; Qianqian Xu; Qingming Huang; Shi Lin; Xun Wang; Changting Lin; Meng Han; Yilei Jiang; Siqi Lai; Yaozhi Zheng; Yifei Song; Xiangyu Yue; Zonglei Jing; Tianyuan Zhang; Zhilei Zhu; Aishan Liu; Jiakai Wang; Siyuan Liang; Xianglong Kong; Hainan Li; Junjie Mu; Haotong Qin; Yue Yu; Lei Chen; Felix Juefei-Xu; Qing Guo; Xinyun Chen; Yew Soon Ong; Xianglong Liu; Dawn Song; Alan Yuille; Philip Torr; Dacheng Tao
  Multimodal Large Language Models (MLLMs) have enabled transformative
advancements across diverse applications but remain susceptible to safety
threats, especially jailbreak attacks that induce harmful outputs. To
systematically evaluate and improve their safety, we organized the Adversarial
Testing & Large-model Alignment Safety Grand Challenge (ATLAS) 2025}. This
technical report presents findings from the competition, which involved 86
teams testing MLLM vulnerabilities via adversarial image-text attacks in two
phases: white-box and black-box evaluations. The competition results highlight
ongoing challenges in securing MLLMs and provide valuable guidance for
developing stronger defense mechanisms. The challenge establishes new
benchmarks for MLLM safety evaluation and lays groundwork for advancing safer
multimodal AI systems. The code and data for this challenge are openly
available at https://github.com/NY1024/ATLAS_Challenge_2025.


http://arxiv.org/abs/2506.12613
Existence of Adversarial Examples for Random Convolutional Networks via Isoperimetric Inequalities on $\mathbb{so}(d)$. (50%)
Amit Daniely
  We show that adversarial examples exist for various random convolutional
networks, and furthermore, that this is a relatively simple consequence of the
isoperimetric inequality on the special orthogonal group $\mathbb{so}(d)$. This
extends and simplifies a recent line of work which shows similar results for
random fully connected networks.


http://arxiv.org/abs/2506.12340
Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models. (15%)
Zongyu Wu; Minhua Lin; Zhiwei Zhang; Fali Wang; Xianren Zhang; Xiang Zhang; Suhang Wang
  Large vision-language models (LVLMs) have demonstrated outstanding
performance in many downstream tasks. However, LVLMs are trained on large-scale
datasets, which can pose privacy risks if training images contain sensitive
information. Therefore, it is important to detect whether an image is used to
train the LVLM. Recent studies have investigated membership inference attacks
(MIAs) against LVLMs, including detecting image-text pairs and single-modality
content. In this work, we focus on detecting whether a target image is used to
train the target LVLM. We design simple yet effective Image Corruption-Inspired
Membership Inference Attacks (ICIMIA) against LLVLMs, which are inspired by
LVLM's different sensitivity to image corruption for member and non-member
images. We first perform an MIA method under the white-box setting, where we
can obtain the embeddings of the image through the vision part of the target
LVLM. The attacks are based on the embedding similarity between the image and
its corrupted version. We further explore a more practical scenario where we
have no knowledge about target LVLMs and we can only query the target LVLMs
with an image and a question. We then conduct the attack by utilizing the
output text embeddings' similarity. Experiments on existing datasets validate
the effectiveness of our proposed attack methods under those two different
settings.


http://arxiv.org/abs/2506.17279
Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models. (2%)
Yash Sinha; Manit Baser; Murari Mandal; Dinil Mon Divakaran; Mohan Kankanhalli
Knowledge erasure in large language models (LLMs) is important for ensuring compliance with data and AI regulations, safeguarding user privacy, mitigating bias, and misinformation. Existing unlearning methods aim to make the process of knowledge erasure more efficient and effective by removing specific knowledge while preserving overall model performance, especially for retained information. However, it has been observed that the unlearning techniques tend to suppress and leave the knowledge beneath the surface, thus making it retrievable with the right prompts. In this work, we demonstrate that \textit{step-by-step reasoning} can serve as a backdoor to recover this hidden information. We introduce a step-by-step reasoning-based black-box attack, Sleek, that systematically exposes unlearning failures. We employ a structured attack framework with three core components: (1) an adversarial prompt generation strategy leveraging step-by-step reasoning built from LLM-generated queries, (2) an attack mechanism that successfully recalls erased content, and exposes unfair suppression of knowledge intended for retention and (3) a categorization of prompts as direct, indirect, and implied, to identify which query types most effectively exploit unlearning weaknesses. Through extensive evaluations on four state-of-the-art unlearning techniques and two widely used LLMs, we show that existing approaches fail to ensure reliable knowledge removal. Of the generated adversarial prompts, 62.5% successfully retrieved forgotten Harry Potter facts from WHP-unlearned Llama, while 50% exposed unfair suppression of retained knowledge. Our work highlights the persistent risks of information leakage, emphasizing the need for more robust unlearning strategies for erasure.

http://arxiv.org/abs/2506.12382
Exploring the Secondary Risks of Large Language Models. (2%)
Jiawei Chen; Zhengwei Fang; Xiao Yang; Chao Yu; Zhaoxia Yin; Hang Su
  Ensuring the safety and alignment of Large Language Models is a significant
challenge with their growing integration into critical applications and
societal functions. While prior research has primarily focused on jailbreak
attacks, less attention has been given to non-adversarial failures that subtly
emerge during benign interactions. We introduce secondary risks a novel class
of failure modes marked by harmful or misleading behaviors during benign
prompts. Unlike adversarial attacks, these risks stem from imperfect
generalization and often evade standard safety mechanisms. To enable systematic
evaluation, we introduce two risk primitives verbose response and speculative
advice that capture the core failure patterns. Building on these definitions,
we propose SecLens, a black-box, multi-objective search framework that
efficiently elicits secondary risk behaviors by optimizing task relevance, risk
activation, and linguistic plausibility. To support reproducible evaluation, we
release SecRiskBench, a benchmark dataset of 650 prompts covering eight diverse
real-world risk categories. Experimental results from extensive evaluations on
16 popular models demonstrate that secondary risks are widespread, transferable
across models, and modality independent, emphasizing the urgent need for
enhanced safety mechanisms to address benign yet harmful LLM behaviors in
real-world deployments.


http://arxiv.org/abs/2507.00015
Vision Transformer with Adversarial Indicator Token against Adversarial Attacks in Radio Signal Classifications. (99%)
Lu Zhang; Sangarapillai Lambotharan; Gan Zheng; Guisheng Liao; Xuekang Liu; Fabio Roli; Carsten Maple
The remarkable success of transformers across various fields such as natural language processing and computer vision has paved the way for their applications in automatic modulation classification, a critical component in the communication systems of Internet of Things (IoT) devices. However, it has been observed that transformer-based classification of radio signals is susceptible to subtle yet sophisticated adversarial attacks. To address this issue, we have developed a defensive strategy for transformer-based modulation classification systems to counter such adversarial attacks. In this paper, we propose a novel vision transformer (ViT) architecture by introducing a new concept known as adversarial indicator (AdvI) token to detect adversarial attacks. To the best of our knowledge, this is the first work to propose an AdvI token in ViT to defend against adversarial attacks. Integrating an adversarial training method with a detection mechanism using AdvI token, we combine a training time defense and running time defense in a unified neural network model, which reduces architectural complexity of the system compared to detecting adversarial perturbations using separate models. We investigate into the operational principles of our method by examining the attention mechanism. We show the proposed AdvI token acts as a crucial element within the ViT, influencing attention weights and thereby highlighting regions or features in the input data that are potentially suspicious or anomalous. Through experimental results, we demonstrate that our approach surpasses several competitive methods in handling white-box attack scenarios, including those utilizing the fast gradient method, projected gradient descent attacks and basic iterative method.

http://arxiv.org/abs/2506.11901
A Neural Rejection System Against Universal Adversarial Perturbations in Radio Signal Classification. (99%)
Lu Zhang; Sangarapillai Lambotharan; Gan Zheng; Fabio Roli
  Advantages of deep learning over traditional methods have been demonstrated
for radio signal classification in the recent years. However, various
researchers have discovered that even a small but intentional feature
perturbation known as adversarial examples can significantly deteriorate the
performance of the deep learning based radio signal classification. Among
various kinds of adversarial examples, universal adversarial perturbation has
gained considerable attention due to its feature of being data independent,
hence as a practical strategy to fool the radio signal classification with a
high success rate. Therefore, in this paper, we investigate a defense system
called neural rejection system to propose against universal adversarial
perturbations, and evaluate its performance by generating white-box universal
adversarial perturbations. We show that the proposed neural rejection system is
able to defend universal adversarial perturbations with significantly higher
accuracy than the undefended deep neural network.


http://arxiv.org/abs/2506.11892
Attention-based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices. (99%)
Lu Zhang; Sangarapillai Lambotharan; Gan Zheng; Guisheng Liao; Basil AsSadhan; Fabio Roli
  Due to great success of transformers in many applications such as natural
language processing and computer vision, transformers have been successfully
applied in automatic modulation classification. We have shown that
transformer-based radio signal classification is vulnerable to imperceptible
and carefully crafted attacks called adversarial examples. Therefore, we
propose a defense system against adversarial examples in transformer-based
modulation classifications. Considering the need for computationally efficient
architecture particularly for Internet of Things (IoT)-based applications or
operation of devices in environment where power supply is limited, we propose a
compact transformer for modulation classification. The advantages of robust
training such as adversarial training in transformers may not be attainable in
compact transformers. By demonstrating this, we propose a novel compact
transformer that can enhance robustness in the presence of adversarial attacks.
The new method is aimed at transferring the adversarial attention map from the
robustly trained large transformer to a compact transformer. The proposed
method outperforms the state-of-the-art techniques for the considered white-box
scenarios including fast gradient method and projected gradient descent
attacks. We have provided reasoning of the underlying working mechanisms and
investigated the transferability of the adversarial examples between different
architectures. The proposed method has the potential to protect the transformer
from the transferability of adversarial examples.


http://arxiv.org/abs/2506.11844
TrustGLM: Evaluating the Robustness of GraphLLMs Against Prompt, Text, and Structure Attacks. (98%)
Qihai Zhang; Xinyue Sheng; Yuanfu Sun; Qiaoyu Tan
  Inspired by the success of large language models (LLMs), there is a
significant research shift from traditional graph learning methods to LLM-based
graph frameworks, formally known as GraphLLMs. GraphLLMs leverage the reasoning
power of LLMs by integrating three key components: the textual attributes of
input nodes, the structural information of node neighborhoods, and
task-specific prompts that guide decision-making. Despite their promise, the
robustness of GraphLLMs against adversarial perturbations remains largely
unexplored-a critical concern for deploying these models in high-stakes
scenarios. To bridge the gap, we introduce TrustGLM, a comprehensive study
evaluating the vulnerability of GraphLLMs to adversarial attacks across three
dimensions: text, graph structure, and prompt manipulations. We implement
state-of-the-art attack algorithms from each perspective to rigorously assess
model resilience. Through extensive experiments on six benchmark datasets from
diverse domains, our findings reveal that GraphLLMs are highly susceptible to
text attacks that merely replace a few semantically similar words in a node's
textual attribute. We also find that standard graph structure attack methods
can significantly degrade model performance, while random shuffling of the
candidate label set in prompt templates leads to substantial performance drops.
Beyond characterizing these vulnerabilities, we investigate defense techniques
tailored to each attack vector through data-augmented training and adversarial
training, which show promising potential to enhance the robustness of
GraphLLMs. We hope that our open-sourced library will facilitate rapid,
equitable evaluation and inspire further innovative research in this field.


http://arxiv.org/abs/2506.11472
On the Natural Robustness of Vision-Language Models Against Visual Perception Attacks in Autonomous Driving. (92%)
Pedram Clemson University, Clemson, SC, USA MohajerAnsari; Amir Clemson University, Clemson, SC, USA Salarpour; Michael Technical University of Munich, Munich, Germany Kühr; Siyu Clemson University, Clemson, SC, USA Huang; Mohammad Technical University of Munich, Munich, Germany Hamad; Sebastian Technical University of Munich, Munich, Germany Steinhorst; Habeeb University of Texas at Arlington, Arlington, TX, USA Olufowobi; Mert D. Clemson University, Clemson, SC, USA Pesé
  Autonomous vehicles (AVs) rely on deep neural networks (DNNs) for critical
tasks such as traffic sign recognition (TSR), automated lane centering (ALC),
and vehicle detection (VD). However, these models are vulnerable to attacks
that can cause misclassifications and compromise safety. Traditional defense
mechanisms, including adversarial training, often degrade benign accuracy and
fail to generalize against unseen attacks. In this work, we introduce Vehicle
Vision Language Models (V2LMs), fine-tuned vision-language models specialized
for AV perception. Our findings demonstrate that V2LMs inherently exhibit
superior robustness against unseen attacks without requiring adversarial
training, maintaining significantly higher accuracy than conventional DNNs
under adversarial conditions. We evaluate two deployment strategies: Solo Mode,
where individual V2LMs handle specific perception tasks, and Tandem Mode, where
a single unified V2LM is fine-tuned for multiple tasks simultaneously.
Experimental results reveal that DNNs suffer performance drops of 33% to 46%
under attacks, whereas V2LMs maintain adversarial accuracy with reductions of
less than 8% on average. The Tandem Mode further offers a memory-efficient
alternative while achieving comparable robustness to Solo Mode. We also explore
integrating V2LMs as parallel components to AV perception to enhance resilience
against adversarial threats. Our results suggest that V2LMs offer a promising
path toward more secure and resilient AV perception systems.


http://arxiv.org/abs/2506.11521
Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models. (83%)
Jinming Wen; Xinyi Wu; Shuai Zhao; Yanhao Jia; Yuwen Li
  Multimodal large language models (MLLMs), which bridge the gap between
audio-visual and natural language processing, achieve state-of-the-art
performance on several audio-visual tasks. Despite the superior performance of
MLLMs, the scarcity of high-quality audio-visual training data and
computational resources necessitates the utilization of third-party data and
open-source MLLMs, a trend that is increasingly observed in contemporary
research. This prosperity masks significant security risks. Empirical studies
demonstrate that the latest MLLMs can be manipulated to produce malicious or
harmful content. This manipulation is facilitated exclusively through
instructions or inputs, including adversarial perturbations and malevolent
queries, effectively bypassing the internal security mechanisms embedded within
the models. To gain a deeper comprehension of the inherent security
vulnerabilities associated with audio-visual-based multimodal models, a series
of surveys investigates various types of attacks, including adversarial and
backdoor attacks. While existing surveys on audio-visual attacks provide a
comprehensive overview, they are limited to specific types of attacks, which
lack a unified review of various types of attacks. To address this issue and
gain insights into the latest trends in the field, this paper presents a
comprehensive and systematic review of audio-visual attacks, which include
adversarial attacks, backdoor attacks, and jailbreak attacks. Furthermore, this
paper also reviews various types of attacks in the latest audio-visual-based
MLLMs, a dimension notably absent in existing surveys. Drawing upon
comprehensive insights from a substantial review, this paper delineates both
challenges and emergent trends for future research on audio-visual attacks and
defense.


http://arxiv.org/abs/2506.11938
Improving Large Language Model Safety with Contrastive Representation Learning. (75%)
Samuel Simko; Mrinmaya Sachan; Bernhard Schölkopf; Zhijing Jin
  Large Language Models (LLMs) are powerful tools with profound societal
impacts, yet their ability to generate responses to diverse and uncontrolled
inputs leaves them vulnerable to adversarial attacks. While existing defenses
often struggle to generalize across varying attack types, recent advancements
in representation engineering offer promising alternatives. In this work, we
propose a defense framework that formulates model defense as a contrastive
representation learning (CRL) problem. Our method finetunes a model using a
triplet-based loss combined with adversarial hard negative mining to encourage
separation between benign and harmful representations. Our experimental results
across multiple models demonstrate that our approach outperforms prior
representation engineering-based defenses, improving robustness against both
input-level and embedding-space attacks without compromising standard
performance. Our code is available at
https://github.com/samuelsimko/crl-llm-defense


http://arxiv.org/abs/2506.12274
InfoFlood: Jailbreaking Large Language Models with Information Overload. (47%)
Advait Yadav; Haibo Jin; Man Luo; Jun Zhuang; Haohan Wang
  Large Language Models (LLMs) have demonstrated remarkable capabilities across
various domains. However, their potential to generate harmful responses has
raised significant societal and regulatory concerns, especially when
manipulated by adversarial techniques known as "jailbreak" attacks. Existing
jailbreak methods typically involve appending carefully crafted prefixes or
suffixes to malicious prompts in order to bypass the built-in safety mechanisms
of these models.
  In this work, we identify a new vulnerability in which excessive linguistic
complexity can disrupt built-in safety mechanisms-without the need for any
added prefixes or suffixes-allowing attackers to elicit harmful outputs
directly. We refer to this phenomenon as Information Overload.
  To automatically exploit this vulnerability, we propose InfoFlood, a
jailbreak attack that transforms malicious queries into complex,
information-overloaded queries capable of bypassing built-in safety mechanisms.
Specifically, InfoFlood: (1) uses linguistic transformations to rephrase
malicious queries, (2) identifies the root cause of failure when an attempt is
unsuccessful, and (3) refines the prompt's linguistic structure to address the
failure while preserving its malicious intent.
  We empirically validate the effectiveness of InfoFlood on four widely used
LLMs-GPT-4o, GPT-3.5-turbo, Gemini 2.0, and LLaMA 3.1-by measuring their
jailbreak success rates. InfoFlood consistently outperforms baseline attacks,
achieving up to 3 times higher success rates across multiple jailbreak
benchmarks. Furthermore, we demonstrate that commonly adopted post-processing
defenses, including OpenAI's Moderation API, Perspective API, and SmoothLLM,
fail to mitigate these attacks. This highlights a critical weakness in
traditional AI safety guardrails when confronted with information
overload-based jailbreaks.


http://arxiv.org/abs/2506.12104
DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents. (47%)
Hao Li; Xiaogeng Liu; Hung-Chun Chiu; Dianqi Li; Ning Zhang; Chaowei Xiao
  Large Language Models (LLMs) are increasingly central to agentic systems due
to their strong reasoning and planning capabilities. By interacting with
external environments through predefined tools, these agents can carry out
complex user tasks. Nonetheless, this interaction also introduces the risk of
prompt injection attacks, where malicious inputs from external sources can
mislead the agent's behavior, potentially resulting in economic loss, privacy
leakage, or system compromise. System-level defenses have recently shown
promise by enforcing static or predefined policies, but they still face two key
challenges: the ability to dynamically update security rules and the need for
memory stream isolation. To address these challenges, we propose DRIFT, a
Dynamic Rule-based Isolation Framework for Trustworthy agentic systems, which
enforces both control- and data-level constraints. A Secure Planner first
constructs a minimal function trajectory and a JSON-schema-style parameter
checklist for each function node based on the user query. A Dynamic Validator
then monitors deviations from the original plan, assessing whether changes
comply with privilege limitations and the user's intent. Finally, an Injection
Isolator detects and masks any instructions that may conflict with the user
query from the memory stream to mitigate long-term risks. We empirically
validate the effectiveness of DRIFT on the AgentDojo benchmark, demonstrating
its strong security performance while maintaining high utility across diverse
models -- showcasing both its robustness and adaptability.


http://arxiv.org/abs/2506.11615
Machine Unlearning for Robust DNNs: Attribution-Guided Partitioning and Neuron Pruning in Noisy Environments. (33%)
Deliang Jin; Gang Chen; Shuo Feng; Yufeng Ling; Haoran Zhu
  Deep neural networks (DNNs) have achieved remarkable success across diverse
domains, but their performance can be severely degraded by noisy or corrupted
training data. Conventional noise mitigation methods often rely on explicit
assumptions about noise distributions or require extensive retraining, which
can be impractical for large-scale models. Inspired by the principles of
machine unlearning, we propose a novel framework that integrates
attribution-guided data partitioning, discriminative neuron pruning, and
targeted fine-tuning to mitigate the impact of noisy samples. Our approach
employs gradient-based attribution to probabilistically distinguish
high-quality examples from potentially corrupted ones without imposing
restrictive assumptions on the noise. It then applies regression-based
sensitivity analysis to identify and prune neurons that are most vulnerable to
noise. Finally, the resulting network is fine-tuned on the high-quality data
subset to efficiently recover and enhance its generalization performance. This
integrated unlearning-inspired framework provides several advantages over
conventional noise-robust learning approaches. Notably, it combines data-level
unlearning with model-level adaptation, thereby avoiding the need for full
model retraining or explicit noise modeling. We evaluate our method on
representative tasks (e.g., CIFAR-10 image classification and speech
recognition) under various noise levels and observe substantial gains in both
accuracy and efficiency. For example, our framework achieves approximately a
10% absolute accuracy improvement over standard retraining on CIFAR-10 with
injected label noise, while reducing retraining time by up to 47% in some
settings. These results demonstrate the effectiveness and scalability of the
proposed approach for achieving robust generalization in noisy environments.


http://arxiv.org/abs/2506.12258
EgoPrivacy: What Your First-Person Camera Says About You? (1%)
Yijiang Li; Genpei Zhang; Jiacheng Cheng; Yi Li; Xiaojun Shan; Dashan Gao; Jiancheng Lyu; Yuan Li; Ning Bi; Nuno Vasconcelos
  While the rapid proliferation of wearable cameras has raised significant
concerns about egocentric video privacy, prior work has largely overlooked the
unique privacy threats posed to the camera wearer. This work investigates the
core question: How much privacy information about the camera wearer can be
inferred from their first-person view videos? We introduce EgoPrivacy, the
first large-scale benchmark for the comprehensive evaluation of privacy risks
in egocentric vision. EgoPrivacy covers three types of privacy (demographic,
individual, and situational), defining seven tasks that aim to recover private
information ranging from fine-grained (e.g., wearer's identity) to
coarse-grained (e.g., age group). To further emphasize the privacy threats
inherent to egocentric vision, we propose Retrieval-Augmented Attack, a novel
attack strategy that leverages ego-to-exo retrieval from an external pool of
exocentric videos to boost the effectiveness of demographic privacy attacks. An
extensive comparison of the different attacks possible under all threat models
is presented, showing that private information of the wearer is highly
susceptible to leakage. For instance, our findings indicate that foundation
models can effectively compromise wearer privacy even in zero-shot settings by
recovering attributes such as identity, scene, gender, and race with 70-80%
accuracy. Our code and data are available at
https://github.com/williamium3000/ego-privacy.


http://arxiv.org/abs/2506.12226
Learning Causality for Modern Machine Learning. (1%)
Yongqiang Chen
  In the past decades, machine learning with Empirical Risk Minimization (ERM)
has demonstrated great capability in learning and exploiting the statistical
patterns from data, or even surpassing humans. Despite the success, ERM avoids
the modeling of causality the way of understanding and handling changes, which
is fundamental to human intelligence. When deploying models beyond the training
environment, distribution shifts are everywhere. For example, an autopilot
system often needs to deal with new weather conditions that have not been seen
during training, An Al-aided drug discovery system needs to predict the
biochemical properties of molecules with respect to new viruses such as
COVID-19. It renders the problem of Out-of-Distribution (OOD) generalization
challenging to conventional machine learning.
  In this thesis, we investigate how to incorporate and realize the causality
for broader tasks in modern machine learning. In particular, we exploit the
invariance implied by the principle of independent causal mechanisms (ICM),
that is, the causal mechanisms generating the effects from causes do not inform
or influence each other. Therefore, the conditional distribution between the
target variable given its causes is invariant under distribution shifts. With
the causal invariance principle, we first instantiate it to graphs -- a general
data structure ubiquitous in many real-world industry and scientific
applications, such as financial networks and molecules. Then, we shall see how
learning the causality benefits many of the desirable properties of modern
machine learning, in terms of (i) OOD generalization capability; (ii)
interpretability; and (iii) robustness to adversarial attacks.
  Realizing the causality in machine learning, on the other hand, raises a
dilemma for optimization in conventional machine learning, as it often
contradicts the objective of ERM...


http://arxiv.org/abs/2506.10685
Unsourced Adversarial CAPTCHA: A Bi-Phase Adversarial CAPTCHA Framework. (99%)
Xia Du; Xiaoyuan Liu; Jizhe Zhou; Zheng Lin; Chi-man Pun; Cong Wu; Tao Li; Zhe Chen; Wei Ni; Jun Luo
  With the rapid advancements in deep learning, traditional CAPTCHA schemes are
increasingly vulnerable to automated attacks powered by deep neural networks
(DNNs). Existing adversarial attack methods often rely on original image
characteristics, resulting in distortions that hinder human interpretation and
limit applicability in scenarios lacking initial input images. To address these
challenges, we propose the Unsourced Adversarial CAPTCHA (UAC), a novel
framework generating high-fidelity adversarial examples guided by
attacker-specified text prompts. Leveraging a Large Language Model (LLM), UAC
enhances CAPTCHA diversity and supports both targeted and untargeted attacks.
For targeted attacks, the EDICT method optimizes dual latent variables in a
diffusion model for superior image quality. In untargeted attacks, especially
for black-box scenarios, we introduce bi-path unsourced adversarial CAPTCHA
(BP-UAC), a two-step optimization strategy employing multimodal gradients and
bi-path optimization for efficient misclassification. Experiments show BP-UAC
achieves high attack success rates across diverse systems, generating natural
CAPTCHAs indistinguishable to humans and DNNs.


http://arxiv.org/abs/2506.10722
TED-LaST: Towards Robust Backdoor Defense Against Adaptive Attacks. (98%)
Xiaoxing Mo; Yuxuan Cheng; Nan Sun; Leo Yu Zhang; Wei Luo; Shang Gao
  Deep Neural Networks (DNNs) are vulnerable to backdoor attacks, where
attackers implant hidden triggers during training to maliciously control model
behavior. Topological Evolution Dynamics (TED) has recently emerged as a
powerful tool for detecting backdoor attacks in DNNs. However, TED can be
vulnerable to backdoor attacks that adaptively distort topological
representation distributions across network layers. To address this limitation,
we propose TED-LaST (Topological Evolution Dynamics against Laundry, Slow
release, and Target mapping attack strategies), a novel defense strategy that
enhances TED's robustness against adaptive attacks. TED-LaST introduces two key
innovations: label-supervised dynamics tracking and adaptive layer emphasis.
These enhancements enable the identification of stealthy threats that evade
traditional TED-based defenses, even in cases of inseparability in topological
space and subtle topological perturbations. We review and classify data
poisoning tricks in state-of-the-art adaptive attacks and propose enhanced
adaptive attack with target mapping, which can dynamically shift malicious
tasks and fully leverage the stealthiness that adaptive attacks possess. Our
comprehensive experiments on multiple datasets (CIFAR-10, GTSRB, and
ImageNet100) and model architectures (ResNet20, ResNet101) show that TED-LaST
effectively counteracts sophisticated backdoors like Adap-Blend, Adapt-Patch,
and the proposed enhanced adaptive attack. TED-LaST sets a new benchmark for
robust backdoor detection, substantially enhancing DNN security against
evolving threats.


http://arxiv.org/abs/2506.10620
Assessing the Resilience of Automotive Intrusion Detection Systems to Adversarial Manipulation. (93%)
Stefano Longari; Paolo Cerracchio; Michele Carminati; Stefano Zanero
  The security of modern vehicles has become increasingly important, with the
controller area network (CAN) bus serving as a critical communication backbone
for various Electronic Control Units (ECUs). The absence of robust security
measures in CAN, coupled with the increasing connectivity of vehicles, makes
them susceptible to cyberattacks. While intrusion detection systems (IDSs) have
been developed to counter such threats, they are not foolproof. Adversarial
attacks, particularly evasion attacks, can manipulate inputs to bypass
detection by IDSs. This paper extends our previous work by investigating the
feasibility and impact of gradient-based adversarial attacks performed with
different degrees of knowledge against automotive IDSs. We consider three
scenarios: white-box (attacker with full system knowledge), grey-box (partial
system knowledge), and the more realistic black-box (no knowledge of the IDS'
internal workings or data). We evaluate the effectiveness of the proposed
attacks against state-of-the-art IDSs on two publicly available datasets.
Additionally, we study effect of the adversarial perturbation on the attack
impact and evaluate real-time feasibility by precomputing evasive payloads for
timed injection based on bus traffic. Our results demonstrate that, besides
attacks being challenging due to the automotive domain constraints, their
effectiveness is strongly dependent on the dataset quality, the target IDS, and
the attacker's degree of knowledge.


http://arxiv.org/abs/2506.10744
ObfusBFA: A Holistic Approach to Safeguarding DNNs from Different Types of Bit-Flip Attacks. (70%)
Xiaobei Yan; Han Qiu; Tianwei Zhang
  Bit-flip attacks (BFAs) represent a serious threat to Deep Neural Networks
(DNNs), where flipping a small number of bits in the model parameters or binary
code can significantly degrade the model accuracy or mislead the model
prediction in a desired way. Existing defenses exclusively focus on protecting
models for specific attacks and platforms, while lacking effectiveness for
other scenarios. We propose ObfusBFA, an efficient and holistic methodology to
mitigate BFAs targeting both the high-level model weights and low-level
codebase (executables or shared libraries). The key idea of ObfusBFA is to
introduce random dummy operations during the model inference, which effectively
transforms the delicate attacks into random bit flips, making it much harder
for attackers to pinpoint and exploit vulnerable bits. We design novel
algorithms to identify critical bits and insert obfuscation operations. We
evaluate ObfusBFA against different types of attacks, including the adaptive
scenarios where the attacker increases the flip bit budget to attempt to
circumvent our defense. The results show that ObfusBFA can consistently
preserve the model accuracy across various datasets and DNN architectures while
significantly reducing the attack success rates. Additionally, it introduces
minimal latency and storage overhead, making it a practical solution for
real-world applications.


http://arxiv.org/abs/2506.10949
Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitors. (68%)
Chen Yueh-Han; Nitish Joshi; Yulin Chen; Maksym Andriushchenko; Rico Angell; He He
  Current LLM safety defenses fail under decomposition attacks, where a
malicious goal is decomposed into benign subtasks that circumvent refusals. The
challenge lies in the existing shallow safety alignment techniques: they only
detect harm in the immediate prompt and do not reason about long-range intent,
leaving them blind to malicious intent that emerges over a sequence of
seemingly benign instructions. We therefore propose adding an external monitor
that observes the conversation at a higher granularity. To facilitate our study
of monitoring decomposition attacks, we curate the largest and most diverse
dataset to date, including question-answering, text-to-image, and agentic
tasks. We verify our datasets by testing them on frontier LLMs and show an 87%
attack success rate on average on GPT-4o. This confirms that decomposition
attack is broadly effective. Additionally, we find that random tasks can be
injected into the decomposed subtasks to further obfuscate malicious intents.
To defend in real time, we propose a lightweight sequential monitoring
framework that cumulatively evaluates each subtask. We show that a carefully
prompt engineered lightweight monitor achieves a 93% defense success rate,
beating reasoning models like o3 mini as a monitor. Moreover, it remains robust
against random task injection and cuts cost by 90% and latency by 50%. Our
findings suggest that lightweight sequential monitors are highly effective in
mitigating decomposition attacks and are viable in deployment.


http://arxiv.org/abs/2506.10888
Lattice Climber Attack: Adversarial attacks for randomized mixtures of classifiers. (54%)
Lucas Gnecco-Heredia; Benjamin Negrevergne; Yann Chevaleyre
  Finite mixtures of classifiers (a.k.a. randomized ensembles) have been
proposed as a way to improve robustness against adversarial attacks. However,
existing attacks have been shown to not suit this kind of classifier. In this
paper, we discuss the problem of attacking a mixture in a principled way and
introduce two desirable properties of attacks based on a geometrical analysis
of the problem (effectiveness and maximality). We then show that existing
attacks do not meet both of these properties. Finally, we introduce a new
attack called {\em lattice climber attack} with theoretical guarantees in the
binary linear setting, and demonstrate its performance by conducting
experiments on synthetic and real datasets.


http://arxiv.org/abs/2506.11373
Deception Against Data-Driven Linear-Quadratic Control. (16%)
Filippos Fotiadis; Aris Kanellopoulos; Kyriakos G. Vamvoudakis; Ufuk Topcu
  Deception is a common defense mechanism against adversaries with an
information disadvantage. It can force such adversaries to select suboptimal
policies for a defender's benefit. We consider a setting where an adversary
tries to learn the optimal linear-quadratic attack against a system, the
dynamics of which it does not know. On the other end, a defender who knows its
dynamics exploits its information advantage and injects a deceptive input into
the system to mislead the adversary. The defender's aim is to then
strategically design this deceptive input: it should force the adversary to
learn, as closely as possible, a pre-selected attack that is different from the
optimal one. We show that this deception design problem boils down to the
solution of a coupled algebraic Riccati and a Lyapunov equation which, however,
are challenging to tackle analytically. Nevertheless, we use a block successive
over-relaxation algorithm to extract their solution numerically and prove the
algorithm's convergence under certain conditions. We perform simulations on a
benchmark aircraft, where we showcase how the proposed algorithm can mislead
adversaries into learning attacks that are less performance-degrading.


http://arxiv.org/abs/2506.10831
Efficiency Robustness of Dynamic Deep Learning Systems. (12%)
Ravishka Rathnasuriya; Tingxi Li; Zexin Xu; Zihe Song; Mirazul Haque; Simin Chen; Wei Yang
  Deep Learning Systems (DLSs) are increasingly deployed in real-time
applications, including those in resourceconstrained environments such as
mobile and IoT devices. To address efficiency challenges, Dynamic Deep Learning
Systems (DDLSs) adapt inference computation based on input complexity, reducing
overhead. While this dynamic behavior improves efficiency, such behavior
introduces new attack surfaces. In particular, efficiency adversarial attacks
exploit these dynamic mechanisms to degrade system performance. This paper
systematically explores efficiency robustness of DDLSs, presenting the first
comprehensive taxonomy of efficiency attacks. We categorize these attacks based
on three dynamic behaviors: (i) attacks on dynamic computations per inference,
(ii) attacks on dynamic inference iterations, and (iii) attacks on dynamic
output production for downstream tasks. Through an in-depth evaluation, we
analyze adversarial strategies that target DDLSs efficiency and identify key
challenges in securing these systems. In addition, we investigate existing
defense mechanisms, demonstrating their limitations against increasingly
popular efficiency attacks and the necessity for novel mitigation strategies to
secure future adaptive DDLSs.


http://arxiv.org/abs/2506.10776
ME: Trigger Element Combination Backdoor Attack on Copyright Infringement. (10%)
Feiyu Yang; Siyuan Liang; Aishan Liu; Dacheng Tao
  The capability of generative diffusion models (DMs) like Stable Diffusion
(SD) in replicating training data could be taken advantage of by attackers to
launch the Copyright Infringement Attack, with duplicated poisoned image-text
pairs. SilentBadDiffusion (SBD) is a method proposed recently, which shew
outstanding performance in attacking SD in text-to-image tasks. However, the
feasible data resources in this area are still limited, some of them are even
constrained or prohibited due to the issues like copyright ownership or
inappropriate contents; And not all of the images in current datasets are
suitable for the proposed attacking methods; Besides, the state-of-the-art
(SoTA) performance of SBD is far from ideal when few generated poisoning
samples could be adopted for attacks. In this paper, we raised new datasets
accessible for researching in attacks like SBD, and proposed Multi-Element (ME)
attack method based on SBD by increasing the number of poisonous visual-text
elements per poisoned sample to enhance the ability of attacking, while
importing Discrete Cosine Transform (DCT) for the poisoned samples to maintain
the stealthiness. The Copyright Infringement Rate (CIR) / First Attack Epoch
(FAE) we got on the two new datasets were 16.78% / 39.50 and 51.20% / 23.60,
respectively close to or even outperformed benchmark Pokemon and Mijourney
datasets. In condition of low subsampling ratio (5%, 6 poisoned samples), MESI
and DCT earned CIR / FAE of 0.23% / 84.00 and 12.73% / 65.50, both better than
original SBD, which failed to attack at all.


http://arxiv.org/abs/2506.10463
Starting Positions Matter: A Study on Better Weight Initialization for Neural Network Quantization. (1%)
Stone Yun; Alexander Wong
  Deep neural network (DNN) quantization for fast, efficient inference has been
an important tool in limiting the cost of machine learning (ML) model
inference. Quantization-specific model development techniques such as
regularization, quantization-aware training, and quantization-robustness
penalties have served to greatly boost the accuracy and robustness of modern
DNNs. However, very little exploration has been done on improving the initial
conditions of DNN training for quantization. Just as random weight
initialization has been shown to significantly impact test accuracy of floating
point models, it would make sense that different weight initialization methods
impact quantization robustness of trained models. We present an extensive study
examining the effects of different weight initializations on a variety of CNN
building blocks commonly used in efficient CNNs. This analysis reveals that
even with varying CNN architectures, the choice of random weight initializer
can significantly affect final quantization robustness. Next, we explore a new
method for quantization-robust CNN initialization -- using Graph Hypernetworks
(GHN) to predict parameters of quantized DNNs. Besides showing that
GHN-predicted parameters are quantization-robust after regular float32
pretraining (of the GHN), we find that finetuning GHNs to predict parameters
for quantized graphs (which we call GHN-QAT) can further improve quantized
accuracy of CNNs. Notably, GHN-QAT shows significant accuracy improvements for
even 4-bit quantization and better-than-random accuracy for 2-bits. To the best
of our knowledge, this is the first in-depth study on quantization-aware DNN
weight initialization. GHN-QAT offers a novel approach to quantized DNN model
design. Future investigations, such as using GHN-QAT-initialized parameters for
quantization-aware training, can further streamline the DNN quantization
process.


http://arxiv.org/abs/2506.10424
SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks. (1%)
Kaiyuan Zhang; Siyuan Cheng; Hanxi Guo; Yuetian Chen; Zian Su; Shengwei An; Yuntao Du; Charles Fleming; Ashish Kundu; Xiangyu Zhang; Ninghui Li
  Large language models (LLMs) have achieved remarkable success and are widely
adopted for diverse applications. However, fine-tuning these models often
involves private or sensitive information, raising critical privacy concerns.
In this work, we conduct the first comprehensive study evaluating the
vulnerability of fine-tuned LLMs to membership inference attacks (MIAs). Our
empirical analysis demonstrates that MIAs exploit the loss reduction during
fine-tuning, making them highly effective in revealing membership information.
These findings motivate the development of our defense. We propose SOFT
(\textbf{S}elective data \textbf{O}bfuscation in LLM
\textbf{F}ine-\textbf{T}uning), a novel defense technique that mitigates
privacy leakage by leveraging influential data selection with an adjustable
parameter to balance utility preservation and privacy protection. Our extensive
experiments span six diverse domains and multiple LLM architectures and scales.
Results show that SOFT effectively reduces privacy risks while maintaining
competitive model performance, offering a practical and scalable solution to
safeguard sensitive information in fine-tuned LLMs.


http://arxiv.org/abs/2506.11415
Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs. (1%)
Linlin Wang; Tianqing Zhu; Laiqiao Qin; Longxiang Gao; Wanlei Zhou
  In Large Language Models, Retrieval-Augmented Generation (RAG) systems can
significantly enhance the performance of large language models by integrating
external knowledge. However, RAG also introduces new security risks. Existing
research focuses mainly on how poisoning attacks in RAG systems affect model
output quality, overlooking their potential to amplify model biases. For
example, when querying about domestic violence victims, a compromised RAG
system might preferentially retrieve documents depicting women as victims,
causing the model to generate outputs that perpetuate gender stereotypes even
when the original query is gender neutral. To show the impact of the bias, this
paper proposes a Bias Retrieval and Reward Attack (BRRA) framework, which
systematically investigates attack pathways that amplify language model biases
through a RAG system manipulation. We design an adversarial document generation
method based on multi-objective reward functions, employ subspace projection
techniques to manipulate retrieval results, and construct a cyclic feedback
mechanism for continuous bias amplification. Experiments on multiple mainstream
large language models demonstrate that BRRA attacks can significantly enhance
model biases in dimensions. In addition, we explore a dual stage defense
mechanism to effectively mitigate the impacts of the attack. This study reveals
that poisoning attacks in RAG systems directly amplify model output biases and
clarifies the relationship between RAG system security and model fairness. This
novel potential attack indicates that we need to keep an eye on the fairness
issues of the RAG system.


http://arxiv.org/abs/2506.11413
Byzantine Outside, Curious Inside: Reconstructing Data Through Malicious Updates. (1%)
Kai Yue; Richeng Jin; Chau-Wai Wong; Huaiyu Dai
  Federated learning (FL) enables decentralized machine learning without
sharing raw data, allowing multiple clients to collaboratively learn a global
model. However, studies reveal that privacy leakage is possible under commonly
adopted FL protocols. In particular, a server with access to client gradients
can synthesize data resembling the clients' training data. In this paper, we
introduce a novel threat model in FL, named the maliciously curious client,
where a client manipulates its own gradients with the goal of inferring private
data from peers. This attacker uniquely exploits the strength of a Byzantine
adversary, traditionally aimed at undermining model robustness, and repurposes
it to facilitate data reconstruction attack. We begin by formally defining this
novel client-side threat model and providing a theoretical analysis that
demonstrates its ability to achieve significant reconstruction success during
FL training. To demonstrate its practical impact, we further develop a
reconstruction algorithm that combines gradient inversion with malicious update
strategies. Our analysis and experimental results reveal a critical blind spot
in FL defenses: both server-side robust aggregation and client-side privacy
mechanisms may fail against our proposed attack. Surprisingly, standard server-
and client-side defenses designed to enhance robustness or privacy may
unintentionally amplify data leakage. Compared to the baseline approach, a
mistakenly used defense may instead improve the reconstructed image quality by
10-15%.


http://arxiv.org/abs/2506.09896
A look at adversarial attacks on radio waveforms from discrete latent space. (99%)
Attanasia Garuso; Silvija Kokalj-Filipovic; Yagna Kaasaragadda
  Having designed a VQVAE that maps digital radio waveforms into discrete
latent space, and yields a perfectly classifiable reconstruction of the
original data, we here analyze the attack suppressing properties of VQVAE when
an adversarial attack is performed on high-SNR radio-frequency (RF)
data-points. To target amplitude modulations from a subset of digitally
modulated waveform classes, we first create adversarial attacks that preserve
the phase between the in-phase and quadrature component whose values are
adversarially changed. We compare them with adversarial attacks of the same
intensity where phase is not preserved. We test the classification accuracy of
such adversarial examples on a classifier trained to deliver 100% accuracy on
the original data. To assess the ability of VQVAE to suppress the strength of
the attack, we evaluate the classifier accuracy on the reconstructions by VQVAE
of the adversarial datapoints and show that VQVAE substantially decreases the
effectiveness of the attack. We also compare the I/Q plane diagram of the
attacked data, their reconstructions and the original data. Finally, using
multiple methods and metrics, we compare the probability distribution of the
VQVAE latent space with and without attack. Varying the attack strength, we
observe interesting properties of the discrete space, which may help detect the
attacks.


http://arxiv.org/abs/2506.09443
LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge. (92%)
Songze Li; Chuokun Xu; Jiaying Wang; Xueluan Gong; Chen Chen; Jirui Zhang; Jun Wang; Kwok-Yan Lam; Shouling Ji
  Large Language Models (LLMs) have demonstrated remarkable intelligence across
various tasks, which has inspired the development and widespread adoption of
LLM-as-a-Judge systems for automated model testing, such as red teaming and
benchmarking. However, these systems are susceptible to adversarial attacks
that can manipulate evaluation outcomes, raising concerns about their
robustness and, consequently, their trustworthiness. Existing evaluation
methods adopted by LLM-based judges are often piecemeal and lack a unified
framework for comprehensive assessment. Furthermore, prompt template and model
selections for improving judge robustness have been rarely explored, and their
performance in real-world settings remains largely unverified. To address these
gaps, we introduce RobustJudge, a fully automated and scalable framework
designed to systematically evaluate the robustness of LLM-as-a-Judge systems.
RobustJudge investigates the impact of attack methods and defense strategies
(RQ1), explores the influence of prompt template and model selection (RQ2), and
assesses the robustness of real-world LLM-as-a-Judge applications (RQ3).Our
main findings are: (1) LLM-as-a-Judge systems are still vulnerable to a range
of adversarial attacks, including Combined Attack and PAIR, while defense
mechanisms such as Re-tokenization and LLM-based Detectors offer improved
protection; (2) Robustness is highly sensitive to the choice of prompt template
and judge models. Our proposed prompt template optimization method can improve
robustness, and JudgeLM-13B demonstrates strong performance as a robust
open-source judge; (3) Applying RobustJudge to Alibaba's PAI platform reveals
previously unreported vulnerabilities. The source code of RobustJudge is
provided at https://github.com/S3IC-Lab/RobustJudge.


http://arxiv.org/abs/2506.10047
GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models. (87%)
Zilong Wang; Xiang Zheng; Xiaosen Wang; Bo Wang; Xingjun Ma; Yu-Gang Jiang
  Text-to-image (T2I) models such as Stable Diffusion have advanced rapidly and
are now widely used in content creation. However, these models can be misused
to generate harmful content, including nudity or violence, posing significant
safety risks. While most platforms employ content moderation systems,
underlying vulnerabilities can still be exploited by determined adversaries.
Recent research on red-teaming and adversarial attacks against T2I models has
notable limitations: some studies successfully generate highly toxic images but
use adversarial prompts that are easily detected and blocked by safety filters,
while others focus on bypassing safety mechanisms but fail to produce genuinely
harmful outputs, neglecting the discovery of truly high-risk prompts.
Consequently, there remains a lack of reliable tools for evaluating the safety
of defended T2I models. To address this gap, we propose GenBreak, a framework
that fine-tunes a red-team large language model (LLM) to systematically explore
underlying vulnerabilities in T2I generators. Our approach combines supervised
fine-tuning on curated datasets with reinforcement learning via interaction
with a surrogate T2I model. By integrating multiple reward signals, we guide
the LLM to craft adversarial prompts that enhance both evasion capability and
image toxicity, while maintaining semantic coherence and diversity. These
prompts demonstrate strong effectiveness in black-box attacks against
commercial T2I generators, revealing practical and concerning safety
weaknesses.


http://arxiv.org/abs/2506.09640
Evasion Attacks Against Bayesian Predictive Models. (76%)
Pablo G. Arce; Roi Naveiro; David Ríos Insua
  There is an increasing interest in analyzing the behavior of machine learning
systems against adversarial attacks. However, most of the research in
adversarial machine learning has focused on studying weaknesses against evasion
or poisoning attacks to predictive models in classical setups, with the
susceptibility of Bayesian predictive models to attacks remaining
underexplored. This paper introduces a general methodology for designing
optimal evasion attacks against such models. We investigate two adversarial
objectives: perturbing specific point predictions and altering the entire
posterior predictive distribution. For both scenarios, we propose novel
gradient-based attacks and study their implementation and properties in various
computational setups.


http://arxiv.org/abs/2506.09538
AngleRoCL: Angle-Robust Concept Learning for Physically View-Invariant T2I Adversarial Patches. (22%)
Wenjun Ji; Yuxiang Fu; Luyang Ying; Deng-Ping Fan; Yuyi Wang; Ming-Ming Cheng; Ivor Tsang; Qing Guo
  Cutting-edge works have demonstrated that text-to-image (T2I) diffusion
models can generate adversarial patches that mislead state-of-the-art object
detectors in the physical world, revealing detectors' vulnerabilities and
risks. However, these methods neglect the T2I patches' attack effectiveness
when observed from different views in the physical world (i.e., angle
robustness of the T2I adversarial patches). In this paper, we study the angle
robustness of T2I adversarial patches comprehensively, revealing their
angle-robust issues, demonstrating that texts affect the angle robustness of
generated patches significantly, and task-specific linguistic instructions fail
to enhance the angle robustness. Motivated by the studies, we introduce
Angle-Robust Concept Learning (AngleRoCL), a simple and flexible approach that
learns a generalizable concept (i.e., text embeddings in implementation)
representing the capability of generating angle-robust patches. The learned
concept can be incorporated into textual prompts and guides T2I models to
generate patches with their attack effectiveness inherently resistant to
viewpoint variations. Through extensive simulation and physical-world
experiments on five SOTA detectors across multiple views, we demonstrate that
AngleRoCL significantly enhances the angle robustness of T2I adversarial
patches compared to baseline methods. Our patches maintain high attack success
rates even under challenging viewing conditions, with over 50% average relative
improvement in attack effectiveness across multiple angles. This research
advances the understanding of physically angle-robust patches and provides
insights into the relationship between textual concepts and physical properties
in T2I-generated contents.


http://arxiv.org/abs/2506.09923
Apollo: A Posteriori Label-Only Membership Inference Attack Towards Machine Unlearning. (9%)
Liou Tang; James Joshi; Ashish Kundu
  Machine Unlearning (MU) aims to update Machine Learning (ML) models following
requests to remove training samples and their influences on a trained model
efficiently without retraining the original ML model from scratch. While MU
itself has been employed to provide privacy protection and regulatory
compliance, it can also increase the attack surface of the model. Existing
privacy inference attacks towards MU that aim to infer properties of the
unlearned set rely on the weaker threat model that assumes the attacker has
access to both the unlearned model and the original model, limiting their
feasibility toward real-life scenarios. We propose a novel privacy attack, A
Posteriori Label-Only Membership Inference Attack towards MU, Apollo, that
infers whether a data sample has been unlearned, following a strict threat
model where an adversary has access to the label-output of the unlearned model
only. We demonstrate that our proposed attack, while requiring less access to
the target model compared to previous attacks, can achieve relatively high
precision on the membership status of the unlearned samples.


http://arxiv.org/abs/2506.09408
Token Constraint Decoding Improves Robustness on Question Answering for Large Language Models. (9%)
Jui-Ming Yao; Hao-Yuan Chen; Zi-Xian Tang; Bing-Jia Tan; Sheng-Wei Peng; Bing-Cheng Xie; Shun-Feng Su
  Large Language Models (LLMs) have demonstrated impressive performance on
multiple-choice question answering (MCQA) benchmarks, yet they remain highly
vulnerable to minor input perturbations. In this paper, we introduce and
evaluate Token Constraint Decoding (TCD). This simple yet effective
inference-time algorithm enforces alignment between token-level predictions to
enhance robustness in noisy settings. Through extensive experiments on
CommonsenseQA, MMLU, and MMLU-Pro, we show that TCD, especially when paired
with prompt engineering (PE) fixes, significantly restores performance degraded
by input noise, yielding up to +39\% absolute gains for weaker models like
Gemma3 1B. Penalty sweep analyses further reveal that TCD implicitly
regularizes overconfident outputs, with different models requiring distinct
penalty schedules to maximize resilience. Our findings establish TCD as a
practical, model-agnostic approach for improving reasoning stability under
real-world imperfections and pave the way for more reliable deployment of LLMs
in safety-critical or user-facing applications.


http://arxiv.org/abs/2506.09562
TooBadRL: Trigger Optimization to Boost Effectiveness of Backdoor Attacks on Deep Reinforcement Learning. (2%)
Songze Li; Mingxuan Zhang; Kang Wei; Shouling Ji
  Deep reinforcement learning (DRL) has achieved remarkable success in a wide
range of sequential decision-making domains, including robotics, healthcare,
smart grids, and finance. Recent research demonstrates that attackers can
efficiently exploit system vulnerabilities during the training phase to execute
backdoor attacks, producing malicious actions when specific trigger patterns
are present in the state observations. However, most existing backdoor attacks
rely primarily on simplistic and heuristic trigger configurations, overlooking
the potential efficacy of trigger optimization. To address this gap, we
introduce TooBadRL (Trigger Optimization to Boost Effectiveness of Backdoor
Attacks on DRL), the first framework to systematically optimize DRL backdoor
triggers along three critical axes, i.e., temporal, spatial, and magnitude.
Specifically, we first introduce a performance-aware adaptive freezing
mechanism for injection timing. Then, we formulate dimension selection as a
cooperative game, utilizing Shapley value analysis to identify the most
influential state variable for the injection dimension. Furthermore, we propose
a gradient-based adversarial procedure to optimize the injection magnitude
under environment constraints. Evaluations on three mainstream DRL algorithms
and nine benchmark tasks show that TooBadRL significantly improves attack
success rates, while ensuring minimal degradation of normal task performance.
These results highlight the previously underappreciated importance of
principled trigger optimization in DRL backdoor attacks. The source code of
TooBadRL can be found at https://github.com/S3IC-Lab/TooBadRL.


http://arxiv.org/abs/2506.09148
Adversarial Text Generation with Dynamic Contextual Perturbation. (99%)
Hetvi Waghela; Jaydip Sen; Sneha Rakshit; Subhasis Dasgupta
  Adversarial attacks on Natural Language Processing (NLP) models expose
vulnerabilities by introducing subtle perturbations to input text, often
leading to misclassification while maintaining human readability. Existing
methods typically focus on word-level or local text segment alterations,
overlooking the broader context, which results in detectable or semantically
inconsistent perturbations. We propose a novel adversarial text attack scheme
named Dynamic Contextual Perturbation (DCP). DCP dynamically generates
context-aware perturbations across sentences, paragraphs, and documents,
ensuring semantic fidelity and fluency. Leveraging the capabilities of
pre-trained language models, DCP iteratively refines perturbations through an
adversarial objective function that balances the dual objectives of inducing
model misclassification and preserving the naturalness of the text. This
comprehensive approach allows DCP to produce more sophisticated and effective
adversarial examples that better mimic natural language patterns. Our
experimental results, conducted on various NLP models and datasets, demonstrate
the efficacy of DCP in challenging the robustness of state-of-the-art NLP
systems. By integrating dynamic contextual analysis, DCP significantly enhances
the subtlety and impact of adversarial attacks. This study highlights the
critical role of context in adversarial attacks and lays the groundwork for
creating more robust NLP systems capable of withstanding sophisticated
adversarial strategies.


http://arxiv.org/abs/2506.08961
Towards Robust Deep Reinforcement Learning against Environmental State Perturbation. (95%)
Chenxu Wang; Huaping Liu
  Adversarial attacks and robustness in Deep Reinforcement Learning (DRL) have
been widely studied in various threat models; however, few consider
environmental state perturbations, which are natural in embodied scenarios. To
improve the robustness of DRL agents, we formulate the problem of environmental
state perturbation, introducing a preliminary non-targeted attack method as a
calibration adversary, and then propose a defense framework, named Boosted
Adversarial Training (BAT), which first tunes the agents via supervised
learning to avoid catastrophic failure and subsequently adversarially trains
the agent with reinforcement learning. Extensive experimental results
substantiate the vulnerability of mainstream agents under environmental state
perturbations and the effectiveness of our proposed attack. The defense results
demonstrate that while existing robust reinforcement learning algorithms may
not be suitable, our BAT framework can significantly enhance the robustness of
agents against environmental state perturbations across various situations.


http://arxiv.org/abs/2506.08885
AdversariaL attacK sAfety aLIgnment(ALKALI): Safeguarding LLMs through GRACE: Geometric Representation-Aware Contrastive Enhancement- Introducing Adversarial Vulnerability Quality Index (AVQI). (92%)
Danush Khanna; Krishna Kumar; Basab Ghosh; Vinija Jain; Vasu Sharma; Aman Chadha; Amitava Das
  Adversarial threats against LLMs are escalating faster than current defenses
can adapt. We expose a critical geometric blind spot in alignment: adversarial
prompts exploit latent camouflage, embedding perilously close to the safe
representation manifold while encoding unsafe intent thereby evading surface
level defenses like Direct Preference Optimization (DPO), which remain blind to
the latent geometry. We introduce ALKALI, the first rigorously curated
adversarial benchmark and the most comprehensive to date spanning 9,000 prompts
across three macro categories, six subtypes, and fifteen attack families.
Evaluation of 21 leading LLMs reveals alarmingly high Attack Success Rates
(ASRs) across both open and closed source models, exposing an underlying
vulnerability we term latent camouflage, a structural blind spot where
adversarial completions mimic the latent geometry of safe ones. To mitigate
this vulnerability, we introduce GRACE - Geometric Representation Aware
Contrastive Enhancement, an alignment framework coupling preference learning
with latent space regularization. GRACE enforces two constraints: latent
separation between safe and adversarial completions, and adversarial cohesion
among unsafe and jailbreak behaviors. These operate over layerwise pooled
embeddings guided by a learned attention profile, reshaping internal geometry
without modifying the base model, and achieve up to 39% ASR reduction.
Moreover, we introduce AVQI, a geometry aware metric that quantifies latent
alignment failure via cluster separation and compactness. AVQI reveals when
unsafe completions mimic the geometry of safe ones, offering a principled lens
into how models internally encode safety. We make the code publicly available
at https://anonymous.4open.science/r/alkali-B416/README.md.


http://arxiv.org/abs/2506.09237
PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies. (82%)
Mojtaba Nafez; Amirhossein Koochakian; Arad Maleki; Jafar Habibi; Mohammad Hossein Rohban
  Anomaly Detection (AD) and Anomaly Localization (AL) are crucial in fields
that demand high reliability, such as medical imaging and industrial
monitoring. However, current AD and AL approaches are often susceptible to
adversarial attacks due to limitations in training data, which typically
include only normal, unlabeled samples. This study introduces PatchGuard, an
adversarially robust AD and AL method that incorporates pseudo anomalies with
localization masks within a Vision Transformer (ViT)-based architecture to
address these vulnerabilities. We begin by examining the essential properties
of pseudo anomalies, and follow it by providing theoretical insights into the
attention mechanisms required to enhance the adversarial robustness of AD and
AL systems. We then present our approach, which leverages Foreground-Aware
Pseudo-Anomalies to overcome the deficiencies of previous anomaly-aware
methods. Our method incorporates these crafted pseudo-anomaly samples into a
ViT-based framework, with adversarial training guided by a novel loss function
designed to improve model robustness, as supported by our theoretical analysis.
Experimental results on well-established industrial and medical datasets
demonstrate that PatchGuard significantly outperforms previous methods in
adversarial settings, achieving performance gains of $53.2\%$ in AD and
$68.5\%$ in AL, while also maintaining competitive accuracy in non-adversarial
settings. The code repository is available at
https://github.com/rohban-lab/PatchGuard .


http://arxiv.org/abs/2506.08482
One Patch to Rule Them All: Transforming Static Patches into Dynamic Attacks in the Physical World. (80%)
Xingshuo Han; Chen Ling; Shiyi Yao; Haozhao Wang; Hangcheng Liu; Yutong Wu; Shengmin Xu; Changhai Ou; Xinyi Huang; Tianwei Zhang
  Numerous methods have been proposed to generate physical adversarial patches
(PAPs) against real-world machine learning systems. However, each existing PAP
typically supports only a single, fixed attack goal, and switching to a
different objective requires re-generating and re-deploying a new PAP. This
rigidity limits their practicality in dynamic environments like autonomous
driving, where traffic conditions and attack goals can change rapidly. For
example, if no obstacles are present around the target vehicle, the attack may
fail to cause meaningful consequences.
  To overcome this limitation, we propose SwitchPatch, a novel PAP that is
static yet enables dynamic and controllable attack outcomes based on real-time
scenarios. Attackers can alter pre-defined conditions, e.g., by projecting
different natural-color lights onto SwitchPatch to seamlessly switch between
attack goals. Unlike prior work, SwitchPatch does not require re-generation or
re-deployment for different objectives, significantly reducing cost and
complexity. Furthermore, SwitchPatch remains benign when the enabling
conditions are absent, enhancing its stealth.
  We evaluate SwitchPatch on two key tasks: traffic sign recognition
(classification and detection) and depth estimation. First, we conduct
theoretical analysis and empirical studies to demonstrate the feasibility of
SwitchPatch and explore how many goals it can support using techniques like
color light projection and occlusion. Second, we perform simulation-based
experiments and ablation studies to verify its effectiveness and
transferability. Third, we conduct outdoor tests using a Unmanned Ground
Vehicle (UGV) to confirm its robustness in the physical world. Overall,
SwitchPatch introduces a flexible and practical adversarial strategy that can
be adapted to diverse tasks and real-world conditions.


http://arxiv.org/abs/2506.09348
Adversarial Surrogate Risk Bounds for Binary Classification. (47%)
Natalie S. Frank
  A central concern in classification is the vulnerability of machine learning
models to adversarial attacks. Adversarial training is one of the most popular
techniques for training robust classifiers, which involves minimizing an
adversarial surrogate risk. Recent work characterized when a minimizing
sequence of an adversarial surrogate risk is also a minimizing sequence of the
adversarial classification risk for binary classification -- a property known
as adversarial consistency. However, these results do not address the rate at
which the adversarial classification risk converges to its optimal value for
such a sequence of functions that minimize the adversarial surrogate. This
paper provides surrogate risk bounds that quantify that convergence rate.
Additionally, we derive distribution-dependent surrogate risk bounds in the
standard (non-adversarial) learning setting, that may be of independent
interest.


http://arxiv.org/abs/2506.11125
ASRJam: Human-Friendly AI Speech Jamming to Prevent Automated Phone Scams. (22%)
Freddie Grabovski; Gilad Gressel; Yisroel Mirsky
  Large Language Models (LLMs), combined with Text-to-Speech (TTS) and
Automatic Speech Recognition (ASR), are increasingly used to automate voice
phishing (vishing) scams. These systems are scalable and convincing, posing a
significant security threat. We identify the ASR transcription step as the most
vulnerable link in the scam pipeline and introduce ASRJam, a proactive defence
framework that injects adversarial perturbations into the victim's audio to
disrupt the attacker's ASR. This breaks the scam's feedback loop without
affecting human callers, who can still understand the conversation. While prior
adversarial audio techniques are often unpleasant and impractical for real-time
use, we also propose EchoGuard, a novel jammer that leverages natural
distortions, such as reverberation and echo, that are disruptive to ASR but
tolerable to humans. To evaluate EchoGuard's effectiveness and usability, we
conducted a 39-person user study comparing it with three state-of-the-art
attacks. Results show that EchoGuard achieved the highest overall utility,
offering the best combination of ASR disruption and human listening experience.


http://arxiv.org/abs/2506.08611
Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation. (22%)
Shiji Zhao; Chi Chen; Ranjie Duan; Xizhe Wang; Xingxing Wei
  Adversarial Training (AT) is widely recognized as an effective approach to
enhance the adversarial robustness of Deep Neural Networks. As a variant of AT,
Adversarial Robustness Distillation (ARD) has shown outstanding performance in
enhancing the robustness of small models. However, both AT and ARD face robust
fairness issue: these models tend to display strong adversarial robustness
against some classes (easy classes) while demonstrating weak adversarial
robustness against others (hard classes). This paper explores the underlying
factors of this problem and points out the smoothness degree of soft labels for
different classes significantly impacts the robust fairness from both empirical
observation and theoretical analysis. Based on the above exploration, we
propose Anti-Bias Soft Label Distillation (ABSLD) within the Knowledge
Distillation framework to enhance the adversarial robust fairness.
Specifically, ABSLD adaptively reduces the student's error risk gap between
different classes, which is accomplished by adjusting the class-wise smoothness
degree of teacher's soft labels during the training process, and the adjustment
is managed by assigning varying temperatures to different classes.
Additionally, as a label-based approach, ABSLD is highly adaptable and can be
integrated with the sample-based methods. Extensive experiments demonstrate
ABSLD outperforms state-of-the-art methods on the comprehensive performance of
robustness and fairness.


http://arxiv.org/abs/2506.08514
DiffGradCAM: A Universal Class Activation Map Resistant to Adversarial Training. (13%)
Jacob Piland; Chris Sweet; Adam Czakja
  Class Activation Mapping (CAM) and its gradient-based variants (e.g.,
GradCAM) have become standard tools for explaining Convolutional Neural Network
(CNN) predictions. However, these approaches typically focus on individual
logits, while for neural networks using softmax, the class membership
probability estimates depend \textit{only} on the \textit{differences} between
logits, not on their absolute values. This disconnect leaves standard CAMs
vulnerable to adversarial manipulation, such as passive fooling, where a model
is trained to produce misleading CAMs without affecting decision performance.
We introduce \textbf{Salience-Hoax Activation Maps (SHAMs)}, an
\emph{entropy-aware form of passive fooling} that serves as a benchmark for CAM
robustness under adversarial conditions. To address the passive fooling
vulnerability, we then propose \textbf{DiffGradCAM}, a novel, lightweight, and
contrastive approach to class activation mapping that is both non-suceptible to
passive fooling, but also matches the output of standard CAM methods such as
GradCAM in the non-adversarial case. Together, SHAM and DiffGradCAM establish a
new framework for probing and improving the robustness of saliency-based
explanations. We validate both contributions across multi-class tasks with few
and many classes.


http://arxiv.org/abs/2506.17265
Does Multimodal Large Language Model Truly Unlearn? Stealthy MLLM Unlearning Attack. (12%)
Xianren Zhang; Hui Liu; Delvin Ce Zhang; Xianfeng Tang; Qi He; Dongwon Lee; Suhang Wang
Multimodal Large Language Models (MLLMs) trained on massive data may memorize sensitive personal information and photos, posing serious privacy risks. To mitigate this, MLLM unlearning methods are proposed, which fine-tune MLLMs to reduce the ``forget'' sensitive information. However, it remains unclear whether the knowledge has been truly forgotten or just hidden in the model. Therefore, we propose to study a novel problem of LLM unlearning attack, which aims to recover the unlearned knowledge of an unlearned LLM. To achieve the goal, we propose a novel framework Stealthy Unlearning Attack (SUA) framework that learns a universal noise pattern. When applied to input images, this noise can trigger the model to reveal unlearned content. While pixel-level perturbations may be visually subtle, they can be detected in the semantic embedding space, making such attacks vulnerable to potential defenses. To improve stealthiness, we introduce an embedding alignment loss that minimizes the difference between the perturbed and denoised image embeddings, ensuring the attack is semantically unnoticeable. Experimental results show that SUA can effectively recover unlearned information from MLLMs. Furthermore, the learned noise generalizes well: a single perturbation trained on a subset of samples can reveal forgotten content in unseen images. This indicates that knowledge reappearance is not an occasional failure, but a consistent behavior.

http://arxiv.org/abs/2506.08473
AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basin. (1%)
Shuo Yang; Qihui Zhang; Yuyang Liu; Yue Huang; Xiaojun Jia; Kunpeng Ning; Jiayu Yao; Jigang Wang; Hailiang Dai; Yibing Song; Li Yuan
  Large language models (LLMs) are vulnerable to safety risks during
fine-tuning, where small amounts of malicious or harmless data can compromise
safeguards. In this paper, building on the concept of alignment direction --
defined by the weight difference between aligned and unaligned models -- we
observe that perturbations along this direction preserve model safety. In
contrast, perturbations along directions orthogonal to this alignment are
strongly linked to harmful direction perturbations, rapidly degrading safety
and framing the parameter space as a narrow safety basin. Based on this
insight, we propose a methodology for safety fine-tuning called AsFT (Anchoring
Safety in Fine-Tuning), which integrates a regularization term into the
training objective. This term uses the alignment direction as an anchor to
suppress updates in harmful directions, ensuring that fine-tuning is
constrained within the narrow safety basin. Extensive experiments on multiple
datasets show that AsFT outperforms Safe LoRA, reducing harmful behavior by
7.60 percent, improving model performance by 3.44 percent, and maintaining
robust performance across various experimental settings. Code is available at
https://github.com/PKU-YuanGroup/AsFT


http://arxiv.org/abs/2506.08837
Design Patterns for Securing LLM Agents against Prompt Injections. (1%)
Luca Beurer-Kellner; Beat Buesser Ana-Maria Creţu; Edoardo Debenedetti; Daniel Dobos; Daniel Fabian; Marc Fischer; David Froelicher; Kathrin Grosse; Daniel Naeff; Ezinwanne Ozoani; Andrew Paverd; Florian Tramèr; Václav Volhejn
  As AI agents powered by Large Language Models (LLMs) become increasingly
versatile and capable of addressing a broad spectrum of tasks, ensuring their
security has become a critical challenge. Among the most pressing threats are
prompt injection attacks, which exploit the agent's resilience on natural
language inputs -- an especially dangerous threat when agents are granted tool
access or handle sensitive information. In this work, we propose a set of
principled design patterns for building AI agents with provable resistance to
prompt injection. We systematically analyze these patterns, discuss their
trade-offs in terms of utility and security, and illustrate their real-world
applicability through a series of case studies.


http://arxiv.org/abs/2506.08602
WGLE:Backdoor-free and Multi-bit Black-box Watermarking for Graph Neural Networks. (1%)
Tingzhi Li; Xuefeng Liu
  Graph Neural Networks (GNNs) are increasingly deployed in graph-related
applications, making ownership verification critical to protect their
intellectual property against model theft. Fingerprinting and black-box
watermarking are two main methods. However, the former relies on determining
model similarity, which is computationally expensive and prone to ownership
collisions after model post-processing such as model pruning or fine-tuning.
The latter embeds backdoors, exposing watermarked models to the risk of
backdoor attacks. Moreover, both methods enable ownership verification but do
not convey additional information. As a result, each distributed model requires
a unique trigger graph, and all trigger graphs must be used to query the
suspect model during verification. Multiple queries increase the financial cost
and the risk of detection.
  To address these challenges, this paper proposes WGLE, a novel black-box
watermarking paradigm for GNNs that enables embedding the multi-bit string as
the ownership information without using backdoors. WGLE builds on a key insight
we term Layer-wise Distance Difference on an Edge (LDDE), which quantifies the
difference between the feature distance and the prediction distance of two
connected nodes. By predefining positive or negative LDDE values for multiple
selected edges, WGLE embeds the watermark encoding the intended information
without introducing incorrect mappings that compromise the primary task. WGLE
is evaluated on six public datasets and six mainstream GNN architectures along
with state-of-the-art methods. The results show that WGLE achieves 100%
ownership verification accuracy, an average fidelity degradation of 0.85%,
comparable robustness against potential attacks, and low embedding overhead.
The code is available in the repository.


http://arxiv.org/abs/2506.07942
Adversarial Attack Classification and Robustness Testing for Large Language Models for Code. (99%)
Yang Liu; Armstrong Foundjem; Foutse Khomh; Heng Li
  Large Language Models (LLMs) have become vital tools in software development
tasks such as code generation, completion, and analysis. As their integration
into workflows deepens, ensuring robustness against vulnerabilities especially
those triggered by diverse or adversarial inputs becomes increasingly
important. Such vulnerabilities may lead to incorrect or insecure code
generation when models encounter perturbed task descriptions, code, or
comments. Prior research often overlooks the role of natural language in
guiding code tasks. This study investigates how adversarial perturbations in
natural language inputs including prompts, comments, and descriptions affect
LLMs for Code (LLM4Code). It examines the effects of perturbations at the
character, word, and sentence levels to identify the most impactful
vulnerabilities. We analyzed multiple projects (e.g., ReCode, OpenAttack) and
datasets (e.g., HumanEval, MBPP), establishing a taxonomy of adversarial
attacks. The first dimension classifies the input type code, prompts, or
comments while the second dimension focuses on granularity: character, word, or
sentence-level changes. We adopted a mixed-methods approach, combining
quantitative performance metrics with qualitative vulnerability analysis.
LLM4Code models show varying robustness across perturbation types.
Sentence-level attacks were least effective, suggesting models are resilient to
broader contextual changes. In contrast, word-level perturbations posed serious
challenges, exposing semantic vulnerabilities. Character-level effects varied,
showing model sensitivity to subtle syntactic deviations.Our study offers a
structured framework for testing LLM4Code robustness and emphasizes the
critical role of natural language in adversarial evaluation. Improving model
resilience to semantic-level disruptions is essential for secure and reliable
code-generation systems.


http://arxiv.org/abs/2506.07804
Enhancing Adversarial Robustness with Conformal Prediction: A Framework for Guaranteed Model Reliability. (98%)
Jie Bao; Chuangyin Dang; Rui Luo; Hanwei Zhang; Zhixin Zhou
  As deep learning models are increasingly deployed in high-risk applications,
robust defenses against adversarial attacks and reliable performance guarantees
become paramount. Moreover, accuracy alone does not provide sufficient
assurance or reliable uncertainty estimates for these models. This study
advances adversarial training by leveraging principles from Conformal
Prediction. Specifically, we develop an adversarial attack method, termed OPSA
(OPtimal Size Attack), designed to reduce the efficiency of conformal
prediction at any significance level by maximizing model uncertainty without
requiring coverage guarantees. Correspondingly, we introduce OPSA-AT
(Adversarial Training), a defense strategy that integrates OPSA within a novel
conformal training paradigm. Experimental evaluations demonstrate that our OPSA
attack method induces greater uncertainty compared to baseline approaches for
various defenses. Conversely, our OPSA-AT defensive model significantly
enhances robustness not only against OPSA but also other adversarial attacks,
and maintains reliable prediction. Our findings highlight the effectiveness of
this integrated approach for developing trustworthy and resilient deep learning
models for safety-critical domains. Our code is available at
https://github.com/bjbbbb/Enhancing-Adversarial-Robustness-with-Conformal-Prediction.


http://arxiv.org/abs/2506.08255
SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense. (83%)
Patryk Krukowski; Łukasz Gorczyca; Piotr Helm; Kamil Książek; Przemysław Spurek
  Traditional deep neural networks suffer from several limitations, including
catastrophic forgetting. When models are adapted to new datasets, they tend to
quickly forget previously learned knowledge. Another significant issue is the
lack of robustness to even small perturbations in the input data. In practice,
we can often easily perform adversarial attacks and change the network's
predictions, adding minimal noise to the input. Dedicated architectures and
training procedures can solve each of the above problems separately.
Unfortunately, currently, no model can simultaneously address both catastrophic
forgetting and vulnerability to adversarial attacks. We introduce SHIELD
(Secure Hypernetworks for Incremental Expansion and Learning Defense), a novel
approach that integrates a hypernetwork-based continual learning approach with
interval arithmetic. SHIELD use the hypernetwork to transfer trainable task
embedding vectors into the weights of a target model dedicated to specific
data. This paradigm allows for the dynamic generation of separate networks for
each subtask, while the hypernetwork aggregates and analyzes information across
all tasks. The target model takes in the input a data sample with a defined
interval range, and by creating a hypercube, produces a prediction for the
given range. Therefore, such target models provide strict guarantees against
all possible attacks for data samples within the interval range. Our approach
enhances security without sacrificing network adaptability, addressing the
overlooked challenge of safety in continual learning.


http://arxiv.org/abs/2506.07590
Explore the vulnerability of black-box models via diffusion models. (76%)
Jiacheng Shi; Yanfu Zhang; Huajie Shao; Ashley Gao
  Recent advancements in diffusion models have enabled high-fidelity and
photorealistic image generation across diverse applications. However, these
models also present security and privacy risks, including copyright violations,
sensitive information leakage, and the creation of harmful or offensive content
that could be exploited maliciously. In this study, we uncover a novel security
threat where an attacker leverages diffusion model APIs to generate synthetic
images, which are then used to train a high-performing substitute model. This
enables the attacker to execute model extraction and transfer-based adversarial
attacks on black-box classification models with minimal queries, without
needing access to the original training data. The generated images are
sufficiently high-resolution and diverse to train a substitute model whose
outputs closely match those of the target model. Across the seven benchmarks,
including CIFAR and ImageNet subsets, our method shows an average improvement
of 27.37% over state-of-the-art methods while using just 0.01 times of the
query budget, achieving a 98.68% success rate in adversarial attacks on the
target model.


http://arxiv.org/abs/2506.07467
Circumventing Backdoor Space via Weight Symmetry. (45%)
Jie Peng; Hongwei Yang; Jing Zhao; Hengji Dong; Hui He; Weizhe Zhang; Haoyu He
  Deep neural networks are vulnerable to backdoor attacks, where malicious
behaviors are implanted during training. While existing defenses can
effectively purify compromised models, they typically require labeled data or
specific training procedures, making them difficult to apply beyond supervised
learning settings. Notably, recent studies have shown successful backdoor
attacks across various learning paradigms, highlighting a critical security
concern. To address this gap, we propose Two-stage Symmetry Connectivity (TSC),
a novel backdoor purification defense that operates independently of data
format and requires only a small fraction of clean samples. Through theoretical
analysis, we prove that by leveraging permutation invariance in neural networks
and quadratic mode connectivity, TSC amplifies the loss on poisoned samples
while maintaining bounded clean accuracy. Experiments demonstrate that TSC
achieves robust performance comparable to state-of-the-art methods in
supervised learning scenarios. Furthermore, TSC generalizes to self-supervised
learning frameworks, such as SimCLR and CLIP, maintaining its strong defense
capabilities. Our code is available at https://github.com/JiePeng104/TSC.


http://arxiv.org/abs/2506.07948
TokenBreak: Bypassing Text Classification Models Through Token Manipulation. (38%)
Kasimir Schulz; Kenneth Yeung; Kieran Evans
  Natural Language Processing (NLP) models are used for text-related tasks such
as classification and generation. To complete these tasks, input data is first
tokenized from human-readable text into a format the model can understand,
enabling it to make inferences and understand context. Text classification
models can be implemented to guard against threats such as prompt injection
attacks against Large Language Models (LLMs), toxic input and cybersecurity
risks such as spam emails. In this paper, we introduce TokenBreak: a novel
attack that can bypass these protection models by taking advantage of the
tokenization strategy they use. This attack technique manipulates input text in
such a way that certain models give an incorrect classification. Importantly,
the end target (LLM or email recipient) can still understand and respond to the
manipulated text and therefore be vulnerable to the very attack the protection
model was put in place to prevent. The tokenizer is tied to model architecture,
meaning it is possible to predict whether or not a model is vulnerable to
attack based on family. We also present a defensive strategy as an added layer
of protection that can be implemented without having to retrain the defensive
model.


http://arxiv.org/abs/2506.08336
Your Agent Can Defend Itself against Backdoor Attacks. (38%)
Li Changjiang; Liang Jiacheng; Cao Bochuan; Chen Jinghui; Wang Ting
  Despite their growing adoption across domains, large language model
(LLM)-powered agents face significant security risks from backdoor attacks
during training and fine-tuning. These compromised agents can subsequently be
manipulated to execute malicious operations when presented with specific
triggers in their inputs or environments. To address this pressing risk, we
present ReAgent, a novel defense against a range of backdoor attacks on
LLM-based agents. Intuitively, backdoor attacks often result in inconsistencies
among the user's instruction, the agent's planning, and its execution. Drawing
on this insight, ReAgent employs a two-level approach to detect potential
backdoors. At the execution level, ReAgent verifies consistency between the
agent's thoughts and actions; at the planning level, ReAgent leverages the
agent's capability to reconstruct the instruction based on its thought
trajectory, checking for consistency between the reconstructed instruction and
the user's instruction. Extensive evaluation demonstrates ReAgent's
effectiveness against various backdoor attacks across tasks. For instance,
ReAgent reduces the attack success rate by up to 90\% in database operation
tasks, outperforming existing defenses by large margins. This work reveals the
potential of utilizing compromised agents themselves to mitigate backdoor
risks.


http://arxiv.org/abs/2506.07468
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models. (22%)
Mickel Liu; Liwei Jiang; Yancheng Liang; Simon Shaolei Du; Yejin Choi; Tim Althoff; Natasha Jaques
  Conventional language model (LM) safety alignment relies on a reactive,
disjoint procedure: attackers exploit a static model, followed by defensive
fine-tuning to patch exposed vulnerabilities. This sequential approach creates
a mismatch -- attackers overfit to obsolete defenses, while defenders
perpetually lag behind emerging threats. To address this, we propose
Self-RedTeam, an online self-play reinforcement learning algorithm where an
attacker and defender agent co-evolve through continuous interaction. We cast
safety alignment as a two-player zero-sum game, where a single model alternates
between attacker and defender roles -- generating adversarial prompts and
safeguarding against them -- while a reward LM adjudicates outcomes. This
enables dynamic co-adaptation. Grounded in the game-theoretic framework of
zero-sum games, we establish a theoretical safety guarantee which motivates the
design of our method: if self-play converges to a Nash Equilibrium, the
defender will reliably produce safe responses to any adversarial input.
Empirically, Self-RedTeam uncovers more diverse attacks (+21.8% SBERT) compared
to attackers trained against static defenders and achieves higher robustness on
safety benchmarks (e.g., +65.5% on WildJailBreak) than defenders trained
against static attackers. We further propose hidden Chain-of-Thought, allowing
agents to plan privately, which boosts adversarial diversity and reduces
over-refusals. Our results motivate a shift from reactive patching to proactive
co-evolution in LM safety training, enabling scalable, autonomous, and robust
self-improvement of LMs via multi-agent reinforcement learning (MARL).


http://arxiv.org/abs/2506.07836
Are Trees Really Green? A Detection Approach of IoT Malware Attacks. (8%)
Silvia Lucia Sanna; Diego Soi; Davide Maiorca; Giorgio Giacinto
  Nowadays, the Internet of Things (IoT) is widely employed, and its usage is
growing exponentially because it facilitates remote monitoring, predictive
maintenance, and data-driven decision making, especially in the healthcare and
industrial sectors. However, IoT devices remain vulnerable due to their
resource constraints and difficulty in applying security patches. Consequently,
various cybersecurity attacks are reported daily, such as Denial of Service,
particularly in IoT-driven solutions. Most attack detection methodologies are
based on Machine Learning (ML) techniques, which can detect attack patterns.
However, the focus is more on identification rather than considering the impact
of ML algorithms on computational resources. This paper proposes a green
methodology to identify IoT malware networking attacks based on flow
privacy-preserving statistical features. In particular, the hyperparameters of
three tree-based models -- Decision Trees, Random Forest and Extra-Trees -- are
optimized based on energy consumption and test-time performance in terms of
Matthew's Correlation Coefficient. Our results show that models maintain high
performance and detection accuracy while consistently reducing power usage in
terms of watt-hours (Wh). This suggests that on-premise ML-based Intrusion
Detection Systems are suitable for IoT and other resource-constrained devices.


http://arxiv.org/abs/2506.07666
ProARD: progressive adversarial robustness distillation: provide wide range of robust students. (5%)
Seyedhamidreza Mousavi; Seyedali Mousavi; Masoud Daneshtalab
  Adversarial Robustness Distillation (ARD) has emerged as an effective method
to enhance the robustness of lightweight deep neural networks against
adversarial attacks. Current ARD approaches have leveraged a large robust
teacher network to train one robust lightweight student. However, due to the
diverse range of edge devices and resource constraints, current approaches
require training a new student network from scratch to meet specific
constraints, leading to substantial computational costs and increased CO2
emissions. This paper proposes Progressive Adversarial Robustness Distillation
(ProARD), enabling the efficient one-time training of a dynamic network that
supports a diverse range of accurate and robust student networks without
requiring retraining. We first make a dynamic deep neural network based on
dynamic layers by encompassing variations in width, depth, and expansion in
each design stage to support a wide range of architectures. Then, we consider
the student network with the largest size as the dynamic teacher network.
ProARD trains this dynamic network using a weight-sharing mechanism to jointly
optimize the dynamic teacher network and its internal student networks.
However, due to the high computational cost of calculating exact gradients for
all the students within the dynamic network, a sampling mechanism is required
to select a subset of students. We show that random student sampling in each
iteration fails to produce accurate and robust students.


http://arxiv.org/abs/2506.07428
HeTa: Relation-wise Heterogeneous Graph Foundation Attack Model. (5%)
Yuling Wang; Zihui Chen; Pengfei Jiao; Xiao Wang
  Heterogeneous Graph Neural Networks (HGNNs) are vulnerable, highlighting the
need for tailored attacks to assess their robustness and ensure security.
However, existing HGNN attacks often require complex retraining of parameters
to generate specific perturbations for new scenarios. Recently, foundation
models have opened new horizons for the generalization of graph neural networks
by capturing shared semantics across various graph distributions. This leads us
to ask:Can we design a foundation attack model for HGNNs that enables
generalizable perturbations across different HGNNs, and quickly adapts to new
heterogeneous graphs (HGs)? Empirical findings reveal that, despite significant
differences in model design and parameter space, different HGNNs surprisingly
share common vulnerability patterns from a relation-aware perspective.
Therefore, we explore how to design foundation HGNN attack criteria by mining
shared attack units. In this paper, we propose a novel relation-wise
heterogeneous graph foundation attack model, HeTa. We introduce a foundation
surrogate model to align heterogeneity and identify the importance of shared
relation-aware attack units. Building on this, we implement a serialized
relation-by-relation attack based on the identified relational weights. In this
way, the perturbation can be transferred to various target HGNNs and easily
fine-tuned for new HGs. Extensive experiments exhibit powerful attack
performances and generalizability of our method.


http://arxiv.org/abs/2506.08346
SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Models. (2%)
Wenhan Yao; Fen Xiao; Xiarun Chen; Jia Liu; YongQiang He; Weiping Wen
  Deep speech classification tasks, including keyword spotting and speaker
verification, are vital in speech-based human-computer interaction. Recently,
the security of these technologies has been revealed to be susceptible to
backdoor attacks. Specifically, attackers use noisy disruption triggers and
speech element triggers to produce poisoned speech samples that train models to
become vulnerable. However, these methods typically create only a limited
number of backdoors due to the inherent constraints of the trigger function. In
this paper, we propose that speech backdoor attacks can strategically focus on
speech elements such as timbre and emotion, leveraging the Speech Large
Language Model (SLLM) to generate diverse triggers. Increasing the number of
triggers may disproportionately elevate the poisoning rate, resulting in higher
attack costs and a lower success rate per trigger. We introduce the Multiple
Gradient Descent Algorithm (MGDA) as a mitigation strategy to address this
challenge. The proposed attack is called the Speech Prompt Backdoor Attack
(SPBA). Building on this foundation, we conducted attack experiments on two
speech classification tasks, demonstrating that SPBA shows significant trigger
effectiveness and achieves exceptional performance in attack metrics.


http://arxiv.org/abs/2506.07645
Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models. (2%)
Maciej Chrabąszcz; Katarzyna Lorenc; Karolina Seweryn
  Large language models (LLMs) have demonstrated impressive capabilities across
various natural language processing (NLP) tasks in recent years. However, their
susceptibility to jailbreaks and perturbations necessitates additional
evaluations. Many LLMs are multilingual, but safety-related training data
contains mainly high-resource languages like English. This can leave them
vulnerable to perturbations in low-resource languages such as Polish. We show
how surprisingly strong attacks can be cheaply created by altering just a few
characters and using a small proxy model for word importance calculation. We
find that these character and word-level attacks drastically alter the
predictions of different LLMs, suggesting a potential vulnerability that can be
used to circumvent their internal safety mechanisms. We validate our attack
construction methodology on Polish, a low-resource language, and find potential
vulnerabilities of LLMs in this language. Additionally, we show how it can be
extended to other languages. We release the created datasets and code for
further research.


http://arxiv.org/abs/2506.07947
Statistical Hypothesis Testing for Auditing Robustness in Language Models. (1%)
Paulius Rauba; Qiyao Wei; der Schaar Mihaela van
  Consider the problem of testing whether the outputs of a large language model
(LLM) system change under an arbitrary intervention, such as an input
perturbation or changing the model variant. We cannot simply compare two LLM
outputs since they might differ due to the stochastic nature of the system, nor
can we compare the entire output distribution due to computational
intractability. While existing methods for analyzing text-based outputs exist,
they focus on fundamentally different problems, such as measuring bias or
fairness. To this end, we introduce distribution-based perturbation analysis, a
framework that reformulates LLM perturbation analysis as a frequentist
hypothesis testing problem. We construct empirical null and alternative output
distributions within a low-dimensional semantic similarity space via Monte
Carlo sampling, enabling tractable inference without restrictive distributional
assumptions. The framework is (i) model-agnostic, (ii) supports the evaluation
of arbitrary input perturbations on any black-box LLM, (iii) yields
interpretable p-values; (iv) supports multiple perturbations via controlled
error rates; and (v) provides scalar effect sizes. We demonstrate the
usefulness of the framework across multiple case studies, showing how we can
quantify response changes, measure true/false positive rates, and evaluate
alignment with reference models. Above all, we see this as a reliable
frequentist hypothesis testing framework for LLM auditing.


http://arxiv.org/abs/2506.08401
Single-Node Trigger Backdoor Attacks in Graph-Based Recommendation Systems. (1%)
Runze Li; Di Jin; Xiaobao Wang; Dongxiao He; Bingdao Feng; Zhen Wang
  Graph recommendation systems have been widely studied due to their ability to
effectively capture the complex interactions between users and items. However,
these systems also exhibit certain vulnerabilities when faced with attacks. The
prevailing shilling attack methods typically manipulate recommendation results
by injecting a large number of fake nodes and edges. However, such attack
strategies face two primary challenges: low stealth and high destructiveness.
To address these challenges, this paper proposes a novel graph backdoor attack
method that aims to enhance the exposure of target items to the target user in
a covert manner, without affecting other unrelated nodes. Specifically, we
design a single-node trigger generator, which can effectively expose multiple
target items to the target user by inserting only one fake user node.
Additionally, we introduce constraint conditions between the target nodes and
irrelevant nodes to mitigate the impact of fake nodes on the recommendation
system's performance. Experimental results show that the exposure of the target
items reaches no less than 50% in 99% of the target users, while the impact on
the recommendation system's performance is controlled within approximately 5%.


http://arxiv.org/abs/2506.17250
Towards Interpretable Adversarial Examples via Sparse Adversarial Attack. (99%)
Fudong Lin; Jiadong Lou; Hao Wang; Brian Jalaian; Xu Yuan
Sparse attacks are to optimize the magnitude of adversarial perturbations for fooling deep neural networks (DNNs) involving only a few perturbed pixels (i.e., under the l0 constraint), suitable for interpreting the vulnerability of DNNs. However, existing solutions fail to yield interpretable adversarial examples due to their poor sparsity. Worse still, they often struggle with heavy computational overhead, poor transferability, and weak attack strength. In this paper, we aim to develop a sparse attack for understanding the vulnerability of CNNs by minimizing the magnitude of initial perturbations under the l0 constraint, to overcome the existing drawbacks while achieving a fast, transferable, and strong attack to DNNs. In particular, a novel and theoretical sound parameterization technique is introduced to approximate the NP-hard l0 optimization problem, making directly optimizing sparse perturbations computationally feasible. Besides, a novel loss function is designed to augment initial perturbations by maximizing the adversary property and minimizing the number of perturbed pixels simultaneously. Extensive experiments are conducted to demonstrate that our approach, with theoretical performance guarantees, outperforms state-of-the-art sparse attacks in terms of computational overhead, transferability, and attack strength, expecting to serve as a benchmark for evaluating the robustness of DNNs. In addition, theoretical and empirical results validate that our approach yields sparser adversarial examples, empowering us to discover two categories of noises, i.e., "obscuring noise" and "leading noise", which will help interpret how adversarial perturbation misleads the classifiers into incorrect predictions. Our code is available at https://github.com/fudong03/SparseAttack.

http://arxiv.org/abs/2506.06992
Boosting Adversarial Transferability via Commonality-Oriented Gradient Optimization. (99%)
Yanting Gao; Yepeng Liu; Junming Liu; Qi Zhang; Hongyun Zhang; Duoqian Miao; Cairong Zhao
  Exploring effective and transferable adversarial examples is vital for
understanding the characteristics and mechanisms of Vision Transformers (ViTs).
However, adversarial examples generated from surrogate models often exhibit
weak transferability in black-box settings due to overfitting. Existing methods
improve transferability by diversifying perturbation inputs or applying uniform
gradient regularization within surrogate models, yet they have not fully
leveraged the shared and unique features of surrogate models trained on the
same task, leading to suboptimal transfer performance. Therefore, enhancing
perturbations of common information shared by surrogate models and suppressing
those tied to individual characteristics offers an effective way to improve
transferability. Accordingly, we propose a commonality-oriented gradient
optimization strategy (COGO) consisting of two components: Commonality
Enhancement (CE) and Individuality Suppression (IS). CE perturbs the mid-to-low
frequency regions, leveraging the fact that ViTs trained on the same dataset
tend to rely more on mid-to-low frequency information for classification. IS
employs adaptive thresholds to evaluate the correlation between backpropagated
gradients and model individuality, assigning weights to gradients accordingly.
Extensive experiments demonstrate that COGO significantly improves the transfer
success rates of adversarial attacks, outperforming current state-of-the-art
methods.


http://arxiv.org/abs/2506.07001
Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text. (98%)
Yize Cheng; Vinu Sankar Sadasivan; Mehrdad Saberi; Shoumik Saha; Soheil Feizi
  The increasing capabilities of Large Language Models (LLMs) have raised
concerns about their misuse in AI-generated plagiarism and social engineering.
While various AI-generated text detectors have been proposed to mitigate these
risks, many remain vulnerable to simple evasion techniques such as
paraphrasing. However, recent detectors have shown greater robustness against
such basic attacks. In this work, we introduce Adversarial Paraphrasing, a
training-free attack framework that universally humanizes any AI-generated text
to evade detection more effectively. Our approach leverages an off-the-shelf
instruction-following LLM to paraphrase AI-generated content under the guidance
of an AI text detector, producing adversarial examples that are specifically
optimized to bypass detection. Extensive experiments show that our attack is
both broadly effective and highly transferable across several detection
systems. For instance, compared to simple paraphrasing attack--which,
ironically, increases the true positive at 1% false positive (T@1%F) by 8.57%
on RADAR and 15.03% on Fast-DetectGPT--adversarial paraphrasing, guided by
OpenAI-RoBERTa-Large, reduces T@1%F by 64.49% on RADAR and a striking 98.96% on
Fast-DetectGPT. Across a diverse set of detectors--including neural
network-based, watermark-based, and zero-shot approaches--our attack achieves
an average T@1%F reduction of 87.88% under the guidance of
OpenAI-RoBERTa-Large. We also analyze the tradeoff between text quality and
attack success to find that our method can significantly reduce detection
rates, with mostly a slight degradation in text quality. Our adversarial setup
highlights the need for more robust and resilient detection strategies in the
light of increasingly sophisticated evasion techniques.


http://arxiv.org/abs/2506.07056
D2R: dual regularization loss with collaborative adversarial generation for model robustness. (96%)
Zhenyu Liu; Huizhi Liang; Rajiv Ranjan; Zhanxing Zhu; Vaclav Snasel; Varun Ojha
  The robustness of Deep Neural Network models is crucial for defending models
against adversarial attacks. Recent defense methods have employed collaborative
learning frameworks to enhance model robustness. Two key limitations of
existing methods are (i) insufficient guidance of the target model via loss
functions and (ii) non-collaborative adversarial generation. We, therefore,
propose a dual regularization loss (D2R Loss) method and a collaborative
adversarial generation (CAG) strategy for adversarial training. D2R loss
includes two optimization steps. The adversarial distribution and clean
distribution optimizations enhance the target model's robustness by leveraging
the strengths of different loss functions obtained via a suitable function
space exploration to focus more precisely on the target model's distribution.
CAG generates adversarial samples using a gradient-based collaboration between
guidance and target models. We conducted extensive experiments on three
benchmark databases, including CIFAR-10, CIFAR-100, Tiny ImageNet, and two
popular target models, WideResNet34-10 and PreActResNet18. Our results show
that D2R loss with CAG produces highly robust models.


http://arxiv.org/abs/2506.07214
Backdoor Attack on Vision Language Models with Stealthy Semantic Manipulation. (50%)
Zhiyuan Zhong; Zhen Sun; Yepang Liu; Xinlei He; Guanhong Tao
  Vision Language Models (VLMs) have shown remarkable performance, but are also
vulnerable to backdoor attacks whereby the adversary can manipulate the model's
outputs through hidden triggers. Prior attacks primarily rely on
single-modality triggers, leaving the crucial cross-modal fusion nature of VLMs
largely unexplored. Unlike prior work, we identify a novel attack surface that
leverages cross-modal semantic mismatches as implicit triggers. Based on this
insight, we propose BadSem (Backdoor Attack with Semantic Manipulation), a data
poisoning attack that injects stealthy backdoors by deliberately misaligning
image-text pairs during training. To perform the attack, we construct SIMBad, a
dataset tailored for semantic manipulation involving color and object
attributes. Extensive experiments across four widely used VLMs show that BadSem
achieves over 98% average ASR, generalizes well to out-of-distribution
datasets, and can transfer across poisoning modalities. Our detailed analysis
using attention visualization shows that backdoored models focus on
semantically sensitive regions under mismatched conditions while maintaining
normal behavior on clean inputs. To mitigate the attack, we try two defense
strategies based on system prompt and supervised fine-tuning but find that both
of them fail to mitigate the semantic backdoor. Our findings highlight the
urgent need to address semantic vulnerabilities in VLMs for their safer
deployment.


http://arxiv.org/abs/2506.11113
Breaking the Reviewer: Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks. (15%)
Tzu-Ling Lin; Wei-Chih Chen; Teng-Fang Hsiao; Hou-I Liu; Ya-Hsin Yeh; Yu Kai Chan; Wen-Sheng Lien; Po-Yen Kuo; Philip S. Yu; Hong-Han Shuai
  Peer review is essential for maintaining academic quality, but the increasing
volume of submissions places a significant burden on reviewers. Large language
models (LLMs) offer potential assistance in this process, yet their
susceptibility to textual adversarial attacks raises reliability concerns. This
paper investigates the robustness of LLMs used as automated reviewers in the
presence of such attacks. We focus on three key questions: (1) The
effectiveness of LLMs in generating reviews compared to human reviewers. (2)
The impact of adversarial attacks on the reliability of LLM-generated reviews.
(3) Challenges and potential mitigation strategies for LLM-based review. Our
evaluation reveals significant vulnerabilities, as text manipulations can
distort LLM assessments. We offer a comprehensive evaluation of LLM performance
in automated peer reviewing and analyze its robustness against adversarial
attacks. Our findings emphasize the importance of addressing adversarial risks
to ensure AI strengthens, rather than compromises, the integrity of scholarly
communication.


http://arxiv.org/abs/2506.07392
From Static to Adaptive Defense: Federated Multi-Agent Deep Reinforcement Learning-Driven Moving Target Defense Against DoS Attacks in UAV Swarm Networks. (2%)
Yuyang Zhou; Guang Cheng; Kang Du; Zihan Chen; Tian Qin; Yuyu Zhao
  The proliferation of unmanned aerial vehicle (UAV) swarms has enabled a wide
range of mission-critical applications, but also exposes UAV networks to severe
Denial-of-Service (DoS) threats due to their open wireless environment, dynamic
topology, and resource constraints. Traditional static or centralized defense
mechanisms are often inadequate for such dynamic and distributed scenarios. To
address these challenges, we propose a novel federated multi-agent deep
reinforcement learning (FMADRL)-driven moving target defense (MTD) framework
for proactive and adaptive DoS mitigation in UAV swarm networks. Specifically,
we design three lightweight and coordinated MTD mechanisms, including leader
switching, route mutation, and frequency hopping, that leverage the inherent
flexibility of UAV swarms to disrupt attacker efforts and enhance network
resilience. The defense problem is formulated as a multi-agent partially
observable Markov decision process (POMDP), capturing the distributed,
resource-constrained, and uncertain nature of UAV swarms under attack. Each UAV
is equipped with a local policy agent that autonomously selects MTD actions
based on partial observations and local experiences. By employing a policy
gradient-based FMADRL algorithm, UAVs collaboratively optimize their defense
policies via reward-weighted aggregation, enabling distributed learning without
sharing raw data and thus reducing communication overhead. Extensive
simulations demonstrate that our approach significantly outperforms
state-of-the-art baselines, achieving up to a 34.6% improvement in attack
mitigation rate, a reduction in average recovery time of up to 94.6%, and
decreases in energy consumption and defense cost by as much as 29.3% and 98.3%,
respectively, while maintaining robust mission continuity under various DoS
attack strategies.


http://arxiv.org/abs/2506.07372
Enhanced Consistency Bi-directional GAN(CBiGAN) for Malware Anomaly Detection. (1%)
Thesath Wijayasiri; Kar Wai Fok; Vrizlynn L. L. Thing
  Static analysis, a cornerstone technique in cybersecurity, offers a
noninvasive method for detecting malware by analyzing dormant software without
executing potentially harmful code. However, traditional static analysis often
relies on biased or outdated datasets, leading to gaps in detection
capabilities against emerging malware threats. To address this, our study
focuses on the binary content of files as key features for malware detection.
These binary contents are transformed and represented as images, which then
serve as inputs to deep learning models. This method takes into account the
visual patterns within the binary data, allowing the model to analyze potential
malware effectively. This paper introduces the application of the CBiGAN in the
domain of malware anomaly detection. Our approach leverages the CBiGAN for its
superior latent space mapping capabilities, critical for modeling complex
malware patterns by utilizing a reconstruction error-based anomaly detection
method. We utilized several datasets including both portable executable (PE)
files as well as Object Linking and Embedding (OLE) files. We then evaluated
our model against a diverse set of both PE and OLE files, including
self-collected malicious executables from 214 malware families. Our findings
demonstrate the robustness of this innovative approach, with the CBiGAN
achieving high Area Under the Curve (AUC) results with good generalizability,
thereby confirming its capability to distinguish between benign and diverse
malicious files with reasonably high accuracy.


http://arxiv.org/abs/2506.06933
Rewriting the Budget: A General Framework for Black-Box Attacks Under Cost Asymmetry. (99%)
Mahdi Salmani; Alireza Abdollahpoorrostam; Seyed-Mohsen Moosavi-Dezfooli
  Traditional decision-based black-box adversarial attacks on image classifiers
aim to generate adversarial examples by slightly modifying input images while
keeping the number of queries low, where each query involves sending an input
to the model and observing its output. Most existing methods assume that all
queries have equal cost. However, in practice, queries may incur asymmetric
costs; for example, in content moderation systems, certain output classes may
trigger additional review, enforcement, or penalties, making them more costly
than others. While prior work has considered such asymmetric cost settings,
effective algorithms for this scenario remain underdeveloped. In this paper, we
propose a general framework for decision-based attacks under asymmetric query
costs, which we refer to as asymmetric black-box attacks. We modify two core
components of existing attacks: the search strategy and the gradient estimation
process. Specifically, we propose Asymmetric Search (AS), a more conservative
variant of binary search that reduces reliance on high-cost queries, and
Asymmetric Gradient Estimation (AGREST), which shifts the sampling distribution
to favor low-cost queries. We design efficient algorithms that minimize total
attack cost by balancing different query types, in contrast to earlier methods
such as stealthy attacks that focus only on limiting expensive (high-cost)
queries. Our method can be integrated into a range of existing black-box
attacks with minimal changes. We perform both theoretical analysis and
empirical evaluation on standard image classification benchmarks. Across
various cost regimes, our method consistently achieves lower total query cost
and smaller perturbations than existing approaches, with improvements of up to
40% in some settings.


http://arxiv.org/abs/2506.06906
KNN-Defense: Defense against 3D Adversarial Point Clouds using Nearest-Neighbor Search. (98%)
Nima Jamali; Matina Mahdizadeh Sani; Hanieh Naderi; Shohreh Kasaei
  Deep neural networks (DNNs) have demonstrated remarkable performance in
analyzing 3D point cloud data. However, their vulnerability to adversarial
attacks-such as point dropping, shifting, and adding-poses a critical challenge
to the reliability of 3D vision systems. These attacks can compromise the
semantic and structural integrity of point clouds, rendering many existing
defense mechanisms ineffective. To address this issue, a defense strategy named
KNN-Defense is proposed, grounded in the manifold assumption and
nearest-neighbor search in feature space. Instead of reconstructing surface
geometry or enforcing uniform point distributions, the method restores
perturbed inputs by leveraging the semantic similarity of neighboring samples
from the training set. KNN-Defense is lightweight and computationally
efficient, enabling fast inference and making it suitable for real-time and
practical applications. Empirical results on the ModelNet40 dataset
demonstrated that KNN-Defense significantly improves robustness across various
attack types. In particular, under point-dropping attacks-where many existing
methods underperform due to the targeted removal of critical points-the
proposed method achieves accuracy gains of 20.1%, 3.6%, 3.44%, and 7.74% on
PointNet, PointNet++, DGCNN, and PCT, respectively. These findings suggest that
KNN-Defense offers a scalable and effective solution for enhancing the
adversarial resilience of 3D point cloud classifiers. (An open-source
implementation of the method, including code and data, is available at
https://github.com/nimajam41/3d-knn-defense).


http://arxiv.org/abs/2506.06742
LADSG: Label-Anonymized Distillation and Similar Gradient Substitution for Label Privacy in Vertical Federated Learning. (68%)
Zeyu Yan; Yifei Yao; Xuanbing Wen; Juli Zhang; Kai Fan
  Vertical federated learning (VFL) has become a key paradigm for collaborative
machine learning, enabling multiple parties to train models over distributed
feature spaces while preserving data privacy. Despite security protocols that
defend against external attacks - such as gradient masking and encryption,
which prevent unauthorized access to sensitive data - recent label inference
attacks from within the system have emerged. These attacks exploit gradients
and semantic embeddings to reconstruct private labels, bypassing traditional
defenses. For example, the passive label inference attack can reconstruct tens
of thousands of participants' private data using just 40 auxiliary labels,
posing a significant security threat. Existing defenses address single leakage
pathways, such as gradient leakage or label exposure. As attack strategies
evolve, their limitations become clear, especially against hybrid attacks that
combine multiple vectors. To address this, we propose Label-Anonymized Defense
with Substitution Gradient (LADSG), a unified defense framework that integrates
gradient substitution, label anonymization, and anomaly detection. LADSG
mitigates both gradient and label leakage while maintaining the scalability and
efficiency of VFL. Experiments on six real-world datasets show that LADSG
reduces label inference attack success rates by 30-60%, with minimal
computational overhead, underscoring the importance of lightweight defenses in
securing VFL.


http://arxiv.org/abs/2506.06891
Can In-Context Reinforcement Learning Recover From Reward Poisoning Attacks? (8%)
Paulius Sasnauskas; Yiğit Yalın; Goran Radanović
  We study the corruption-robustness of in-context reinforcement learning
(ICRL), focusing on the Decision-Pretrained Transformer (DPT, Lee et al.,
2023). To address the challenge of reward poisoning attacks targeting the DPT,
we propose a novel adversarial training framework, called Adversarially Trained
Decision-Pretrained Transformer (AT-DPT). Our method simultaneously trains an
attacker to minimize the true reward of the DPT by poisoning environment
rewards, and a DPT model to infer optimal actions from the poisoned data. We
evaluate the effectiveness of our approach against standard bandit algorithms,
including robust baselines designed to handle reward contamination. Our results
show that the proposed method significantly outperforms these baselines in
bandit settings, under a learned attacker. We additionally evaluate AT-DPT on
an adaptive attacker, and observe similar results. Furthermore, we extend our
evaluation to the MDP setting, confirming that the robustness observed in
bandit scenarios generalizes to more complex environments.


http://arxiv.org/abs/2506.06730
Fuse and Federate: Enhancing EV Charging Station Security with Multimodal Fusion and Federated Learning. (1%)
Rabah Rahal; Abdelaziz Amara Korba; Yacine Ghamri-Doudane
  The rapid global adoption of electric vehicles (EVs) has established electric
vehicle supply equipment (EVSE) as a critical component of smart grid
infrastructure. While essential for ensuring reliable energy delivery and
accessibility, EVSE systems face significant cybersecurity challenges,
including network reconnaissance, backdoor intrusions, and distributed
denial-of-service (DDoS) attacks. These emerging threats, driven by the
interconnected and autonomous nature of EVSE, require innovative and adaptive
security mechanisms that go beyond traditional intrusion detection systems
(IDS). Existing approaches, whether network-based or host-based, often fail to
detect sophisticated and targeted attacks specifically crafted to exploit new
vulnerabilities in EVSE infrastructure. This paper proposes a novel intrusion
detection framework that leverages multimodal data sources, including network
traffic and kernel events, to identify complex attack patterns. The framework
employs a distributed learning approach, enabling collaborative intelligence
across EVSE stations while preserving data privacy through federated learning.
Experimental results demonstrate that the proposed framework outperforms
existing solutions, achieving a detection rate above 98% and a precision rate
exceeding 97% in decentralized environments. This solution addresses the
evolving challenges of EVSE security, offering a scalable and privacypreserving
response to advanced cyber threats


http://arxiv.org/abs/2506.06884
FREE: Fast and Robust Vision Language Models with Early Exits. (1%)
Divya Jyoti Bajpai; Manjesh Kumar Hanawal
  In recent years, Vision-Language Models (VLMs) have shown remarkable
performance improvements in Vision-Language tasks. However, their large size
poses challenges for real-world applications where inference latency is a
concern. To tackle this issue, we propose employing Early Exit (EE) strategies
in VLMs. However, training exit classifiers in VLMs is challenging,
particularly with limited labeled training data. To address this, we introduce
FREE, an adversarial training approach within a GAN-based framework. Here, each
exit consists of a transformer layer and a classifier. The transformer layer is
adversarially trained to produce feature representations similar to the final
layer, while a feature classifier serves as the discriminator. Our method
focuses on performing input-adaptive inference that increases inference speed
with minimal drop in performance. Experimental results demonstrate the
effectiveness of our approach in enhancing accuracy and model robustness by
mitigating overthinking and the phenomenon of mid-crisis that we highlight. We
experimentally validate that our method speeds up the inference process by more
than 1.51x while retaining comparable performance. The source code is available
at https://github.com/Div290/FREE.


http://arxiv.org/abs/2506.06563
Securing Traffic Sign Recognition Systems in Autonomous Vehicles. (98%)
Thushari Hapuarachchi; Long Dang; Kaiqi Xiong
  Deep Neural Networks (DNNs) are widely used for traffic sign recognition
because they can automatically extract high-level features from images. These
DNNs are trained on large-scale datasets obtained from unknown sources.
Therefore, it is important to ensure that the models remain secure and are not
compromised or poisoned during training. In this paper, we investigate the
robustness of DNNs trained for traffic sign recognition. First, we perform the
error-minimizing attacks on DNNs used for traffic sign recognition by adding
imperceptible perturbations on training data. Then, we propose a data
augmentation-based training method to mitigate the error-minimizing attacks.
The proposed training method utilizes nonlinear transformations to disrupt the
perturbations and improve the model robustness. We experiment with two
well-known traffic sign datasets to demonstrate the severity of the attack and
the effectiveness of our mitigation scheme. The error-minimizing attacks reduce
the prediction accuracy of the DNNs from 99.90% to 10.6%. However, our
mitigation scheme successfully restores the prediction accuracy to 96.05%.
Moreover, our approach outperforms adversarial training in mitigating the
error-minimizing attacks. Furthermore, we propose a detection model capable of
identifying poisoned data even when the perturbations are imperceptible to
human inspection. Our detection model achieves a success rate of over 99% in
identifying the attack. This research highlights the need to employ advanced
training methods for DNNs in traffic sign recognition systems to mitigate the
effects of data poisoning attacks.


http://arxiv.org/abs/2506.05937
Quantifying Adversarial Uncertainty in Evidential Deep Learning using Conflict Resolution. (67%)
Charmaine Barker; Daniel Bethell; Simos Gerasimou
  Reliability of deep learning models is critical for deployment in high-stakes
applications, where out-of-distribution or adversarial inputs may lead to
detrimental outcomes. Evidential Deep Learning, an efficient paradigm for
uncertainty quantification, models predictions as Dirichlet distributions of a
single forward pass. However, EDL is particularly vulnerable to adversarially
perturbed inputs, making overconfident errors. Conflict-aware Evidential Deep
Learning (C-EDL) is a lightweight post-hoc uncertainty quantification approach
that mitigates these issues, enhancing adversarial and OOD robustness without
retraining. C-EDL generates diverse, task-preserving transformations per input
and quantifies representational disagreement to calibrate uncertainty estimates
when needed. C-EDL's conflict-aware prediction adjustment improves detection of
OOD and adversarial inputs, maintaining high in-distribution accuracy and low
computational overhead. Our experimental evaluation shows that C-EDL
significantly outperforms state-of-the-art EDL variants and competitive
baselines, achieving substantial reductions in coverage for OOD data (up to
55%) and adversarial data (up to 90%), across a range of datasets, attack
types, and uncertainty metrics.


http://arxiv.org/abs/2506.06556
SDN-Based False Data Detection With Its Mitigation and Machine Learning Robustness for In-Vehicle Networks. (56%)
Long Dang; Thushari Hapuarachchi; Kaiqi Xiong; Yi Li
  As the development of autonomous and connected vehicles advances, the
complexity of modern vehicles increases, with numerous Electronic Control Units
(ECUs) integrated into the system. In an in-vehicle network, these ECUs
communicate with one another using an standard protocol called Controller Area
Network (CAN). Securing communication among ECUs plays a vital role in
maintaining the safety and security of the vehicle. This paper proposes a
robust SDN-based False Data Detection and Mitigation System (FDDMS) for
in-vehicle networks. Leveraging the unique capabilities of Software-Defined
Networking (SDN), FDDMS is designed to monitor and detect false data injection
attacks in real-time. Specifically, we focus on brake-related ECUs within an
SDN-enabled in-vehicle network. First, we decode raw CAN data to create an
attack model that illustrates how false data can be injected into the system.
Then, FDDMS, incorporating a Long Short Term Memory (LSTM)-based detection
model, is used to identify false data injection attacks. We further propose an
effective variant of DeepFool attack to evaluate the model's robustness. To
countermeasure the impacts of four adversarial attacks including Fast gradient
descent method, Basic iterative method, DeepFool, and the DeepFool variant, we
further enhance a re-training technique method with a threshold based selection
strategy. Finally, a mitigation scheme is implemented to redirect attack
traffic by dynamically updating flow rules through SDN. Our experimental
results show that the proposed FDDMS is robust against adversarial attacks and
effectively detects and mitigates false data injection attacks in real-time.


http://arxiv.org/abs/2506.06151
Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems. (47%)
Haowei Wang; Rupeng Zhang; Junjie Wang; Mingyang Li; Yuekai Huang; Dandan Wang; Qing Wang
  Retrieval-Augmented Generation (RAG) systems enhance Large Language Models
(LLMs) by retrieving relevant documents from external corpora before generating
responses. This approach significantly expands LLM capabilities by leveraging
vast, up-to-date external knowledge. However, this reliance on external
knowledge makes RAG systems vulnerable to corpus poisoning attacks that
manipulate generated outputs via poisoned document injection. Existing
poisoning attack strategies typically treat the retrieval and generation stages
as disjointed, limiting their effectiveness. We propose Joint-GCG, the first
framework to unify gradient-based attacks across both retriever and generator
models through three innovations: (1) Cross-Vocabulary Projection for aligning
embedding spaces, (2) Gradient Tokenization Alignment for synchronizing
token-level gradient signals, and (3) Adaptive Weighted Fusion for dynamically
balancing attacking objectives. Evaluations demonstrate that Joint-GCG achieves
at most 25% and an average of 5% higher attack success rate than previous
methods across multiple retrievers and generators. While optimized under a
white-box assumption, the generated poisons show unprecedented transferability
to unseen models. Joint-GCG's innovative unification of gradient-based attacks
across retrieval and generation stages fundamentally reshapes our understanding
of vulnerabilities within RAG systems. Our code is available at
https://github.com/NicerWang/Joint-GCG.


http://arxiv.org/abs/2506.05739
To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt. (45%)
Zhilong Wang; Neha Nagaraja; Lan Zhang; Hayretdin Bahsi; Pawan Patil; Peng Liu
  LLM agents are widely used as agents for customer support, content
generation, and code assistance. However, they are vulnerable to prompt
injection attacks, where adversarial inputs manipulate the model's behavior.
Traditional defenses like input sanitization, guard models, and guardrails are
either cumbersome or ineffective. In this paper, we propose a novel,
lightweight defense mechanism called Polymorphic Prompt Assembling (PPA), which
protects against prompt injection with near-zero overhead. The approach is
based on the insight that prompt injection requires guessing and breaking the
structure of the system prompt. By dynamically varying the structure of system
prompts, PPA prevents attackers from predicting the prompt structure, thereby
enhancing security without compromising performance. We conducted experiments
to evaluate the effectiveness of PPA against existing attacks and compared it
with other defense methods.


http://arxiv.org/abs/2506.06119
SATversary: Adversarial Attacks on Satellite Fingerprinting. (31%)
Joshua Smailes; Sebastian Köhler; Simon Birnbach; Martin Strohmeier; Ivan Martinovic
  As satellite systems become increasingly vulnerable to physical layer attacks
via SDRs, novel countermeasures are being developed to protect critical
systems, particularly those lacking cryptographic protection, or those which
cannot be upgraded to support modern cryptography. Among these is transmitter
fingerprinting, which provides mechanisms by which communication can be
authenticated by looking at characteristics of the transmitter, expressed as
impairments on the signal.
  Previous works show that fingerprinting can be used to classify satellite
transmitters, or authenticate them against SDR-equipped attackers under simple
replay scenarios. In this paper we build upon this by looking at attacks
directly targeting the fingerprinting system, with an attacker optimizing for
maximum impact in jamming, spoofing, and dataset poisoning attacks, and
demonstrate these attacks on the SatIQ system designed to authenticate Iridium
transmitters. We show that an optimized jamming signal can cause a 50% error
rate with attacker-to-victim ratios as low as -30dB (far less power than
traditional jamming) and demonstrate successful identity forgery during
spoofing attacks, with an attacker successfully removing their own
transmitter's fingerprint from messages. We also present a data poisoning
attack, enabling persistent message spoofing by altering the data used to
authenticate incoming messages to include the fingerprint of the attacker's
transmitter.
  Finally, we show that our model trained to optimize spoofing attacks can also
be used to detect spoofing and replay attacks, even when it has never seen the
attacker's transmitter before. Furthermore, this technique works even when the
training dataset includes only a single transmitter, enabling fingerprinting to
be used to protect small constellations and even individual satellites,
providing additional protection where it is needed the most.


http://arxiv.org/abs/2506.06060
Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models. (16%)
Yingqi Hu; Zhuo Zhang; Jingyuan Zhang; Lizhen Qu; Zenglin Xu
  Federated fine-tuning of large language models (FedLLMs) presents a promising
approach for achieving strong model performance while preserving data privacy
in sensitive domains. However, the inherent memorization ability of LLMs makes
them vulnerable to training data extraction attacks. To investigate this risk,
we introduce simple yet effective extraction attack algorithms specifically
designed for FedLLMs. In contrast to prior "verbatim" extraction attacks, which
assume access to fragments from all training data, our approach operates under
a more realistic threat model, where the attacker only has access to a single
client's data and aims to extract previously unseen personally identifiable
information (PII) from other clients. This requires leveraging contextual
prefixes held by the attacker to generalize across clients. To evaluate the
effectiveness of our approaches, we propose two rigorous metrics-coverage rate
and efficiency-and extend a real-world legal dataset with PII annotations
aligned with CPIS, GDPR, and CCPA standards, achieving 89.9% human-verified
precision. Experimental results show that our method can extract up to 56.57%
of victim-exclusive PII, with "Address," "Birthday," and "Name" being the most
vulnerable categories. Our findings underscore the pressing need for robust
defense strategies and contribute a new benchmark and evaluation framework for
future research in privacy-preserving federated learning.


http://arxiv.org/abs/2506.06027
Sample-Specific Noise Injection For Diffusion-Based Adversarial Purification. (9%)
Yuhao Sun; Jiacheng Zhang; Zesheng Ye; Chaowei Xiao; Feng Liu
  Diffusion-based purification (DBP) methods aim to remove adversarial noise
from the input sample by first injecting Gaussian noise through a forward
diffusion process, and then recovering the clean example through a reverse
generative process. In the above process, how much Gaussian noise is injected
to the input sample is key to the success of DBP methods, which is controlled
by a constant noise level $t^*$ for all samples in existing methods. In this
paper, we discover that an optimal $t^*$ for each sample indeed could be
different. Intuitively, the cleaner a sample is, the less the noise it should
be injected, and vice versa. Motivated by this finding, we propose a new
framework, called Sample-specific Score-aware Noise Injection (SSNI).
Specifically, SSNI uses a pre-trained score network to estimate how much a data
point deviates from the clean data distribution (i.e., score norms). Then,
based on the magnitude of score norms, SSNI applies a reweighting function to
adaptively adjust $t^*$ for each sample, achieving sample-specific noise
injections. Empirically, incorporating our framework with existing DBP methods
results in a notable improvement in both accuracy and robustness on CIFAR-10
and ImageNet-1K, highlighting the necessity to allocate distinct noise levels
to different samples in DBP methods. Our code is available at:
https://github.com/tmlr-group/SSNI.


http://arxiv.org/abs/2506.06003
What Really is a Member? Discrediting Membership Inference via Poisoning. (8%)
Neal Mangaokar; Ashish Hooda; Zhuohang Li; Bradley A. Malin; Kassem Fawaz; Somesh Jha; Atul Prakash; Amrita Roy Chowdhury
  Membership inference tests aim to determine whether a particular data point
was included in a language model's training set. However, recent works have
shown that such tests often fail under the strict definition of membership
based on exact matching, and have suggested relaxing this definition to include
semantic neighbors as members as well. In this work, we show that membership
inference tests are still unreliable under this relaxation - it is possible to
poison the training dataset in a way that causes the test to produce incorrect
predictions for a target point. We theoretically reveal a trade-off between a
test's accuracy and its robustness to poisoning. We also present a concrete
instantiation of this poisoning attack and empirically validate its
effectiveness. Our results show that it can degrade the performance of existing
tests to well below random.


http://arxiv.org/abs/2506.05867
Stealix: Model Stealing via Prompt Evolution. (4%)
Zhixiong Zhuang; Hui-Po Wang; Maria-Irina Nicolae; Mario Fritz
  Model stealing poses a significant security risk in machine learning by
enabling attackers to replicate a black-box model without access to its
training data, thus jeopardizing intellectual property and exposing sensitive
information. Recent methods that use pre-trained diffusion models for data
synthesis improve efficiency and performance but rely heavily on manually
crafted prompts, limiting automation and scalability, especially for attackers
with little expertise. To assess the risks posed by open-source pre-trained
models, we propose a more realistic threat model that eliminates the need for
prompt design skills or knowledge of class names. In this context, we introduce
Stealix, the first approach to perform model stealing without predefined
prompts. Stealix uses two open-source pre-trained models to infer the victim
model's data distribution, and iteratively refines prompts through a genetic
algorithm, progressively improving the precision and diversity of synthetic
images. Our experimental results demonstrate that Stealix significantly
outperforms other methods, even those with access to class names or
fine-grained prompts, while operating under the same query budget. These
findings highlight the scalability of our approach and suggest that the risks
posed by pre-trained generative models in model stealing may be greater than
previously recognized.


http://arxiv.org/abs/2506.06518
A Systematic Review of Poisoning Attacks Against Large Language Models. (2%)
Neil Fendley; Edward W. Staley; Joshua Carney; William Redman; Marie Chau; Nathan Drenkow
  With the widespread availability of pretrained Large Language Models (LLMs)
and their training datasets, concerns about the security risks associated with
their usage has increased significantly. One of these security risks is the
threat of LLM poisoning attacks where an attacker modifies some part of the LLM
training process to cause the LLM to behave in a malicious way. As an emerging
area of research, the current frameworks and terminology for LLM poisoning
attacks are derived from earlier classification poisoning literature and are
not fully equipped for generative LLM settings. We conduct a systematic review
of published LLM poisoning attacks to clarify the security implications and
address inconsistencies in terminology across the literature. We propose a
comprehensive poisoning threat model applicable to categorize a wide range of
LLM poisoning attacks. The poisoning threat model includes four poisoning
attack specifications that define the logistics and manipulation strategies of
an attack as well as six poisoning metrics used to measure key characteristics
of an attack. Under our proposed framework, we organize our discussion of
published LLM poisoning literature along four critical dimensions of LLM
poisoning attacks: concept poisons, stealthy poisons, persistent poisons, and
poisons for unique tasks, to better understand the current landscape of
security risks.


http://arxiv.org/abs/2506.06613
Robust Learnability of Sample-Compressible Distributions under Noisy or Adversarial Perturbations. (2%)
Arefe Boushehrian; Amir Najafi
  Learning distribution families over $\mathbb{R}^d$ is a fundamental problem
in unsupervised learning and statistics. A central question in this setting is
whether a given family of distributions possesses sufficient structure to be
(at least) information-theoretically learnable and, if so, to characterize its
sample complexity. In 2018, Ashtiani et al. reframed \emph{sample
compressibility}, originally due to Littlestone and Warmuth (1986), as a
structural property of distribution classes, proving that it guarantees
PAC-learnability. This discovery subsequently enabled a series of recent
advancements in deriving nearly tight sample complexity bounds for various
high-dimensional open problems. It has been further conjectured that the
converse also holds: every learnable class admits a tight sample compression
scheme.
  In this work, we establish that sample compressible families remain learnable
even from perturbed samples, subject to a set of necessary and sufficient
conditions. We analyze two models of data perturbation: (i) an additive
independent noise model, and (ii) an adversarial corruption model, where an
adversary manipulates a limited subset of the samples unknown to the learner.
Our results are general and rely on as minimal assumptions as possible. We
develop a perturbation-quantization framework that interfaces naturally with
the compression scheme and leads to sample complexity bounds that scale
gracefully with the noise level and corruption budget. As concrete
applications, we establish new sample complexity bounds for learning finite
mixtures of high-dimensional uniform distributions under both noise and
adversarial perturbations, as well as for learning Gaussian mixture models from
adversarially corrupted samples, resolving two open problems in the literature.


http://arxiv.org/abs/2506.06565
Adapting Under Fire: Multi-Agent Reinforcement Learning for Adversarial Drift in Network Security. (2%)
Emilia Rivas; Sabrina Saika; Ahtesham Bakht; Aritran Piplai; Nathaniel D. Bastian; Ankit Shah
  Evolving attacks are a critical challenge for the long-term success of
Network Intrusion Detection Systems (NIDS). The rise of these changing patterns
has exposed the limitations of traditional network security methods. While
signature-based methods are used to detect different types of attacks, they
often fail to detect unknown attacks. Moreover, the system requires frequent
updates with new signatures as the attackers are constantly changing their
tactics. In this paper, we design an environment where two agents improve their
policies over time. The adversarial agent, referred to as the red agent,
perturbs packets to evade the intrusion detection mechanism, whereas the blue
agent learns new defensive policies using drift adaptation techniques to
counter the attacks. Both agents adapt iteratively: the red agent responds to
the evolving NIDS, while the blue agent adjusts to emerging attack patterns. By
studying the model's learned policy, we offer concrete insights into drift
adaptation techniques with high utility. Experiments show that the blue agent
boosts model accuracy by 30% with just 2 to 3 adaptation steps using only 25 to
30 samples each.


http://arxiv.org/abs/2506.06572
Cyber Security of Sensor Systems for State Sequence Estimation: an AI Approach. (1%)
Xubin Fang; Rick S. Blum; Ramesh Bharadwaj; Brian M. Sadler
  Sensor systems are extremely popular today and vulnerable to sensor data
attacks. Due to possible devastating consequences, counteracting sensor data
attacks is an extremely important topic, which has not seen sufficient study.
This paper develops the first methods that accurately identify/eliminate only
the problematic attacked sensor data presented to a sequence
estimation/regression algorithm under a powerful attack model constructed based
on known/observed attacks. The approach does not assume a known form for the
statistical model of the sensor data, allowing data-driven and machine learning
sequence estimation/regression algorithms to be protected. A simple protection
approach for attackers not endowed with knowledge of the details of our
protection approach is first developed, followed by additional processing for
attacks based on protection system knowledge. In the cases tested for which it
was designed, experimental results show that the simple approach achieves
performance indistinguishable, to two decimal places, from that for an approach
which knows which sensors are attacked. For cases where the attacker has
knowledge of the protection approach, experimental results indicate the
additional processing can be configured so that the worst-case degradation
under the additional processing and a large number of sensors attacked can be
made significantly smaller than the worst-case degradation of the simple
approach, and close to an approach which knows which sensors are attacked, for
the same number of attacked sensors with just a slight degradation under no
attacks. Mathematical descriptions of the worst-case attacks are used to
demonstrate the additional processing will provide similar advantages for cases
for which we do not have numerical results. All the data-driven processing used
in our approaches employ only unattacked training data.


http://arxiv.org/abs/2506.05430
Explainer-guided Targeted Adversarial Attacks against Binary Code Similarity Detection Models. (99%)
Mingjie Zhejiang University Chen; Tiancheng Huazhong University of Science and Technology Zhu; Mingxue The State Key Laboratory of Blockchain and Data Security, Zhejiang University & Hangzhou High-Tech Zone Zhang; Yiling University College London He; Minghao University of Southern California Lin; Penghui Columbia University Li; Kui The State Key Laboratory of Blockchain and Data Security, Zhejiang University Ren
  Binary code similarity detection (BCSD) serves as a fundamental technique for
various software engineering tasks, e.g., vulnerability detection and
classification. Attacks against such models have therefore drawn extensive
attention, aiming at misleading the models to generate erroneous predictions.
Prior works have explored various approaches to generating semantic-preserving
variants, i.e., adversarial samples, to evaluate the robustness of the models
against adversarial attacks. However, they have mainly relied on heuristic
criteria or iterative greedy algorithms to locate salient code influencing the
model output, failing to operate on a solid theoretical basis. Moreover, when
processing programs with high complexities, such attacks tend to be
time-consuming.
  In this work, we propose a novel optimization for adversarial attacks against
BCSD models. In particular, we aim to improve the attacks in a challenging
scenario, where the attack goal is to limit the model predictions to a specific
range, i.e., the targeted attacks. Our attack leverages the superior capability
of black-box, model-agnostic explainers in interpreting the model decision
boundaries, thereby pinpointing the critical code snippet to apply
semantic-preserving perturbations. The evaluation results demonstrate that
compared with the state-of-the-art attacks, the proposed attacks achieve higher
attack success rate in almost all scenarios, while also improving the
efficiency and transferability. Our real-world case studies on vulnerability
detection and classification further demonstrate the security implications of
our attacks, highlighting the urgent need to further enhance the robustness of
existing BCSD models.


http://arxiv.org/abs/2506.06389
Exploring Adversarial Watermarking in Transformer-Based Models: Transferability and Robustness Against Defense Mechanism for Medical Images. (99%)
Rifat Sadik; Tanvir Rahman; Arpan Bhattacharjee; Bikash Chandra Halder; Ismail Hossain
  Deep learning models have shown remarkable success in dermatological image
analysis, offering potential for automated skin disease diagnosis. Previously,
convolutional neural network(CNN) based architectures have achieved immense
popularity and success in computer vision (CV) based task like skin image
recognition, generation and video analysis. But with the emergence of
transformer based models, CV tasks are now are nowadays carrying out using
these models. Vision Transformers (ViTs) is such a transformer-based models
that have shown success in computer vision. It uses self-attention mechanisms
to achieve state-of-the-art performance across various tasks. However, their
reliance on global attention mechanisms makes them susceptible to adversarial
perturbations. This paper aims to investigate the susceptibility of ViTs for
medical images to adversarial watermarking-a method that adds so-called
imperceptible perturbations in order to fool models. By generating adversarial
watermarks through Projected Gradient Descent (PGD), we examine the
transferability of such attacks to CNNs and analyze the performance defense
mechanism -- adversarial training. Results indicate that while performance is
not compromised for clean images, ViTs certainly become much more vulnerable to
adversarial attacks: an accuracy drop of as low as 27.6%. Nevertheless,
adversarial training raises it up to 90.0%.


http://arxiv.org/abs/2506.04743
SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs. (92%)
Shuhan Xu; Siyuan Liang; Hongling Zheng; Yong Luo; Aishan Liu; Dacheng Tao
  Vision-Language Models (VLMs) have achieved remarkable performance in image
captioning, but recent studies show they are vulnerable to backdoor attacks.
Attackers can inject imperceptible perturbations-such as local pixel triggers
or global semantic phrases-into the training data, causing the model to
generate malicious, attacker-controlled captions for specific inputs. These
attacks are hard to detect and defend due to their stealthiness and cross-modal
nature. By analyzing attack samples, we identify two key vulnerabilities: (1)
abnormal attention concentration on specific image regions, and (2) semantic
drift and incoherence in generated captions. To counter this, we propose
Semantic Reward Defense (SRD), a reinforcement learning framework that
mitigates backdoor behavior without prior knowledge of triggers. SRD uses a
Deep Q-Network to learn policies for applying discrete perturbations (e.g.,
occlusion, color masking) to sensitive image regions, aiming to disrupt the
activation of malicious pathways. We design a semantic fidelity score as the
reward signal, which jointly evaluates semantic consistency and linguistic
fluency of the output, guiding the agent toward generating robust yet faithful
captions. Experiments across mainstream VLMs and datasets show SRD reduces
attack success rates to 5.6%, while preserving caption quality on clean inputs
with less than 10% performance drop. SRD offers a trigger-agnostic,
interpretable defense paradigm against stealthy backdoor threats in multimodal
generative models.


http://arxiv.org/abs/2506.05434
Efficient Robust Conformal Prediction via Lipschitz-Bounded Networks. (86%)
Thomas IRIT, DTIPG - SNCF, UT3 Massena; Léo IMT, DTIPG - SNCF, UT3 andéol; Thibaut IRIT, UT3 Boissin; Franck IRIT, UT3 Mamalet; Corentin IRIT, UT3 Friedrich; Mathieu IRIT, UT3 Serrurier; Sébastien IMT Gerchinovitz
  Conformal Prediction (CP) has proven to be an effective post-hoc method for
improving the trustworthiness of neural networks by providing prediction sets
with finite-sample guarantees. However, under adversarial attacks, classical
conformal guarantees do not hold anymore: this problem is addressed in the
field of Robust Conformal Prediction. Several methods have been proposed to
provide robust CP sets with guarantees under adversarial perturbations, but,
for large scale problems, these sets are either too large or the methods are
too computationally demanding to be deployed in real life scenarios. In this
work, we propose a new method that leverages Lipschitz-bounded networks to
precisely and efficiently estimate robust CP sets. When combined with a
1-Lipschitz robust network, we demonstrate that our lip-rcp method outperforms
state-of-the-art results in both the size of the robust CP sets and
computational efficiency in medium and large-scale scenarios such as ImageNet.
Taking a different angle, we also study vanilla CP under attack, and derive new
worst-case coverage bounds of vanilla CP sets, which are valid simultaneously
for all adversarial attack levels. Our lip-rcp method makes this second
approach as efficient as vanilla CP while also allowing robustness guarantees.


http://arxiv.org/abs/2506.04951
Robustness as Architecture: Designing IQA Models to Withstand Adversarial Perturbations. (81%)
Igor Meleshin; Anna Chistyakova; Anastasia Antsiferova; Dmitriy Vatolin
  Image Quality Assessment (IQA) models are increasingly relied upon to
evaluate image quality in real-world systems -- from compression and
enhancement to generation and streaming. Yet their adoption brings a
fundamental risk: these models are inherently unstable. Adversarial
manipulations can easily fool them, inflating scores and undermining trust.
Traditionally, such vulnerabilities are addressed through data-driven defenses
-- adversarial retraining, regularization, or input purification. But what if
this is the wrong lens? What if robustness in perceptual models is not
something to learn but something to design? In this work, we propose a
provocative idea: robustness as an architectural prior. Rather than training
models to resist perturbations, we reshape their internal structure to suppress
sensitivity from the ground up. We achieve this by enforcing orthogonal
information flow, constraining the network to norm-preserving operations -- and
further stabilizing the system through pruning and fine-tuning. The result is a
robust IQA architecture that withstands adversarial attacks without requiring
adversarial training or significant changes to the original model. This
approach suggests a shift in perspective: from optimizing robustness through
data to engineering it through design.


http://arxiv.org/abs/2506.05032
Identifying and Understanding Cross-Class Features in Adversarial Training. (68%)
Zeming Wei; Yiwen Guo; Yisen Wang
  Adversarial training (AT) has been considered one of the most effective
methods for making deep neural networks robust against adversarial attacks,
while the training mechanisms and dynamics of AT remain open research problems.
In this paper, we present a novel perspective on studying AT through the lens
of class-wise feature attribution. Specifically, we identify the impact of a
key family of features on AT that are shared by multiple classes, which we call
cross-class features. These features are typically useful for robust
classification, which we offer theoretical evidence to illustrate through a
synthetic data model. Through systematic studies across multiple model
architectures and settings, we find that during the initial stage of AT, the
model tends to learn more cross-class features until the best robustness
checkpoint. As AT further squeezes the training robust loss and causes robust
overfitting, the model tends to make decisions based on more class-specific
features. Based on these discoveries, we further provide a unified view of two
existing properties of AT, including the advantage of soft-label training and
robust overfitting. Overall, these insights refine the current understanding of
AT mechanisms and provide new perspectives on studying them. Our code is
available at https://github.com/PKU-ML/Cross-Class-Features-AT.


http://arxiv.org/abs/2506.05429
Coordinated Robustness Evaluation Framework for Vision-Language Models. (54%)
Ashwin Ramesh Babu; Sajad Mousavi; Vineet Gundecha; Sahand Ghorbanpour; Avisek Naug; Antonio Guillen; Ricardo Luna Gutierrez; Soumyendu Sarkar
  Vision-language models, which integrate computer vision and natural language
processing capabilities, have demonstrated significant advancements in tasks
such as image captioning and visual question and answering. However, similar to
traditional models, they are susceptible to small perturbations, posing a
challenge to their robustness, particularly in deployment scenarios. Evaluating
the robustness of these models requires perturbations in both the vision and
language modalities to learn their inter-modal dependencies. In this work, we
train a generic surrogate model that can take both image and text as input and
generate joint representation which is further used to generate adversarial
perturbations for both the text and image modalities. This coordinated attack
strategy is evaluated on the visual question and answering and visual reasoning
datasets using various state-of-the-art vision-language models. Our results
indicate that the proposed strategy outperforms other multi-modal attacks and
single-modality attacks from the recent literature. Our results demonstrate
their effectiveness in compromising the robustness of several state-of-the-art
pre-trained multi-modal models such as instruct-BLIP, ViLT and others.


http://arxiv.org/abs/2506.04823
Fool the Stoplight: Realistic Adversarial Patch Attacks on Traffic Light Detectors. (47%)
Svetlana Pavlitska; Jamie Robb; Nikolai Polley; Melih Yazgan; J. Marius Zöllner
  Realistic adversarial attacks on various camera-based perception tasks of
autonomous vehicles have been successfully demonstrated so far. However, only a
few works considered attacks on traffic light detectors. This work shows how
CNNs for traffic light detection can be attacked with printed patches. We
propose a threat model, where each instance of a traffic light is attacked with
a patch placed under it, and describe a training strategy. We demonstrate
successful adversarial patch attacks in universal settings. Our experiments
show realistic targeted red-to-green label-flipping attacks and attacks on
pictogram classification. Finally, we perform a real-world evaluation with
printed patches and demonstrate attacks in the lab settings with a mobile
traffic light for construction sites and in a test area with stationary traffic
lights. Our code is available at
https://github.com/KASTEL-MobilityLab/attacks-on-traffic-light-detection.


http://arxiv.org/abs/2506.04879
Invisible Backdoor Triggers in Image Editing Model via Deep Watermarking. (26%)
Yu-Feng Chen; Tzuhsuan Huang; Pin-Yen Chiu; Jun-Cheng Chen
  Diffusion models have achieved remarkable progress in both image generation
and editing. However, recent studies have revealed their vulnerability to
backdoor attacks, in which specific patterns embedded in the input can
manipulate the model's behavior. Most existing research in this area has
proposed attack frameworks focused on the image generation pipeline, leaving
backdoor attacks in image editing relatively unexplored. Among the few studies
targeting image editing, most utilize visible triggers, which are impractical
because they introduce noticeable alterations to the input image before
editing. In this paper, we propose a novel attack framework that embeds
invisible triggers into the image editing process via poisoned training data.
We leverage off-the-shelf deep watermarking models to encode imperceptible
watermarks as backdoor triggers. Our goal is to make the model produce the
predefined backdoor target when it receives watermarked inputs, while editing
clean images normally according to the given prompt. With extensive experiments
across different watermarking models, the proposed method achieves promising
attack success rates. In addition, the analysis results of the watermark
characteristics in term of backdoor attack further support the effectiveness of
our approach. The code is available
at:https://github.com/aiiu-lab/BackdoorImageEditing


http://arxiv.org/abs/2506.05431
Robustness Evaluation for Video Models with Reinforcement Learning. (12%)
Ashwin Ramesh Babu; Sajad Mousavi; Vineet Gundecha; Sahand Ghorbanpour; Avisek Naug; Antonio Guillen; Ricardo Luna Gutierrez; Soumyendu Sarkar
  Evaluating the robustness of Video classification models is very challenging,
specifically when compared to image-based models. With their increased temporal
dimension, there is a significant increase in complexity and computational
cost. One of the key challenges is to keep the perturbations to a minimum to
induce misclassification. In this work, we propose a multi-agent reinforcement
learning approach (spatial and temporal) that cooperatively learns to identify
the given video's sensitive spatial and temporal regions. The agents consider
temporal coherence in generating fine perturbations, leading to a more
effective and visually imperceptible attack. Our method outperforms the
state-of-the-art solutions on the Lp metric and the average queries. Our method
enables custom distortion types, making the robustness evaluation more relevant
to the use case. We extensively evaluate 4 popular models for video action
recognition on two popular datasets, HMDB-51 and UCF-101.


http://arxiv.org/abs/2506.05346
Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets. (12%)
Lei Hsiung; Tianyu Pang; Yung-Chen Tang; Linyue Song; Tsung-Yi Ho; Pin-Yu Chen; Yaoqing Yang
  Recent advancements in large language models (LLMs) have underscored their
vulnerability to safety alignment jailbreaks, particularly when subjected to
downstream fine-tuning. However, existing mitigation strategies primarily focus
on reactively addressing jailbreak incidents after safety guardrails have been
compromised, removing harmful gradients during fine-tuning, or continuously
reinforcing safety alignment throughout fine-tuning. As such, they tend to
overlook a critical upstream factor: the role of the original safety-alignment
data. This paper therefore investigates the degradation of safety guardrails
through the lens of representation similarity between upstream alignment
datasets and downstream fine-tuning tasks. Our experiments demonstrate that
high similarity between these datasets significantly weakens safety guardrails,
making models more susceptible to jailbreaks. Conversely, low similarity
between these two types of datasets yields substantially more robust models and
thus reduces harmfulness score by up to 10.33%. By highlighting the importance
of upstream dataset design in the building of durable safety guardrails and
reducing real-world vulnerability to jailbreak attacks, these findings offer
actionable insights for fine-tuning service providers.


http://arxiv.org/abs/2506.04713
Robust Few-Shot Vision-Language Model Adaptation. (8%)
Hanxin Wang; Tian Liu; Shu Kong
  Pretrained VLMs achieve strong performance on downstream tasks when adapted
with just a few labeled examples. As the adapted models inevitably encounter
out-of-distribution (OOD) test data that deviates from the in-distribution (ID)
task-specific training data, enhancing OOD generalization in few-shot
adaptation is critically important. We study robust few-shot VLM adaptation,
aiming to increase both ID and OOD accuracy. By comparing different adaptation
methods (e.g., prompt tuning, linear probing, contrastive finetuning, and full
finetuning), we uncover three key findings: (1) finetuning with proper
hyperparameters significantly outperforms the popular VLM adaptation methods
prompt tuning and linear probing; (2) visual encoder-only finetuning achieves
better efficiency and accuracy than contrastively finetuning both visual and
textual encoders; (3) finetuning the top layers of the visual encoder provides
the best balance between ID and OOD accuracy. Building on these findings, we
propose partial finetuning of the visual encoder empowered with two simple
augmentation techniques: (1) retrieval augmentation which retrieves
task-relevant data from the VLM's pretraining dataset to enhance adaptation,
and (2) adversarial perturbation which promotes robustness during finetuning.
Results show that the former/latter boosts OOD/ID accuracy while slightly
sacrificing the ID/OOD accuracy. Yet, perhaps understandably, naively combining
the two does not maintain their best OOD/ID accuracy. We address this dilemma
with the developed SRAPF, Stage-wise Retrieval Augmentation-based Adversarial
Partial Finetuning. SRAPF consists of two stages: (1) partial finetuning the
visual encoder using both ID and retrieved data, and (2) adversarial partial
finetuning with few-shot ID data. Extensive experiments demonstrate that SRAPF
achieves the state-of-the-art ID and OOD accuracy on the ImageNet OOD
benchmarks.


http://arxiv.org/abs/2506.06384
Detection Method for Prompt Injection by Integrating Pre-trained Model and Heuristic Feature Engineering. (2%)
Yi Ji; Runzhi Li; Baolei Mao
  With the widespread adoption of Large Language Models (LLMs), prompt
injection attacks have emerged as a significant security threat. Existing
defense mechanisms often face critical trade-offs between effectiveness and
generalizability. This highlights the urgent need for efficient prompt
injection detection methods that are applicable across a wide range of LLMs. To
address this challenge, we propose DMPI-PMHFE, a dual-channel feature fusion
detection framework. It integrates a pretrained language model with heuristic
feature engineering to detect prompt injection attacks. Specifically, the
framework employs DeBERTa-v3-base as a feature extractor to transform input
text into semantic vectors enriched with contextual information. In parallel,
we design heuristic rules based on known attack patterns to extract explicit
structural features commonly observed in attacks. Features from both channels
are subsequently fused and passed through a fully connected neural network to
produce the final prediction. This dual-channel approach mitigates the
limitations of relying only on DeBERTa to extract features. Experimental
results on diverse benchmark datasets demonstrate that DMPI-PMHFE outperforms
existing methods in terms of accuracy, recall, and F1-score. Furthermore, when
deployed actually, it significantly reduces attack success rates across
mainstream LLMs, including GLM-4, LLaMA 3, Qwen 2.5, and GPT-4o.


http://arxiv.org/abs/2506.03765
Prediction Inconsistency Helps Achieve Generalizable Detection of Adversarial Examples. (99%)
Sicong Han; Chenhao Lin; Zhengyu Zhao; Xiyuan Wang; Xinlei He; Qian Li; Cong Wang; Qian Wang; Chao Shen
  Adversarial detection protects models from adversarial attacks by refusing
suspicious test samples. However, current detection methods often suffer from
weak generalization: their effectiveness tends to degrade significantly when
applied to adversarially trained models rather than naturally trained ones, and
they generally struggle to achieve consistent effectiveness across both
white-box and black-box attack settings. In this work, we observe that an
auxiliary model, differing from the primary model in training strategy or model
architecture, tends to assign low confidence to the primary model's predictions
on adversarial examples (AEs), while preserving high confidence on normal
examples (NEs). Based on this discovery, we propose Prediction Inconsistency
Detector (PID), a lightweight and generalizable detection framework to
distinguish AEs from NEs by capturing the prediction inconsistency between the
primal and auxiliary models. PID is compatible with both naturally and
adversarially trained primal models and outperforms four detection methods
across 3 white-box, 3 black-box, and 1 mixed adversarial attacks. Specifically,
PID achieves average AUC scores of 99.29\% and 99.30\% on CIFAR-10 when the
primal model is naturally and adversarially trained, respectively, and 98.31%
and 96.81% on ImageNet under the same conditions, outperforming existing SOTAs
by 4.70%$\sim$25.46%.


http://arxiv.org/abs/2506.03988
RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors. (99%)
Hicham Eddoubi; Jonas Ricker; Federico Cocchi; Lorenzo Baraldi; Angelo Sotgiu; Maura Pintor; Marcella Cornia; Lorenzo Baraldi; Asja Fischer; Rita Cucchiara; Battista Biggio
  AI-generated images have reached a quality level at which humans are
incapable of reliably distinguishing them from real images. To counteract the
inherent risk of fraud and disinformation, the detection of AI-generated images
is a pressing challenge and an active research topic. While many of the
presented methods claim to achieve high detection accuracy, they are usually
evaluated under idealized conditions. In particular, the adversarial robustness
is often neglected, potentially due to a lack of awareness or the substantial
effort required to conduct a comprehensive robustness analysis. In this work,
we tackle this problem by providing a simpler means to assess the robustness of
AI-generated image detectors. We present RAID (Robust evaluation of
AI-generated image Detectors), a dataset of 72k diverse and highly transferable
adversarial examples. The dataset is created by running attacks against an
ensemble of seven state-of-the-art detectors and images generated by four
different text-to-image models. Extensive experiments show that our methodology
generates adversarial images that transfer with a high success rate to unseen
detectors, which can be used to quickly provide an approximate yet still
reliable estimate of a detector's adversarial robustness. Our findings indicate
that current state-of-the-art AI-generated image detectors can be easily
deceived by adversarial examples, highlighting the critical need for the
development of more robust methods. We release our dataset at
https://huggingface.co/datasets/aimagelab/RAID and evaluation code at
https://github.com/pralab/RAID.


http://arxiv.org/abs/2506.03627
Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks. (98%)
Lin Mu; Guowei Chu; Li Ni; Lei Sang; Zhize Wu; Peiquan Jin; Yiwen Zhang
  Large Language Models (LLMs) have demonstrated remarkable performance across
various tasks by effectively utilizing a prompting strategy. However, they are
highly sensitive to input perturbations, such as typographical errors or slight
character order errors, which can substantially degrade their performance.
Despite advances in prompting techniques, developing a prompting strategy that
explicitly mitigates the negative impact of such perturbations remains an open
challenge. To bridge this gap, we propose Robustness of Prompting (RoP), a
novel prompting strategy specifically designed to enhance the robustness of
LLMs. RoP consists of two stages: Error Correction and Guidance. In the Error
Correction stage, RoP applies diverse perturbation methods to generate
adversarial examples, which are then used to construct prompts that
automatically correct input errors. In the Guidance stage, RoP generates an
optimal guidance prompting based on the corrected input, steering the model
toward more robust and accurate inferences. Through comprehensive experiments
spanning arithmetic, commonsense, and logical reasoning tasks, we demonstrate
that RoP significantly improves LLMs' robustness against adversarial
perturbations. Notably, it maintains model accuracy with only minimal
degradation compared to clean input scenarios, thereby establishing RoP as a
practical and effective approach for enhancing LLM robustness in real-world
applications.


http://arxiv.org/abs/2506.05403
Poisoning Behavioral-based Worker Selection in Mobile Crowdsensing using Generative Adversarial Networks. (93%)
Ruba Nasser; Ahmed Alagha; Shakti Singh; Rabeb Mizouni; Hadi Otrok; Jamal Bentahar
  With the widespread adoption of Artificial intelligence (AI), AI-based tools
and components are becoming omnipresent in today's solutions. However, these
components and tools are posing a significant threat when it comes to
adversarial attacks. Mobile Crowdsensing (MCS) is a sensing paradigm that
leverages the collective participation of workers and their smart devices to
collect data. One of the key challenges faced at the selection stage is
ensuring task completion due to workers' varying behavior. AI has been utilized
to tackle this challenge by building unique models for each worker to predict
their behavior. However, the integration of AI into the system introduces
vulnerabilities that can be exploited by malicious insiders to reduce the
revenue obtained by victim workers. This work proposes an adversarial attack
targeting behavioral-based selection models in MCS. The proposed attack
leverages Generative Adversarial Networks (GANs) to generate poisoning points
that can mislead the models during the training stage without being detected.
This way, the potential damage introduced by GANs on worker selection in MCS
can be anticipated. Simulation results using a real-life dataset show the
effectiveness of the proposed attack in compromising the victim workers' model
and evading detection by an outlier detector, compared to a benchmark. In
addition, the impact of the attack on reducing the payment obtained by victim
workers is evaluated.


http://arxiv.org/abs/2506.03933
DiffCAP: Diffusion-based Cumulative Adversarial Purification for Vision Language Models. (92%)
Jia Fu; Yongtao Wu; Yihang Chen; Kunyu Peng; Xiao Zhang; Volkan Cevher; Sepideh Pashami; Anders Holst
  Vision Language Models (VLMs) have shown remarkable capabilities in
multimodal understanding, yet their susceptibility to perturbations poses a
significant threat to their reliability in real-world applications. Despite
often being imperceptible to humans, these perturbations can drastically alter
model outputs, leading to erroneous interpretations and decisions. This paper
introduces DiffCAP, a novel diffusion-based purification strategy that can
effectively neutralize adversarial corruptions in VLMs. We observe that adding
minimal noise to an adversarially corrupted image significantly alters its
latent embedding with respect to VLMs. Building on this insight, DiffCAP
cumulatively injects random Gaussian noise into adversarially perturbed input
data. This process continues until the embeddings of two consecutive noisy
images reach a predefined similarity threshold, indicating a potential approach
to neutralize the adversarial effect. Subsequently, a pretrained diffusion
model is employed to denoise the stabilized image, recovering a clean
representation suitable for the VLMs to produce an output. Through extensive
experiments across six datasets with three VLMs under varying attack strengths
in three task scenarios, we show that DiffCAP consistently outperforms existing
defense techniques by a substantial margin. Notably, DiffCAP significantly
reduces both hyperparameter tuning complexity and the required diffusion time,
thereby accelerating the denoising process. Equipped with strong theoretical
and empirical support, DiffCAP provides a robust and practical solution for
securely deploying VLMs in adversarial environments.


http://arxiv.org/abs/2506.04390
Through the Stealth Lens: Rethinking Attacks and Defenses in RAG. (81%)
Sarthak Choudhary; Nils Palumbo; Ashish Hooda; Krishnamurthy Dj Dvijotham; Somesh Jha
  Retrieval-augmented generation (RAG) systems are vulnerable to attacks that
inject poisoned passages into the retrieved set, even at low corruption rates.
We show that existing attacks are not designed to be stealthy, allowing
reliable detection and mitigation. We formalize stealth using a
distinguishability-based security game. If a few poisoned passages are designed
to control the response, they must differentiate themselves from benign ones,
inherently compromising stealth. This motivates the need for attackers to
rigorously analyze intermediate signals involved in
generation$\unicode{x2014}$such as attention patterns or next-token probability
distributions$\unicode{x2014}$to avoid easily detectable traces of
manipulation. Leveraging attention patterns, we propose a passage-level
score$\unicode{x2014}$the Normalized Passage Attention
Score$\unicode{x2014}$used by our Attention-Variance Filter algorithm to
identify and filter potentially poisoned passages. This method mitigates
existing attacks, improving accuracy by up to $\sim 20 \%$ over baseline
defenses. To probe the limits of attention-based defenses, we craft stealthier
adaptive attacks that obscure such traces, achieving up to $35 \%$ attack
success rate, and highlight the challenges in improving stealth.


http://arxiv.org/abs/2506.04036
Privacy and Security Threat for OpenAI GPTs. (67%)
Wei Wenying; Zhao Kaifa; Xue Lei; Fan Ming
  Large language models (LLMs) demonstrate powerful information handling
capabilities and are widely integrated into chatbot applications. OpenAI
provides a platform for developers to construct custom GPTs, extending
ChatGPT's functions and integrating external services. Since its release in
November 2023, over 3 million custom GPTs have been created. However, such a
vast ecosystem also conceals security and privacy threats. For developers,
instruction leaking attacks threaten the intellectual property of instructions
in custom GPTs through carefully crafted adversarial prompts. For users,
unwanted data access behavior by custom GPTs or integrated third-party services
raises significant privacy concerns. To systematically evaluate the scope of
threats in real-world LLM applications, we develop three phases instruction
leaking attacks target GPTs with different defense level. Our widespread
experiments on 10,000 real-world custom GPTs reveal that over 98.8% of GPTs are
vulnerable to instruction leaking attacks via one or more adversarial prompts,
and half of the remaining GPTs can also be attacked through multiround
conversations. We also developed a framework to assess the effectiveness of
defensive strategies and identify unwanted behaviors in custom GPTs. Our
findings show that 77.5% of custom GPTs with defense strategies are vulnerable
to basic instruction leaking attacks. Additionally, we reveal that 738 custom
GPTs collect user conversational information, and identified 8 GPTs exhibiting
data access behaviors that are unnecessary for their intended functionalities.
Our findings raise awareness among GPT developers about the importance of
integrating specific defensive strategies in their instructions and highlight
users' concerns about data privacy when using LLM-based applications.


http://arxiv.org/abs/2506.04556
BESA: Boosting Encoder Stealing Attack with Perturbation Recovery. (67%)
Xuhao Ren; Haotian Liang; Yajie Wang; Chuan Zhang; Zehui Xiong; Liehuang Zhu
  To boost the encoder stealing attack under the perturbation-based defense
that hinders the attack performance, we propose a boosting encoder stealing
attack with perturbation recovery named BESA. It aims to overcome
perturbation-based defenses. The core of BESA consists of two modules:
perturbation detection and perturbation recovery, which can be combined with
canonical encoder stealing attacks. The perturbation detection module utilizes
the feature vectors obtained from the target encoder to infer the defense
mechanism employed by the service provider. Once the defense mechanism is
detected, the perturbation recovery module leverages the well-designed
generative model to restore a clean feature vector from the perturbed one.
Through extensive evaluations based on various datasets, we demonstrate that
BESA significantly enhances the surrogate encoder accuracy of existing encoder
stealing attacks by up to 24.63\% when facing state-of-the-art defenses and
combinations of multiple defenses.


http://arxiv.org/abs/2506.03683
PRJ: Perception-Retrieval-Judgement for Generated Images. (1%)
Qiang Fu; Zonglei Jing; Zonghao Ying; Xiaoqian Li
  The rapid progress of generative AI has enabled remarkable creative
capabilities, yet it also raises urgent concerns regarding the safety of
AI-generated visual content in real-world applications such as content
moderation, platform governance, and digital media regulation. This includes
unsafe material such as sexually explicit images, violent scenes, hate symbols,
propaganda, and unauthorized imitations of copyrighted artworks. Existing image
safety systems often rely on rigid category filters and produce binary outputs,
lacking the capacity to interpret context or reason about nuanced,
adversarially induced forms of harm. In addition, standard evaluation metrics
(e.g., attack success rate) fail to capture the semantic severity and dynamic
progression of toxicity. To address these limitations, we propose
Perception-Retrieval-Judgement (PRJ), a cognitively inspired framework that
models toxicity detection as a structured reasoning process. PRJ follows a
three-stage design: it first transforms an image into descriptive language
(perception), then retrieves external knowledge related to harm categories and
traits (retrieval), and finally evaluates toxicity based on legal or normative
rules (judgement). This language-centric structure enables the system to detect
both explicit and implicit harms with improved interpretability and categorical
granularity. In addition, we introduce a dynamic scoring mechanism based on a
contextual toxicity risk matrix to quantify harmfulness across different
semantic dimensions. Experiments show that PRJ surpasses existing safety
checkers in detection accuracy and robustness while uniquely supporting
structured category-level toxicity interpretation.


http://arxiv.org/abs/2506.03701
Tournament Robustness via Redundancy. (1%)
Klim Efremenko; Hendrik Molter; Meirav Zehavi
  A knockout tournament is one of the most simple and popular forms of
competition. Here, we are given a binary tournament tree where all leaves are
labeled with seed position names. The players participating in the tournament
are assigned to the seed positions. In each round, the two players assigned to
leaves of the tournament tree with a common parent compete, and the winner is
promoted to the parent. The last remaining player is the winner of the
tournament.
  In this work, we study the problem of making knock-out tournaments robust
against manipulation, where the form of manipulation we consider is changing
the outcome of a game. We assume that our input is only the number of players
that compete in the tournament, and the number of manipulations against which
the tournament should be robust. Furthermore, we assume that there is a
strongest player, that is, a player that beats any of the other players.
However, the identity of this player is not part of the problem input.
  To ensure robustness against manipulation, we uncover an unexpected
connection between the problem at hand and communication protocols that utilize
a feedback channel, offering resilience against adversarial noise. We explore
the trade-off between the size of the robust tournament tree and the degree of
protection against manipulation. Specifically, we demonstrate that it is
possible to tolerate up to a $1/3$ fraction of manipulations along each
leaf-to-root path, at the cost of only a polynomial blow-up in the tournament
size.


http://arxiv.org/abs/2506.05382
How stealthy is stealthy? Studying the Efficacy of Black-Box Adversarial Attacks in the Real World. (99%)
Francesco Panebianco; Mario D'Onghia; Stefano Zanero aand Michele Carminati
  Deep learning systems, critical in domains like autonomous vehicles, are
vulnerable to adversarial examples (crafted inputs designed to mislead
classifiers). This study investigates black-box adversarial attacks in computer
vision. This is a realistic scenario, where attackers have query-only access to
the target model. Three properties are introduced to evaluate attack
feasibility: robustness to compression, stealthiness to automatic detection,
and stealthiness to human inspection. State-of-the-Art methods tend to
prioritize one criterion at the expense of others. We propose ECLIPSE, a novel
attack method employing Gaussian blurring on sampled gradients and a local
surrogate model. Comprehensive experiments on a public dataset highlight
ECLIPSE's advantages, demonstrating its contribution to the trade-off between
the three properties.


http://arxiv.org/abs/2506.04263
Dynamic Epsilon Scheduling: A Multi-Factor Adaptive Perturbation Budget for Adversarial Training. (99%)
Alan Mitkiy; James Smith; Hana Satou; Hiroshi Tanaka; Emily Johnson; F Monkey
  Adversarial training is among the most effective strategies for defending
deep neural networks against adversarial examples. A key limitation of existing
adversarial training approaches lies in their reliance on a fixed perturbation
budget, which fails to account for instance-specific robustness
characteristics. While prior works such as IAAT and MMA introduce
instance-level adaptations, they often rely on heuristic or static
approximations of data robustness. In this paper, we propose Dynamic Epsilon
Scheduling (DES), a novel framework that adaptively adjusts the adversarial
perturbation budget per instance and per training iteration. DES integrates
three key factors: (1) the distance to the decision boundary approximated via
gradient-based proxies, (2) prediction confidence derived from softmax entropy,
and (3) model uncertainty estimated via Monte Carlo dropout. By combining these
cues into a unified scheduling strategy, DES tailors the perturbation budget
dynamically to guide more effective adversarial learning. Experimental results
on CIFAR-10 and CIFAR-100 show that our method consistently improves both
adversarial robustness and standard accuracy compared to fixed-epsilon
baselines and prior adaptive methods. Moreover, we provide theoretical insights
into the stability and convergence of our scheduling policy. This work opens a
new avenue for instance-aware, data-driven adversarial training methods.


http://arxiv.org/abs/2506.02660
Tarallo: Evading Behavioral Malware Detectors in the Problem Space. (98%)
Gabriele Digregorio; Salvatore Maccarrone; Mario D'Onghia; Luigi Gallo; Michele Carminati; Mario Polino; Stefano Zanero
  Machine learning algorithms can effectively classify malware through dynamic
behavior but are susceptible to adversarial attacks. Existing attacks, however,
often fail to find an effective solution in both the feature and problem
spaces. This issue arises from not addressing the intrinsic nondeterministic
nature of malware, namely executing the same sample multiple times may yield
significantly different behaviors. Hence, the perturbations computed for a
specific behavior may be ineffective for others observed in subsequent
executions. In this paper, we show how an attacker can augment their chance of
success by leveraging a new and more efficient feature space algorithm for
sequential data, which we have named PS-FGSM, and by adopting two problem space
strategies specially tailored to address nondeterminism in the problem space.
We implement our novel algorithm and attack strategies in Tarallo, an
end-to-end adversarial framework that significantly outperforms previous works
in both white and black-box scenarios. Our preliminary analysis in a sandboxed
environment and against two RNN-based malware detectors, shows that Tarallo
achieves a success rate up to 99% on both feature and problem space attacks
while significantly minimizing the number of modifications required for
misclassification.


http://arxiv.org/abs/2506.02711
Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack. (98%)
Jing Xue; Zhishen Sun; Haishan Ye; Luo Luo; Xiangyu Chang; Ivor Tsang; Guang Dai
  Membership inference attack (MIA) has become one of the most widely used and
effective methods for evaluating the privacy risks of machine learning models.
These attacks aim to determine whether a specific sample is part of the model's
training set by analyzing the model's output. While traditional membership
inference attacks focus on leveraging the model's posterior output, such as
confidence on the target sample, we propose IMIA, a novel attack strategy that
utilizes the process of generating adversarial samples to infer membership. We
propose to infer the member properties of the target sample using the number of
iterations required to generate its adversarial sample. We conduct experiments
across multiple models and datasets, and our results demonstrate that the
number of iterations for generating an adversarial sample is a reliable feature
for membership inference, achieving strong performance both in black-box and
white-box attack scenarios. This work provides a new perspective for evaluating
model privacy and highlights the potential of adversarial example-based
features for privacy leakage assessment.


http://arxiv.org/abs/2506.02978
On the Robustness of Tabular Foundation Models: Test-Time Attacks and In-Context Defenses. (96%)
Mohamed Djilani; Thibault Simonetto; Karim Tit; Florian Tambon; Paul Récamier; Salah Ghamizi; Maxime Cordy; Mike Papadakis
  Recent tabular Foundational Models (FM) such as TabPFN and TabICL, leverage
in-context learning to achieve strong performance without gradient updates or
fine-tuning. However, their robustness to adversarial manipulation remains
largely unexplored. In this work, we present a comprehensive study of the
adversarial vulnerabilities of tabular FM, focusing on both their fragility to
targeted test-time attacks and their potential misuse as adversarial tools. We
show on three benchmarks in finance, cybersecurity and healthcare, that small,
structured perturbations to test inputs can significantly degrade prediction
accuracy, even when training context remain fixed. Additionally, we demonstrate
that tabular FM can be repurposed to generate transferable evasion to
conventional models such as random forests and XGBoost, and on a lesser extent
to deep tabular models. To improve tabular FM, we formulate the robustification
problem as an optimization of the weights (adversarial fine-tuning), or the
context (adversarial in-context learning). We introduce an in-context
adversarial training strategy that incrementally replaces the context with
adversarial perturbed instances, without updating model weights. Our approach
improves robustness across multiple tabular benchmarks. Together, these
findings position tabular FM as both a target and a source of adversarial
threats, highlighting the urgent need for robust training and evaluation
practices in this emerging paradigm.


http://arxiv.org/abs/2506.05402
Sylva: Tailoring Personalized Adversarial Defense in Pre-trained Models via Collaborative Fine-tuning. (95%)
Tianyu Qi; Lei Xue; Yufeng Zhan; Xiaobo Ma
  The growing adoption of large pre-trained models in edge computing has made
deploying model inference on mobile clients both practical and popular. These
devices are inherently vulnerable to direct adversarial attacks, which pose a
substantial threat to the robustness and security of deployed models. Federated
adversarial training (FAT) has emerged as an effective solution to enhance
model robustness while preserving client privacy. However, FAT frequently
produces a generalized global model, which struggles to address the diverse and
heterogeneous data distributions across clients, resulting in insufficiently
personalized performance, while also encountering substantial communication
challenges during the training process. In this paper, we propose
\textit{Sylva}, a personalized collaborative adversarial training framework
designed to deliver customized defense models for each client through a
two-phase process. In Phase 1, \textit{Sylva} employs LoRA for local
adversarial fine-tuning, enabling clients to personalize model robustness while
drastically reducing communication costs by uploading only LoRA parameters
during federated aggregation. In Phase 2, a game-based layer selection strategy
is introduced to enhance accuracy on benign data, further refining the
personalized model. This approach ensures that each client receives a tailored
defense model that balances robustness and accuracy effectively. Extensive
experiments on benchmark datasets demonstrate that \textit{Sylva} can achieve
up to 50$\times$ improvements in communication efficiency compared to
state-of-the-art algorithms, while achieving up to 29.5\% and 50.4\%
enhancements in adversarial robustness and benign accuracy, respectively.


http://arxiv.org/abs/2506.05394
Attacking Attention of Foundation Models Disrupts Downstream Tasks. (87%)
Hondamunige Prasanna Silva; Federico Becattini; Lorenzo Seidenari
  Foundation models represent the most prominent and recent paradigm shift in
artificial intelligence. Foundation models are large models, trained on broad
data that deliver high accuracy in many downstream tasks, often without
fine-tuning. For this reason, models such as CLIP , DINO or Vision Transfomers
(ViT), are becoming the bedrock of many industrial AI-powered applications.
However, the reliance on pre-trained foundation models also introduces
significant security concerns, as these models are vulnerable to adversarial
attacks. Such attacks involve deliberately crafted inputs designed to deceive
AI systems, jeopardizing their reliability. This paper studies the
vulnerabilities of vision foundation models, focusing specifically on CLIP and
ViTs, and explores the transferability of adversarial attacks to downstream
tasks. We introduce a novel attack, targeting the structure of
transformer-based architectures in a task-agnostic fashion. We demonstrate the
effectiveness of our attack on several downstream tasks: classification,
captioning, image/text retrieval, segmentation and depth estimation. Code
available at:https://github.com/HondamunigePrasannaSilva/attack-attention


http://arxiv.org/abs/2506.02679
Poster: FedBlockParadox -- A Framework for Simulating and Securing Decentralized Federated Learning. (64%)
Gabriele Digregorio; Francesco Bleggi; Federico Caroli; Michele Carminati; Stefano Zanero; Stefano Longari
  A significant body of research in decentralized federated learning focuses on
combining the privacy-preserving properties of federated learning with the
resilience and transparency offered by blockchain-based systems. While these
approaches are promising, they often lack flexible tools to evaluate system
robustness under adversarial conditions. To fill this gap, we present
FedBlockParadox, a modular framework for modeling and evaluating decentralized
federated learning systems built on blockchain technologies, with a focus on
resilience against a broad spectrum of adversarial attack scenarios. It
supports multiple consensus protocols, validation methods, aggregation
strategies, and configurable attack models. By enabling controlled experiments,
FedBlockParadox provides a valuable resource for researchers developing secure,
decentralized learning solutions. The framework is open-source and built to be
extensible by the community.


http://arxiv.org/abs/2506.05401
Robust Anti-Backdoor Instruction Tuning in LVLMs. (47%)
Yuan Xun; Siyuan Liang; Xiaojun Jia; Xinwei Liu; Xiaochun Cao
  Large visual language models (LVLMs) have demonstrated excellent
instruction-following capabilities, yet remain vulnerable to stealthy backdoor
attacks when finetuned using contaminated data. Existing backdoor defense
techniques are usually developed for single-modal visual or language models
under fully parameter-adjustable settings or rely on supervisory knowledge
during training. However, in real-world scenarios, defenders cannot modify
frozen visual encoders or core LLM parameters, nor possess prior knowledge of
unknown trigger patterns or target responses. Motivated by the empirical
finding that LVLMs readily overfit to fixed, unknown triggers, which can embed
malicious associations during adapter-level tuning, we aim to design a defense
that operates without access to core weights or attack priors. To this end, we
introduce a lightweight, certified-agnostic defense framework, Robust
Instruction Tuning, that finetunes only adapter modules and text embedding
layers under instruction tuning. Our method integrates two complementary
regularizations: (1) Input Diversity Regularization, which perturbs trigger
components across training samples to disrupt consistent spurious cues; and (2)
Anomalous Activation Regularization, which dynamically sparses adapter weights
exhibiting abnormally sharp activations linked to backdoor patterns. These
mechanisms jointly guide the model toward learning semantically grounded
representations rather than memorizing superficial trigger-response mappings.
  Extensive experiments against seven attacks on Flickr30k and MSCOCO
demonstrate that ours
  reduces their attack success rate to nearly zero, with an increase in
training cost of less than 15%.


http://arxiv.org/abs/2506.02479
BitBypass: A New Direction in Jailbreaking Aligned Large Language Models with Bitstream Camouflage. (22%)
Kalyan Nakka; Nitesh Saxena
  The inherent risk of generating harmful and unsafe content by Large Language
Models (LLMs), has highlighted the need for their safety alignment. Various
techniques like supervised fine-tuning, reinforcement learning from human
feedback, and red-teaming were developed for ensuring the safety alignment of
LLMs. However, the robustness of these aligned LLMs is always challenged by
adversarial attacks that exploit unexplored and underlying vulnerabilities of
the safety alignment. In this paper, we develop a novel black-box jailbreak
attack, called BitBypass, that leverages hyphen-separated bitstream camouflage
for jailbreaking aligned LLMs. This represents a new direction in jailbreaking
by exploiting fundamental information representation of data as continuous
bits, rather than leveraging prompt engineering or adversarial manipulations.
Our evaluation of five state-of-the-art LLMs, namely GPT-4o, Gemini 1.5, Claude
3.5, Llama 3.1, and Mixtral, in adversarial perspective, revealed the
capabilities of BitBypass in bypassing their safety alignment and tricking them
into generating harmful and unsafe content. Further, we observed that BitBypass
outperforms several state-of-the-art jailbreak attacks in terms of stealthiness
and attack success. Overall, these results highlights the effectiveness and
efficiency of BitBypass in jailbreaking these state-of-the-art LLMs.


http://arxiv.org/abs/2506.03350
Adversarial Attacks on Robotic Vision Language Action Models. (13%)
Eliot Krzysztof Jones; Alexander Robey; Andy Zou; Zachary Ravichandran; George J. Pappas; Hamed Hassani; Matt Fredrikson; J. Zico Kolter
  The emergence of vision-language-action models (VLAs) for end-to-end control
is reshaping the field of robotics by enabling the fusion of multimodal sensory
inputs at the billion-parameter scale. The capabilities of VLAs stem primarily
from their architectures, which are often based on frontier large language
models (LLMs). However, LLMs are known to be susceptible to adversarial misuse,
and given the significant physical risks inherent to robotics, questions remain
regarding the extent to which VLAs inherit these vulnerabilities. Motivated by
these concerns, in this work we initiate the study of adversarial attacks on
VLA-controlled robots. Our main algorithmic contribution is the adaptation and
application of LLM jailbreaking attacks to obtain complete control authority
over VLAs. We find that textual attacks, which are applied once at the
beginning of a rollout, facilitate full reachability of the action space of
commonly used VLAs and often persist over longer horizons. This differs
significantly from LLM jailbreaking literature, as attacks in the real world do
not have to be semantically linked to notions of harm. We make all code
available at https://github.com/eliotjones1/robogcg .


http://arxiv.org/abs/2506.03355
Robustness in Both Domains: CLIP Needs a Robust Text Encoder. (3%)
Elias Abad Rocamora; Christian Schlarmann; Naman Deep Singh; Yongtao Wu; Matthias Hein; Volkan Cevher
  Adversarial input attacks can cause a significant shift of CLIP embeddings.
This can affect the downstream robustness of models incorporating CLIP in the
pipeline, such as text-to-image generative models or large vision language
models. While some efforts have been done towards making the CLIP image
encoders robust, the robustness of text encoders remains unexplored. In this
work, we cover this gap in the literature. We propose LEAF: an efficient
adversarial finetuning method for the text domain, with the ability to scale to
large CLIP models. Our models significantly improve the zero-shot adversarial
accuracy in the text domain, while maintaining the vision performance provided
by robust image encoders. When combined with text-to-image diffusion models, we
can improve the generation quality under adversarial noise. When employing our
robust CLIP encoders in multimodal retrieval tasks, we improve the recall under
adversarial noise over standard CLIP models. Finally, we show that robust text
encoders facilitate better reconstruction of input text from its embedding via
direct optimization.


http://arxiv.org/abs/2506.03089
Explicitly Modeling Subcortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness. (2%)
Lucas Piper; Arlindo L. Oliveira; Tiago Marques
  Convolutional neural networks (CNNs) trained on object recognition achieve
high task performance but continue to exhibit vulnerability under a range of
visual perturbations and out-of-domain images, when compared with biological
vision. Prior work has demonstrated that coupling a standard CNN with a
front-end block (VOneBlock) that mimics the primate primary visual cortex (V1)
can improve overall model robustness. Expanding on this, we introduce Early
Vision Networks (EVNets), a new class of hybrid CNNs that combine the VOneBlock
with a novel SubcorticalBlock, whose architecture draws from computational
models in neuroscience and is parameterized to maximize alignment with
subcortical responses reported across multiple experimental studies. Without
being optimized to do so, the assembly of the SubcorticalBlock with the
VOneBlock improved V1 alignment across most standard V1 benchmarks, and better
modeled extra-classical receptive field phenomena. In addition, EVNets exhibit
stronger emergent shape bias and overperform the base CNN architecture by 8.5%
on an aggregate benchmark of robustness evaluations, including adversarial
perturbations, common corruptions, and domain shifts. Finally, we show that
EVNets can be further improved when paired with a state-of-the-art data
augmentation technique, surpassing the performance of the isolated data
augmentation approach by 7.3% on our robustness benchmark. This result reveals
complementary benefits between changes in architecture to better mimic biology
and training-based machine learning approaches.


http://arxiv.org/abs/2506.03234
BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF. (1%)
Kaiwen Duan; Hongwei Yao; Yufei Chen; Ziyun Li; Tong Qiao; Zhan Qin; Cong Wang
  Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning
text-to-image (T2I) models with human preferences. However, RLHF's feedback
mechanism also opens new pathways for adversaries. This paper demonstrates the
feasibility of hijacking T2I models by poisoning a small fraction of preference
data with natural-appearing examples. Specifically, we propose BadReward, a
stealthy clean-label poisoning attack targeting the reward model in multi-modal
RLHF. BadReward operates by inducing feature collisions between visually
contradicted preference data instances, thereby corrupting the reward model and
indirectly compromising the T2I model's integrity. Unlike existing alignment
poisoning techniques focused on single (text) modality, BadReward is
independent of the preference annotation process, enhancing its stealth and
practical threat. Extensive experiments on popular T2I models show that
BadReward can consistently guide the generation towards improper outputs, such
as biased or violent imagery, for targeted concepts. Our findings underscore
the amplified threat landscape for RLHF in multi-modal systems, highlighting
the urgent need for robust defenses. Disclaimer. This paper contains uncensored
toxic content that might be offensive or disturbing to the readers.


http://arxiv.org/abs/2506.01511
Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment. (98%)
Kaixun Jiang; Zhaoyu Chen; Haijing Guo; Jinglun Li; Jiyuan Fu; Pinxue Guo; Hao Tang; Bo Li; Wenqiang Zhang
  Preference alignment in diffusion models has primarily focused on benign
human preferences (e.g., aesthetic). In this paper, we propose a novel
perspective: framing unrestricted adversarial example generation as a problem
of aligning with adversary preferences. Unlike benign alignment, adversarial
alignment involves two inherently conflicting preferences: visual consistency
and attack effectiveness, which often lead to unstable optimization and reward
hacking (e.g., reducing visual quality to improve attack success). To address
this, we propose APA (Adversary Preferences Alignment), a two-stage framework
that decouples conflicting preferences and optimizes each with differentiable
rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency
using rule-based similarity reward. In the second stage, APA updates either the
image latent or prompt embedding based on feedback from a substitute
classifier, guided by trajectory-level and step-wise rewards. To enhance
black-box transferability, we further incorporate a diffusion augmentation
strategy. Experiments demonstrate that APA achieves significantly better attack
transferability while maintaining high visual consistency, inspiring further
research to approach adversarial attacks from an alignment perspective. Code
will be available at https://github.com/deep-kaixun/APA.


http://arxiv.org/abs/2506.02362
MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models. (81%)
Xueqi Cheng; Minxing Zheng; Shixiang Zhu; Yushun Dong
  Model extraction attacks aim to replicate the functionality of a black-box
model through query access, threatening the intellectual property (IP) of
machine-learning-as-a-service (MLaaS) providers. Defending against such attacks
is challenging, as it must balance efficiency, robustness, and utility
preservation in the real-world scenario. Despite the recent advances, most
existing defenses presume that attacker queries have out-of-distribution (OOD)
samples, enabling them to detect and disrupt suspicious inputs. However, this
assumption is increasingly unreliable, as modern models are trained on diverse
datasets and attackers often operate under limited query budgets. As a result,
the effectiveness of these defenses is significantly compromised in realistic
deployment scenarios. To address this gap, we propose MISLEADER (enseMbles of
dIStiLled modEls Against moDel ExtRaction), a novel defense strategy that does
not rely on OOD assumptions. MISLEADER formulates model protection as a bilevel
optimization problem that simultaneously preserves predictive fidelity on
benign inputs and reduces extractability by potential clone models. Our
framework combines data augmentation to simulate attacker queries with an
ensemble of heterogeneous distilled models to improve robustness and diversity.
We further provide a tractable approximation algorithm and derive theoretical
error bounds to characterize defense effectiveness. Extensive experiments
across various settings validate the utility-preserving and
extraction-resistant properties of our proposed defense strategy. Our code is
available at https://github.com/LabRAI/MISLEADER.


http://arxiv.org/abs/2506.01591
Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation. (78%)
Yuan Gan; Jiaxu Miao; Yunze Wang; Yi Yang
  Advances in talking-head animation based on Latent Diffusion Models (LDM)
enable the creation of highly realistic, synchronized videos. These fabricated
videos are indistinguishable from real ones, increasing the risk of potential
misuse for scams, political manipulation, and misinformation. Hence, addressing
these ethical concerns has become a pressing issue in AI security. Recent
proactive defense studies focused on countering LDM-based models by adding
perturbations to portraits. However, these methods are ineffective at
protecting reference portraits from advanced image-to-video animation. The
limitations are twofold: 1) they fail to prevent images from being manipulated
by audio signals, and 2) diffusion-based purification techniques can
effectively eliminate protective perturbations. To address these challenges, we
propose Silencer, a two-stage method designed to proactively protect the
privacy of portraits. First, a nullifying loss is proposed to ignore audio
control in talking-head generation. Second, we apply anti-purification loss in
LDM to optimize the inverted latent feature to generate robust perturbations.
Extensive experiments demonstrate the effectiveness of Silencer in proactively
protecting portrait privacy. We hope this work will raise awareness among the
AI security community regarding critical ethical issues related to talking-head
generation techniques. Code: https://github.com/yuangan/Silencer.


http://arxiv.org/abs/2506.01825
Which Factors Make Code LLMs More Vulnerable to Backdoor Attacks? A Systematic Study. (54%)
Chenyu Wang; Zhou Yang; Yaniv Harel; David Lo
  Code LLMs are increasingly employed in software development. However, studies
have shown that they are vulnerable to backdoor attacks: when a trigger (a
specific input pattern) appears in the input, the backdoor will be activated
and cause the model to generate malicious outputs. Researchers have designed
various triggers and demonstrated the feasibility of implanting backdoors by
poisoning a fraction of the training data. Some basic conclusions have been
made, such as backdoors becoming easier to implant when more training data are
modified. However, existing research has not explored other factors influencing
backdoor attacks on Code LLMs, such as training batch size, epoch number, and
the broader design space for triggers, e.g., trigger length.
  To bridge this gap, we use code summarization as an example to perform an
empirical study that systematically investigates the factors affecting backdoor
effectiveness and understands the extent of the threat posed. Three categories
of factors are considered: data, model, and inference, revealing previously
overlooked findings. We find that the prevailing consensus -- that attacks are
ineffective at extremely low poisoning rates -- is incorrect. The absolute
number of poisoned samples matters as well. Specifically, poisoning just 20 out
of 454K samples (0.004\% poisoning rate -- far below the minimum setting of
0.1\% in prior studies) successfully implants backdoors! Moreover, the common
defense is incapable of removing even a single poisoned sample from it.
Additionally, small batch sizes increase the risk of backdoor attacks. We also
uncover other critical factors such as trigger types, trigger length, and the
rarity of tokens in the triggers, leading to valuable insights for assessing
Code LLMs' vulnerability to backdoor attacks. Our study highlights the urgent
need for defense mechanisms against extremely low poisoning rate settings.


http://arxiv.org/abs/2506.02156
Mitigating Data Poisoning Attacks to Local Differential Privacy. (13%)
Xiaolin Li; Ninghui Li; Boyang Wang; Wenhai Sun
  The distributed nature of local differential privacy (LDP) invites data
poisoning attacks and poses unforeseen threats to the underlying LDP-supported
applications. In this paper, we propose a comprehensive mitigation framework
for popular frequency estimation, which contains a suite of novel defenses,
including malicious user detection, attack pattern recognition, and damaged
utility recovery. In addition to existing attacks, we explore new adaptive
adversarial activities for our mitigation design. For detection, we present a
new method to precisely identify bogus reports and thus LDP aggregation can be
performed over the ``clean'' data. When the attack behavior becomes stealthy
and direct filtering out malicious users is difficult, we further propose a
detection that can effectively recognize hidden adversarial patterns, thus
facilitating the decision-making of service providers. These detection methods
require no additional data and attack information and incur minimal
computational cost. Our experiment demonstrates their excellent performance and
substantial improvement over previous work in various settings. In addition, we
conduct an empirical analysis of LDP post-processing for corrupted data
recovery and propose a new post-processing method, through which we reveal new
insights into protocol recommendations in practice and key design principles
for future research.


http://arxiv.org/abs/2506.01307
Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models. (12%)
Youze Wang; Wenbo Hu; Yinpeng Dong; Jing Liu; Hanwang Zhang; Richang Hong
  Large Language Models (LLMs) have evolved into Multimodal Large Language
Models (MLLMs), significantly enhancing their capabilities by integrating
visual information and other types, thus aligning more closely with the nature
of human intelligence, which processes a variety of data forms beyond just
text. Despite advancements, the undesirable generation of these models remains
a critical concern, particularly due to vulnerabilities exposed by text-based
jailbreak attacks, which have represented a significant threat by challenging
existing safety protocols. Motivated by the unique security risks posed by the
integration of new and old modalities for MLLMs, we propose a unified
multimodal universal jailbreak attack framework that leverages iterative
image-text interactions and transfer-based strategy to generate a universal
adversarial suffix and image. Our work not only highlights the interaction of
image-text modalities can be used as a critical vulnerability but also
validates that multimodal universal jailbreak attacks can bring higher-quality
undesirable generations across different MLLMs. We evaluate the undesirable
context generation of MLLMs like LLaVA, Yi-VL, MiniGPT4, MiniGPT-v2, and
InstructBLIP, and reveal significant multimodal safety alignment issues,
highlighting the inadequacy of current safety mechanisms against sophisticated
multimodal attacks. This study underscores the urgent need for robust safety
measures in MLLMs, advocating for a comprehensive review and enhancement of
security protocols to mitigate potential risks associated with multimodal
capabilities.


http://arxiv.org/abs/2506.01767
Predictive-CSM: Lightweight Fragment Security for 6LoWPAN IoT Networks. (1%)
Somayeh Sobati-M
  Fragmentation is a routine part of communication in 6LoWPAN-based IoT
networks,
  designed to accommodate small frame sizes on constrained wireless links.
However, this process
  introduces a critical vulnerability fragments are typically stored and
processed before their
  legitimacy is confirmed, allowing attackers to exploit this gap with minimal
effort.
  In this work, we explore a defense strategy that takes a more adaptive,
behavior-aware approach to this problem. Our system, called Predictive-CSM,
introduces a combination of two
  lightweight mechanisms. The first tracks how each node behaves over time,
rewarding consistent
  and successful interactions while quickly penalizing suspicious or failing
patterns. The second
  checks the integrity of packet fragments using a chained hash, allowing
incomplete or manipulated sequences to be caught early, before they can occupy
memory or waste processing time.
  We put this system to the test using a set of targeted attack simulations,
including early fragment injection, replayed headers, and flooding with fake
data. Across all scenarios, Predictive CSM preserved network delivery and
maintained energy efficiency, even under pressure. Rather
  than relying on heavyweight cryptography or rigid filters, this approach
allows constrained de vices to adapt their defenses in real time based on what
they observe, not just what they're
  told. In that way, it offers a step forward for securing fragmented
communication in real world
  IoT systems


http://arxiv.org/abs/2506.01625
Robust Satisficing Gaussian Process Bandits Under Adversarial Attacks. (1%)
Artun Saday; Yaşar Cahit Yıldırım; Cem Tekin
  We address the problem of Gaussian Process (GP) optimization in the presence
of unknown and potentially varying adversarial perturbations. Unlike
traditional robust optimization approaches that focus on maximizing performance
under worst-case scenarios, we consider a robust satisficing objective, where
the goal is to consistently achieve a predefined performance threshold $\tau$,
even under adversarial conditions. We propose two novel algorithms based on
distinct formulations of robust satisficing, and show that they are instances
of a general robust satisficing framework. Further, each algorithm offers
different guarantees depending on the nature of the adversary. Specifically, we
derive two regret bounds: one that is sublinear over time, assuming certain
conditions on the adversary and the satisficing threshold $\tau$, and another
that scales with the perturbation magnitude but requires no assumptions on the
adversary. Through extensive experiments, we demonstrate that our approach
outperforms the established robust optimization methods in achieving the
satisficing objective, particularly when the ambiguity set of the robust
optimization framework is inaccurately specified.


http://arxiv.org/abs/2506.00978
CAPAA: Classifier-Agnostic Projector-Based Adversarial Attack. (99%)
Zhan Li; Mingyu Zhao; Xin Dong; Haibin Ling; Bingyao Huang
  Projector-based adversarial attack aims to project carefully designed light
patterns (i.e., adversarial projections) onto scenes to deceive deep image
classifiers. It has potential applications in privacy protection and the
development of more robust classifiers. However, existing approaches primarily
focus on individual classifiers and fixed camera poses, often neglecting the
complexities of multi-classifier systems and scenarios with varying camera
poses. This limitation reduces their effectiveness when introducing new
classifiers or camera poses. In this paper, we introduce Classifier-Agnostic
Projector-Based Adversarial Attack (CAPAA) to address these issues. First, we
develop a novel classifier-agnostic adversarial loss and optimization framework
that aggregates adversarial and stealthiness loss gradients from multiple
classifiers. Then, we propose an attention-based gradient weighting mechanism
that concentrates perturbations on regions of high classification activation,
thereby improving the robustness of adversarial projections when applied to
scenes with varying camera poses. Our extensive experimental evaluations
demonstrate that CAPAA achieves both a higher attack success rate and greater
stealthiness compared to existing baselines. Codes are available at:
https://github.com/ZhanLiQxQ/CAPAA.


http://arxiv.org/abs/2506.01064
Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs. (99%)
Yudong Zhang; Ruobing Xie; Yiqing Huang; Jiansheng Chen; Xingwu Sun; Zhanhui Kang; Di Wang; Yu Wang
  Recent advances in large vision-language models (LVLMs) have showcased their
remarkable capabilities across a wide range of multimodal vision-language
tasks. However, these models remain vulnerable to visual adversarial attacks,
which can substantially compromise their performance. Despite their potential
impact, the development of effective methods for purifying such adversarial
examples has received relatively limited attention. In this paper, we introduce
F3, a novel adversarial purification framework that employs a counterintuitive
"fighting fire with fire" strategy: intentionally introducing simple
perturbations to adversarial examples to mitigate their harmful effects.
Specifically, F3 leverages cross-modal attentions derived from randomly
perturbed adversary examples as reference targets. By injecting noise into
these adversarial examples, F3 effectively refines their attention, resulting
in cleaner and more reliable model outputs. Remarkably, this seemingly
paradoxical approach of employing noise to counteract adversarial attacks
yields impressive purification results. Furthermore, F3 offers several distinct
advantages: it is training-free and straightforward to implement, and exhibits
significant computational efficiency improvements compared to existing
purification methods. These attributes render F3 particularly suitable for
large-scale industrial applications where both robust performance and
operational efficiency are critical priorities. The code will be made publicly
available.


http://arxiv.org/abs/2506.00874
Breaking Latent Prior Bias in Detectors for Generalizable AIGC Image Detection. (96%)
Yue Zhou; Xinan He; KaiQing Lin; Bin Fan; Feng Ding; Bin Li
  Current AIGC detectors often achieve near-perfect accuracy on images produced
by the same generator used for training but struggle to generalize to outputs
from unseen generators. We trace this failure in part to latent prior bias:
detectors learn shortcuts tied to patterns stemming from the initial noise
vector rather than learning robust generative artifacts. To address this, we
propose On-Manifold Adversarial Training (OMAT): by optimizing the initial
latent noise of diffusion models under fixed conditioning, we generate
on-manifold adversarial examples that remain on the generator's output
manifold-unlike pixel-space attacks, which introduce off-manifold perturbations
that the generator itself cannot reproduce and that can obscure the true
discriminative artifacts. To test against state-of-the-art generative models,
we introduce GenImage++, a test-only benchmark of outputs from advanced
generators (Flux.1, SD3) with extended prompts and diverse styles. We apply our
adversarial-training paradigm to ResNet50 and CLIP baselines and evaluate
across existing AIGC forensic benchmarks and recent challenge datasets.
Extensive experiments show that adversarially trained detectors significantly
improve cross-generator performance without any network redesign. Our findings
on latent-prior bias offer valuable insights for future dataset construction
and detector evaluation, guiding the development of more robust and
generalizable AIGC forensic methodologies.


http://arxiv.org/abs/2506.01267
Adversarial learning for nonparametric regression: Minimax rate and adaptive estimation. (74%)
Jingfu Peng; Yuhong Yang
  Despite tremendous advancements of machine learning models and algorithms in
various application domains, they are known to be vulnerable to subtle, natural
or intentionally crafted perturbations in future input data, known as
adversarial attacks. While numerous adversarial learning methods have been
proposed, fundamental questions about their statistical optimality in robust
loss remain largely unanswered. In particular, the minimax rate of convergence
and the construction of rate-optimal estimators under future $X$-attacks are
yet to be worked out.
  In this paper, we address this issue in the context of nonparametric
regression, under suitable assumptions on the smoothness of the regression
function and the geometric structure of the input perturbation set. We first
establish the minimax rate of convergence under adversarial $L_q$-risks with $1
\leq q \leq \infty$ and propose a piecewise local polynomial estimator that
achieves the minimax optimality. The established minimax rate elucidates how
the smoothness level and perturbation magnitude affect the fundamental limit of
adversarial learning under future $X$-attacks. Furthermore, we construct a
data-driven adaptive estimator that is shown to achieve, within a logarithmic
factor, the optimal rate across a broad scale of nonparametric and adversarial
classes.


http://arxiv.org/abs/2506.01055
Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Execution. (5%)
Meysam Alizadeh; Zeynab Samei; Daria Stetsenko; Fabrizio Gilardi
  Previous benchmarks on prompt injection in large language models (LLMs) have
primarily focused on generic tasks and attacks, offering limited insights into
more complex threats like data exfiltration. This paper examines how prompt
injection can cause tool-calling agents to leak personal data observed during
task execution. Using a fictitious banking agent, we develop data flow-based
attacks and integrate them into AgentDojo, a recent benchmark for agentic
security. To enhance its scope, we also create a richer synthetic dataset of
human-AI banking conversations. In 16 user tasks from AgentDojo, LLMs show a
15-50 percentage point drop in utility under attack, with average attack
success rates (ASR) around 20 percent; some defenses reduce ASR to zero. Most
LLMs, even when successfully tricked by the attack, avoid leaking highly
sensitive data like passwords, likely due to safety alignments, but they remain
vulnerable to disclosing other personal data. The likelihood of password
leakage increases when a password is requested along with one or two additional
personal details. In an extended evaluation across 48 tasks, the average ASR is
around 15 percent, with no built-in AgentDojo defense fully preventing leakage.
Tasks involving data extraction or authorization workflows, which closely
resemble the structure of exfiltration attacks, exhibit the highest ASRs,
highlighting the interaction between task type, agent performance, and defense
efficacy.


http://arxiv.org/abs/2506.01213
On the Stability of Graph Convolutional Neural Networks: A Probabilistic Perspective. (2%)
Ning Zhang; Henry Kenlay; Li Zhang; Mihai Cucuringu; Xiaowen Dong
  Graph convolutional neural networks (GCNNs) have emerged as powerful tools
for analyzing graph-structured data, achieving remarkable success across
diverse applications. However, the theoretical understanding of the stability
of these models, i.e., their sensitivity to small changes in the graph
structure, remains in rather limited settings, hampering the development and
deployment of robust and trustworthy models in practice. To fill this gap, we
study how perturbations in the graph topology affect GCNN outputs and propose a
novel formulation for analyzing model stability. Unlike prior studies that
focus only on worst-case perturbations, our distribution-aware formulation
characterizes output perturbations across a broad range of input data. This
way, our framework enables, for the first time, a probabilistic perspective on
the interplay between the statistical properties of the node data and
perturbations in the graph topology. We conduct extensive experiments to
validate our theoretical findings and demonstrate their benefits over existing
baselines, in terms of both representation stability and adversarial attacks on
downstream tasks. Our results demonstrate the practical significance of the
proposed formulation and highlight the importance of incorporating data
distribution into stability analysis.


http://arxiv.org/abs/2506.01054
No Soundness in the Real World: On the Challenges of the Verification of Deployed Neural Networks. (2%)
Attila Szász; Balázs Bánhelyi; Márk Jelasity
  The ultimate goal of verification is to guarantee the safety of deployed
neural networks. Here, we claim that all the state-of-the-art verifiers we are
aware of fail to reach this goal. Our key insight is that theoretical soundness
(bounding the full-precision output while computing with floating point) does
not imply practical soundness (bounding the floating point output in a
potentially stochastic environment). We prove this observation for the
approaches that are currently used to achieve provable theoretical soundness,
such as interval analysis and its variants. We also argue that achieving
practical soundness is significantly harder computationally. We support our
claims empirically as well by evaluating several well-known verification
methods. To mislead the verifiers, we create adversarial networks that detect
and exploit features of the deployment environment, such as the order and
precision of floating point operations. We demonstrate that all the tested
verifiers are vulnerable to our new deployment-specific attacks, which proves
that they are not practically sound.


http://arxiv.org/abs/2506.00661
Poster: Adapting Pretrained Vision Transformers with LoRA Against Attack Vectors. (99%)
Richard E. Neddo; Sean Willis; Zander Blasingame; Chen Liu
  Image classifiers, such as those used for autonomous vehicle navigation, are
largely known to be susceptible to adversarial attacks that target the input
image set. There is extensive discussion on adversarial attacks including
perturbations that alter the input images to cause malicious misclassifications
without perceivable modification. This work proposes a countermeasure for such
attacks by adjusting the weights and classes of pretrained vision transformers
with a low-rank adaptation to become more robust against adversarial attacks
and allow for scalable fine-tuning without retraining.


http://arxiv.org/abs/2506.00548
Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities. (98%)
Jiahui Geng; Thy Thy Tran; Preslav Nakov; Iryna Gurevych
  Existing attacks against multimodal language models (MLLMs) primarily
communicate instructions through text accompanied by adversarial images. In
contrast, we exploit the capabilities of MLLMs to interpret non-textual
instructions, specifically, adversarial images or audio generated by our novel
method, Con Instruction. We optimize these adversarial examples to align
closely with target instructions in the embedding space, revealing the
detrimental implications of MLLMs' sophisticated understanding. Unlike prior
work, our method does not require training data or preprocessing of textual
instructions. While these non-textual adversarial examples can effectively
bypass MLLM safety mechanisms, their combination with various text inputs
substantially amplifies attack success. We further introduce a new Attack
Response Categorization (ARC) framework, which evaluates both the quality of
the model's response and its relevance to the malicious instructions.
Experimental results demonstrate that Con Instruction effectively bypasses
safety mechanisms in multiple vision- and audio-language models, including
LLaVA-v1.5, InternVL, Qwen-VL, and Qwen-Audio, evaluated on two standard
benchmarks: AdvBench and SafeBench. Specifically, our method achieves the
highest attack success rates, reaching 81.3% and 86.6% on LLaVA-v1.5 (13B). On
the defense side, we explore various countermeasures against our attacks and
uncover a substantial performance gap among existing techniques. Our
implementation is made publicly available.


http://arxiv.org/abs/2506.00821
SafeGenes: Evaluating the Adversarial Robustness of Genomic Foundation Models. (98%)
Huixin Zhan; Jason H. Moore
  Genomic Foundation Models (GFMs), such as Evolutionary Scale Modeling (ESM),
have demonstrated significant success in variant effect prediction. However,
their adversarial robustness remains largely unexplored. To address this gap,
we propose SafeGenes: a framework for Secure analysis of genomic foundation
models, leveraging adversarial attacks to evaluate robustness against both
engineered near-identical adversarial Genes and embedding-space manipulations.
In this study, we assess the adversarial vulnerabilities of GFMs using two
approaches: the Fast Gradient Sign Method (FGSM) and a soft prompt attack. FGSM
introduces minimal perturbations to input sequences, while the soft prompt
attack optimizes continuous embeddings to manipulate model predictions without
modifying the input tokens. By combining these techniques, SafeGenes provides a
comprehensive assessment of GFM susceptibility to adversarial manipulation.
Targeted soft prompt attacks led to substantial performance degradation, even
in large models such as ESM1b and ESM1v. These findings expose critical
vulnerabilities in current foundation models, opening new research directions
toward improving their security and robustness in high-stakes genomic
applications such as variant effect prediction.


http://arxiv.org/abs/2506.00668
SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues. (11%)
Martin Kuo; Jianyi Zhang; Aolin Ding; Louis DiValentin; Amin Hass; Benjamin F Morris; Isaac Jacobson; Randolph Linderman; James Kiessling; Nicolas Ramos; Bhavna Gopal; Maziyar Baran Pouyan; Changwei Liu; Hai Li; Yiran Chen
  Malicious attackers can exploit large language models (LLMs) by engaging them
in multi-turn dialogues to achieve harmful objectives, posing significant
safety risks to society. To address this challenge, we propose a novel defense
mechanism: SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues
(STREAM). STREAM defends LLMs against multi-turn attacks while preserving their
functional capabilities. Our approach involves constructing a human-annotated
dataset, the Safety Reasoning Multi-turn Dialogues dataset, which is used to
fine-tune a plug-and-play safety reasoning moderator. This model is designed to
identify malicious intent hidden within multi-turn conversations and alert the
target LLM of potential risks. We evaluate STREAM across multiple LLMs against
prevalent multi-turn attack strategies. Experimental results demonstrate that
our method significantly outperforms existing defense techniques, reducing the
Attack Success Rate (ASR) by 51.2%, all while maintaining comparable LLM
capability.


http://arxiv.org/abs/2506.00496
Monitoring Robustness and Individual Fairness. (10%)
Ashutosh Gupta; Thomas A. Henzinger; Konstantin Kueffner; Kaushik Mallik; David Pape
  Input-output robustness appears in various different forms in the literature,
such as robustness of AI models to adversarial or semantic perturbations and
individual fairness of AI models that make decisions about humans.
  We propose runtime monitoring of input-output robustness of deployed,
black-box AI models, where the goal is to design monitors that would observe
one long execution sequence of the model, and would raise an alarm whenever it
is detected that two similar inputs from the past led to dissimilar outputs.
  This way, monitoring will complement existing offline ``robustification''
approaches to increase the trustworthiness of AI decision-makers.
  We show that the monitoring problem can be cast as the fixed-radius nearest
neighbor (FRNN) search problem, which, despite being well-studied, lacks
suitable online solutions.
  We present our tool Clemont, which offers a number of lightweight monitors,
some of which use upgraded online variants of existing FRNN algorithms, and one
uses a novel algorithm based on binary decision diagrams -- a data-structure
commonly used in software and hardware verification.
  We have also developed an efficient parallelization technique that can
substantially cut down the computation time of monitors for which the distance
between input-output pairs is measured using the $L_\infty$ norm.
  Using standard benchmarks from the literature of adversarial and semantic
robustness and individual fairness, we perform a comparative study of different
monitors in \tool, and demonstrate their effectiveness in correctly detecting
robustness violations at runtime.


http://arxiv.org/abs/2506.00676
SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning. (10%)
Saad Hossain; Samanvay Vajpayee; Sirisha Rambhatla
  As large language models (LLMs) become ubiquitous, parameter-efficient
fine-tuning methods and safety-first defenses have proliferated rapidly.
However, the number of approaches and their recent increase have resulted in
diverse evaluations-varied datasets, metrics, and inconsistent threat
settings-making it difficult to fairly compare safety, utility, and robustness
across methods. To address this, we introduce SafeTuneBed, a benchmark and
toolkit unifying fine-tuning and defense evaluation. SafeTuneBed (i) curates a
diverse repository of multiple fine-tuning datasets spanning sentiment
analysis, question-answering, multi-step reasoning, and open-ended instruction
tasks, and allows for the generation of harmful-variant splits; (ii) enables
integration of state-of-the-art defenses, including alignment-stage
immunization, in-training safeguards, and post-tuning repair; and (iii)
provides evaluators for safety (attack success rate, refusal consistency) and
utility. Built on Python-first, dataclass-driven configs and plugins,
SafeTuneBed requires minimal additional code to specify any fine-tuning regime,
defense method, and metric suite, while ensuring end-to-end reproducibility. We
showcase its value by benchmarking representative defenses across varied
poisoning scenarios and tasks. By standardizing data, code, and metrics,
SafeTuneBed is the first focused toolkit of its kind to accelerate rigorous and
comparable research in safe LLM fine-tuning. Code is available at:
https://github.com/criticalml-uw/SafeTuneBed


http://arxiv.org/abs/2506.00789
RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems. (1%)
Yixiao Zeng; Tianyu Cao; Danqing Wang; Xinran Zhao; Zimeng Qiu; Morteza Ziyadi; Tongshuang Wu; Lei Li
  Retrieval-Augmented Generation (RAG) enhances recency and factuality in
answers. However, existing evaluations rarely test how well these systems cope
with real-world noise, conflicting between internal and external retrieved
contexts, or fast-changing facts. We introduce Retrieval-Aware Robustness
Evaluation (RARE), a unified framework and large-scale benchmark that jointly
stress-tests query and document perturbations over dynamic, time-sensitive
corpora. One of the central features of RARE is a knowledge-graph-driven
synthesis pipeline (RARE-Get) that automatically extracts single and multi-hop
relations from the customized corpus and generates multi-level question sets
without manual intervention. Leveraging this pipeline, we construct a dataset
(RARE-Set) spanning 400 expert-level time-sensitive finance, economics, and
policy documents and 48,322 questions whose distribution evolves as the
underlying sources change. To quantify resilience, we formalize
retrieval-conditioned robustness metrics (RARE-Met) that capture a model's
ability to remain correct or recover when queries, documents, or real-world
retrieval results are systematically altered. Our results show that RAG systems
exhibit surprising vulnerability to perturbations, with document robustness
consistently being the weakest point regardless of generator size or
architecture. RAG systems consistently show lower robustness on multi-hop
queries than single-hop queries across all domains.


http://arxiv.org/abs/2505.24703
PatchDEMUX: A Certifiably Robust Framework for Multi-label Classifiers Against Adversarial Patches. (92%)
Dennis Jacob; Chong Xiang; Prateek Mittal
  Deep learning techniques have enabled vast improvements in computer vision
technologies. Nevertheless, these models are vulnerable to adversarial patch
attacks which catastrophically impair performance. The physically realizable
nature of these attacks calls for certifiable defenses, which feature provable
guarantees on robustness. While certifiable defenses have been successfully
applied to single-label classification, limited work has been done for
multi-label classification. In this work, we present PatchDEMUX, a certifiably
robust framework for multi-label classifiers against adversarial patches. Our
approach is a generalizable method which can extend any existing certifiable
defense for single-label classification; this is done by considering the
multi-label classification task as a series of isolated binary classification
problems to provably guarantee robustness. Furthermore, in the scenario where
an attacker is limited to a single patch we propose an additional certification
procedure that can provide tighter robustness bounds. Using the current
state-of-the-art (SOTA) single-label certifiable defense PatchCleanser as a
backbone, we find that PatchDEMUX can achieve non-trivial robustness on the
MS-COCO and PASCAL VOC datasets while maintaining high clean performance


http://arxiv.org/abs/2506.00280
3D Gaussian Splat Vulnerabilities. (88%)
Matthew Hull; Haoyang Yang; Pratham Mehta; Mansi Phute; Aeree Cho; Haoran Wang; Matthew Lau; Wenke Lee; Willian T. Lunardi; Martin Andreoni; Polo Chau
  With 3D Gaussian Splatting (3DGS) being increasingly used in safety-critical
applications, how can an adversary manipulate the scene to cause harm? We
introduce CLOAK, the first attack that leverages view-dependent Gaussian
appearances - colors and textures that change with viewing angle - to embed
adversarial content visible only from specific viewpoints. We further
demonstrate DAGGER, a targeted adversarial attack directly perturbing 3D
Gaussians without access to underlying training data, deceiving multi-stage
object detectors e.g., Faster R-CNN, through established methods such as
projected gradient descent. These attacks highlight underexplored
vulnerabilities in 3DGS, introducing a new potential threat to robotic learning
for autonomous navigation and other safety-critical 3DGS applications.


http://arxiv.org/abs/2506.00373
Adversarial Machine Learning for Robust Password Strength Estimation. (87%)
Pappu Jha; Hanzla Hamid; Oluseyi Olukola; Ashim Dahal; Nick Rahimi
  Passwords remain one of the most common methods for securing sensitive data
in the digital age. However, weak password choices continue to pose significant
risks to data security and privacy. This study aims to solve the problem by
focusing on developing robust password strength estimation models using
adversarial machine learning, a technique that trains models on intentionally
crafted deceptive passwords to expose and address vulnerabilities posed by such
passwords. We apply five classification algorithms and use a dataset with more
than 670,000 samples of adversarial passwords to train the models. Results
demonstrate that adversarial training improves password strength classification
accuracy by up to 20% compared to traditional machine learning models. It
highlights the importance of integrating adversarial machine learning into
security systems to enhance their robustness against modern adaptive threats.
  Keywords: adversarial attack, password strength, classification, machine
learning


http://arxiv.org/abs/2505.24842
Cascading Adversarial Bias from Injection to Distillation in Language Models. (86%)
Harsh Chaudhari; Jamie Hayes; Matthew Jagielski; Ilia Shumailov; Milad Nasr; Alina Oprea
  Model distillation has become essential for creating smaller, deployable
language models that retain larger system capabilities. However, widespread
deployment raises concerns about resilience to adversarial manipulation. This
paper investigates vulnerability of distilled models to adversarial injection
of biased content during training. We demonstrate that adversaries can inject
subtle biases into teacher models through minimal data poisoning, which
propagates to student models and becomes significantly amplified. We propose
two propagation modes: Untargeted Propagation, where bias affects multiple
tasks, and Targeted Propagation, focusing on specific tasks while maintaining
normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning
rate), student models generate biased responses 76.9% of the time in targeted
scenarios - higher than 69.4% in teacher models. For untargeted propagation,
adversarial bias appears 6x-29x more frequently in student models on unseen
tasks. We validate findings across six bias types (targeted advertisements,
phishing links, narrative manipulations, insecure coding practices), various
distillation methods, and different modalities spanning text and code
generation. Our evaluation reveals shortcomings in current defenses -
perplexity filtering, bias detection systems, and LLM-based autorater
frameworks - against these attacks. Results expose significant security
vulnerabilities in distilled models, highlighting need for specialized
safeguards. We propose practical design principles for building effective
adversarial bias mitigation strategies.


http://arxiv.org/abs/2506.00325
Towards Effective and Efficient Adversarial Defense with Diffusion Models for Robust Visual Tracking. (83%)
Long Xu; Peng Gao; Wen-Jia Tang; Fei Wang; Ru-Yue Yuan
  Although deep learning-based visual tracking methods have made significant
progress, they exhibit vulnerabilities when facing carefully designed
adversarial attacks, which can lead to a sharp decline in tracking performance.
To address this issue, this paper proposes for the first time a novel
adversarial defense method based on denoise diffusion probabilistic models,
termed DiffDf, aimed at effectively improving the robustness of existing visual
tracking methods against adversarial attacks. DiffDf establishes a multi-scale
defense mechanism by combining pixel-level reconstruction loss, semantic
consistency loss, and structural similarity loss, effectively suppressing
adversarial perturbations through a gradual denoising process. Extensive
experimental results on several mainstream datasets show that the DiffDf method
demonstrates excellent generalization performance for trackers with different
architectures, significantly improving various evaluation metrics while
achieving real-time inference speeds of over 30 FPS, showcasing outstanding
defense performance and efficiency. Codes are available at
https://github.com/pgao-lab/DiffDf.


http://arxiv.org/abs/2505.24445
Learning Safety Constraints for Large Language Models. (81%)
Xin Chen; Yarden As; Andreas Krause
  Large language models (LLMs) have emerged as powerful tools but pose
significant safety risks through harmful outputs and vulnerability to
adversarial attacks. We propose SaP, short for Safety Polytope, a geometric
approach to LLM safety that learns and enforces multiple safety constraints
directly in the model's representation space. We develop a framework that
identifies safe and unsafe regions via the polytope's facets, enabling both
detection and correction of unsafe outputs through geometric steering. Unlike
existing approaches that modify model weights, SaP operates post-hoc in the
representation space, preserving model capabilities while enforcing safety
constraints. Experiments across multiple LLMs demonstrate that our method can
effectively detect unethical inputs, reduce adversarial attack success rates
while maintaining performance on standard tasks, thus highlighting the
importance of having an explicit geometric model for safety. Analysis of the
learned polytope facets reveals emergence of specialization in detecting
different semantic notions of safety, providing interpretable insights into how
safety is captured in LLMs' representation space.


http://arxiv.org/abs/2505.24654
Black-box Adversarial Attacks on CNN-based SLAM Algorithms. (75%)
Maria Rafaela Gkeka; Bowen Sun; Evgenia Smirni; Christos D. Antonopoulos; Spyros Lalis; Nikolaos Bellas
  Continuous advancements in deep learning have led to significant progress in
feature detection, resulting in enhanced accuracy in tasks like Simultaneous
Localization and Mapping (SLAM). Nevertheless, the vulnerability of deep neural
networks to adversarial attacks remains a challenge for their reliable
deployment in applications, such as navigation of autonomous agents. Even
though CNN-based SLAM algorithms are a growing area of research there is a
notable absence of a comprehensive presentation and examination of adversarial
attacks targeting CNN-based feature detectors, as part of a SLAM system. Our
work introduces black-box adversarial perturbations applied to the RGB images
fed into the GCN-SLAM algorithm. Our findings on the TUM dataset [30] reveal
that even attacks of moderate scale can lead to tracking failure in as many as
76% of the frames. Moreover, our experiments highlight the catastrophic impact
of attacking depth instead of RGB input images on the SLAM system.


http://arxiv.org/abs/2506.00191
Heterogeneous Graph Backdoor Attack. (69%)
Jiawei Chen; Lusi Li; Daniel Takabi; Masha Sosonkina; Rui Ning
  Heterogeneous Graph Neural Networks (HGNNs) excel in modeling complex,
multi-typed relationships across diverse domains, yet their vulnerability to
backdoor attacks remains unexplored. To address this gap, we conduct the first
investigation into the susceptibility of HGNNs to existing graph backdoor
attacks, revealing three critical issues: (1) high attack budget required for
effective backdoor injection, (2) inefficient and unreliable backdoor
activation, and (3) inaccurate attack effectiveness evaluation. To tackle these
issues, we propose the Heterogeneous Graph Backdoor Attack (HGBA), the first
backdoor attack specifically designed for HGNNs, introducing a novel
relation-based trigger mechanism that establishes specific connections between
a strategically selected trigger node and poisoned nodes via the backdoor
metapath. HGBA achieves efficient and stealthy backdoor injection with minimal
structural modifications and supports easy backdoor activation through two
flexible strategies: Self-Node Attack and Indiscriminate Attack. Additionally,
we improve the ASR measurement protocol, enabling a more accurate assessment of
attack effectiveness. Extensive experiments demonstrate that HGBA far surpasses
multiple state-of-the-art graph backdoor attacks in black-box settings,
efficiently attacking HGNNs with low attack budgets. Ablation studies show that
the strength of HBGA benefits from our trigger node selection method and
backdoor metapath selection strategy. In addition, HGBA shows superior
robustness against node feature perturbations and multiple types of existing
graph backdoor defense mechanisms. Finally, extension experiments demonstrate
that the relation-based trigger mechanism can effectively extend to tasks in
homogeneous graph scenarios, thereby posing severe threats to broader
security-critical domains.


http://arxiv.org/abs/2506.15711
Shadow defense against gradient inversion attack in federated learning. (68%)
Le Jiang; Liyan Ma; Guang Yang
Federated learning (FL) has emerged as a transformative framework for privacy-preserving distributed training, allowing clients to collaboratively train a global model without sharing their local data. This is especially crucial in sensitive fields like healthcare, where protecting patient data is paramount. However, privacy leakage remains a critical challenge, as the communication of model updates can be exploited by potential adversaries. Gradient inversion attacks (GIAs), for instance, allow adversaries to approximate the gradients used for training and reconstruct training images, thus stealing patient privacy. Existing defense mechanisms obscure gradients, yet lack a nuanced understanding of which gradients or types of image information are most vulnerable to such attacks. These indiscriminate calibrated perturbations result in either excessive privacy protection degrading model accuracy, or insufficient one failing to safeguard sensitive information. Therefore, we introduce a framework that addresses these challenges by leveraging a shadow model with interpretability for identifying sensitive areas. This enables a more targeted and sample-specific noise injection. Specially, our defensive strategy achieves discrepancies of 3.73 in PSNR and 0.2 in SSIM compared to the circumstance without defense on the ChestXRay dataset, and 2.78 in PSNR and 0.166 in the EyePACS dataset. Moreover, it minimizes adverse effects on model performance, with less than 1\% F1 reduction compared to SOTA methods. Our extensive experiments, conducted across diverse types of medical images, validate the generalization of the proposed framework. The stable defense improvements for FedAvg are consistently over 1.5\% times in LPIPS and SSIM. It also offers a universal defense against various GIA types, especially for these sensitive areas in images.

http://arxiv.org/abs/2506.00281
Adversarial Threat Vectors and Risk Mitigation for Retrieval-Augmented Generation Systems. (68%)
Chris M. Ward; Josh Harguess
  Retrieval-Augmented Generation (RAG) systems, which integrate Large Language
Models (LLMs) with external knowledge sources, are vulnerable to a range of
adversarial attack vectors. This paper examines the importance of RAG systems
through recent industry adoption trends and identifies the prominent attack
vectors for RAG: prompt injection, data poisoning, and adversarial query
manipulation. We analyze these threats under risk management lens, and propose
robust prioritized control list that includes risk-mitigating actions like
input validation, adversarial training, and real-time monitoring.


http://arxiv.org/abs/2505.24523
Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors. (56%)
Andrea Pedrotti; Michele Papucci; Cristiano Ciaccio; Alessio Miaschi; Giovanni Puccetti; Felice Dell'Orletta; Andrea Esuli
  Recent advancements in Generative AI and Large Language Models (LLMs) have
enabled the creation of highly realistic synthetic content, raising concerns
about the potential for malicious use, such as misinformation and manipulation.
Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the
lack of robust benchmarks that assess generalization to real-world scenarios.
In this work, we present a pipeline to test the resilience of state-of-the-art
MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed
adversarial attacks. To challenge the detectors, we fine-tune language models
using Direct Preference Optimization (DPO) to shift the MGT style toward
human-written text (HWT). This exploits the detectors' reliance on stylistic
clues, making new generations more challenging to detect. Additionally, we
analyze the linguistic shifts induced by the alignment and which features are
used by detectors to detect MGT texts. Our results show that detectors can be
easily fooled with relatively few examples, resulting in a significant drop in
detection performance. This highlights the importance of improving detection
methods and making them robust to unseen in-domain texts.


http://arxiv.org/abs/2506.02032
Towards Secure MLOps: Surveying Attacks, Mitigation Strategies, and Research Challenges. (41%)
Raj Patel; Himanshu Tripathi; Jasper Stone; Noorbakhsh Amiri Golilarz; Sudip Mittal; Shahram Rahimi; Vini Chaudhary
  The rapid adoption of machine learning (ML) technologies has driven
organizations across diverse sectors to seek efficient and reliable methods to
accelerate model development-to-deployment. Machine Learning Operations (MLOps)
has emerged as an integrative approach addressing these requirements by
unifying relevant roles and streamlining ML workflows. As the MLOps market
continues to grow, securing these pipelines has become increasingly critical.
However, the unified nature of MLOps ecosystem introduces vulnerabilities,
making them susceptible to adversarial attacks where a single misconfiguration
can lead to compromised credentials, severe financial losses, damaged public
trust, and the poisoning of training data. Our paper presents a systematic
application of the MITRE ATLAS (Adversarial Threat Landscape for
Artificial-Intelligence Systems) framework, a comprehensive and continuously
updated catalog of AI-focused attacks, to systematically assess attacks across
different phases of the MLOps ecosystem. We begin by examining the preparatory
phases during which adversaries acquire the essential intelligence required to
initiate their attacks. We then present a structured taxonomy of attack
techniques explicitly mapped to corresponding phases of the MLOps ecosystem,
supported by examples drawn from red-teaming exercises and real-world
incidents. This is followed by a taxonomy of mitigation strategies aligned with
these attack categories, offering actionable early-stage defenses to strengthen
the security of MLOps ecosystem. Given the rapid evolution and adoption of
MLOps, we further highlight key research gaps that require immediate attention.
Our work emphasizes the importance of implementing robust security protocols
from the outset, empowering practitioners to safeguard MLOps ecosystem against
evolving cyber attacks.


http://arxiv.org/abs/2505.24369
Adversarial Preference Learning for Robust LLM Alignment. (38%)
Yuanfu Wang; Pengyu Wang; Chenyang Xi; Bo Tang; Junyi Zhu; Wenqiang Wei; Chen Chen; Chao Yang; Jingfeng Zhang; Chaochao Lu; Yijun Niu; Keming Mao; Zhiyu Li; Feiyu Xiong; Jie Hu; Mingchuan Yang
  Modern language models often rely on Reinforcement Learning from Human
Feedback (RLHF) to encourage safe behaviors. However, they remain vulnerable to
adversarial attacks due to three key limitations: (1) the inefficiency and high
cost of human annotation, (2) the vast diversity of potential adversarial
attacks, and (3) the risk of feedback bias and reward hacking. To address these
challenges, we introduce Adversarial Preference Learning (APL), an iterative
adversarial training method incorporating three key innovations. First, a
direct harmfulness metric based on the model's intrinsic preference
probabilities, eliminating reliance on external assessment. Second, a
conditional generative attacker that synthesizes input-specific adversarial
variations. Third, an iterative framework with automated closed-loop feedback,
enabling continuous adaptation through vulnerability discovery and mitigation.
Experiments on Mistral-7B-Instruct-v0.3 demonstrate that APL significantly
enhances robustness, achieving 83.33% harmlessness win rate over the base model
(evaluated by GPT-4o), reducing harmful outputs from 5.88% to 0.43% (measured
by LLaMA-Guard), and lowering attack success rate by up to 65% according to
HarmBench. Notably, APL maintains competitive utility, with an MT-Bench score
of 6.59 (comparable to the baseline 6.78) and an LC-WinRate of 46.52% against
the base model.


http://arxiv.org/abs/2505.24227
Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models. (13%)
Ying Yang; Jie Zhang; Xiao Lv; Di Lin; Tao Xiang; Qing Guo
  While adversarial attacks on vision-and-language pretraining (VLP) models
have been explored, generating natural adversarial samples crafted through
realistic and semantically meaningful perturbations remains an open challenge.
Existing methods, primarily designed for classification tasks, struggle when
adapted to VLP models due to their restricted optimization spaces, leading to
ineffective attacks or unnatural artifacts. To address this, we propose
\textbf{LightD}, a novel framework that generates natural adversarial samples
for VLP models via semantically guided relighting. Specifically, LightD
leverages ChatGPT to propose context-aware initial lighting parameters and
integrates a pretrained relighting model (IC-light) to enable diverse lighting
adjustments. LightD expands the optimization space while ensuring perturbations
align with scene semantics. Additionally, gradient-based optimization is
applied to the reference lighting image to further enhance attack effectiveness
while maintaining visual naturalness. The effectiveness and superiority of the
proposed LightD have been demonstrated across various VLP models in tasks such
as image captioning and visual question answering.


http://arxiv.org/abs/2505.24592
A Flat Minima Perspective on Understanding Augmentations and Model Robustness. (13%)
Weebum Yoo; Sung Whan Yoon
  Model robustness indicates a model's capability to generalize well on
unforeseen distributional shifts, including data corruption, adversarial
attacks, and domain shifts. Data augmentation is one of the prevalent and
effective ways to enhance robustness. Despite the great success of
augmentations in different fields, a general theoretical understanding of their
efficacy in improving model robustness is lacking. We offer a unified
theoretical framework to clarify how augmentations can enhance model robustness
through the lens of loss surface flatness and PAC generalization bound. Our
work diverges from prior studies in that our analysis i) broadly encompasses
much of the existing augmentation methods, and ii) is not limited to specific
types of distribution shifts like adversarial attacks. We confirm our theories
through simulations on the existing common corruption and adversarial
robustness benchmarks based on the CIFAR and ImageNet datasets, as well as
domain generalization benchmarks including PACS and OfficeHome.


http://arxiv.org/abs/2506.00359
Keeping an Eye on LLM Unlearning: The Hidden Risk and Remedy. (3%)
Jie Ren; Zhenwei Dai; Xianfeng Tang; Yue Xing; Shenglai Zeng; Hui Liu; Jingying Zeng; Qiankun Peng; Samarth Varshney; Suhang Wang; Qi He; Charu C. Aggarwal; Hui Liu
  Although Large Language Models (LLMs) have demonstrated impressive
capabilities across a wide range of tasks, growing concerns have emerged over
the misuse of sensitive, copyrighted, or harmful data during training. To
address these concerns, unlearning techniques have been developed to remove the
influence of specific data without retraining from scratch. However, this paper
reveals a critical vulnerability in fine-tuning-based unlearning: a malicious
user can craft a manipulated forgetting request that stealthily degrades the
model's utility for benign users. We demonstrate this risk through a
red-teaming Stealthy Attack (SA), which is inspired by two key limitations of
existing unlearning (the inability to constrain the scope of unlearning effect
and the failure to distinguish benign tokens from unlearning signals). Prior
work has shown that unlearned models tend to memorize forgetting data as
unlearning signals, and respond with hallucinations or feigned ignorance when
unlearning signals appear in the input. By subtly increasing the presence of
common benign tokens in the forgetting data, SA enhances the connection between
benign tokens and unlearning signals. As a result, when normal users include
such tokens in their prompts, the model exhibits unlearning behaviors, leading
to unintended utility degradation. To address this vulnerability, we propose
Scope-aware Unlearning (SU), a lightweight enhancement that introduces a scope
term into the unlearning objective, encouraging the model to localize the
forgetting effect. Our method requires no additional data processing,
integrates seamlessly with existing fine-tuning frameworks, and significantly
improves robustness against SA. Extensive experiments validate the
effectiveness of both SA and SU.


http://arxiv.org/abs/2505.24379
Breaking the Gold Standard: Extracting Forgotten Data under Exact Unlearning in Large Language Models. (3%)
Xiaoyu Wu; Yifei Pang; Terrance Liu; Zhiwei Steven Wu
  Large language models are typically trained on datasets collected from the
web, which may inadvertently contain harmful or sensitive personal information.
To address growing privacy concerns, unlearning methods have been proposed to
remove the influence of specific data from trained models. Of these, exact
unlearning -- which retrains the model from scratch without the target data --
is widely regarded the gold standard, believed to be robust against
privacy-related attacks. In this paper, we challenge this assumption by
introducing a novel data extraction attack that compromises even exact
unlearning. Our method leverages both the pre- and post-unlearning models: by
guiding the post-unlearning model using signals from the pre-unlearning model,
we uncover patterns that reflect the removed data distribution. Combining model
guidance with a token filtering strategy, our attack significantly improves
extraction success rates -- doubling performance in some cases -- across common
benchmarks such as MUSE, TOFU, and WMDP. Furthermore, we demonstrate our
attack's effectiveness on a simulated medical diagnosis dataset to highlight
real-world privacy risks associated with exact unlearning. In light of our
findings, which suggest that unlearning may, in a contradictory way, increase
the risk of privacy leakage, we advocate for evaluation of unlearning methods
to consider broader threat models that account not only for post-unlearning
models but also for adversarial access to prior checkpoints.


http://arxiv.org/abs/2505.24428
Model Unlearning via Sparse Autoencoder Subspace Guided Projections. (2%)
Xu Wang; Zihao Li; Benyou Wang; Yan Hu; Difan Zou
  Large language models (LLMs) store vast amounts of information, making them
powerful yet raising privacy and safety concerns when selective knowledge
removal is required. Existing unlearning strategies, ranging from
gradient-based fine-tuning and model editing to sparse autoencoder (SAE)
steering, either lack interpretability or fail to provide a robust defense
against adversarial prompts. We propose SAE-Guided Subspace Projection
Unlearning (SSPU), a novel framework that leverages SAE features to drive
targeted updates in the model's parameter space, enabling precise,
interpretable, and robust unlearning. SSPU's three-stage pipeline performs
data-driven layer and feature selection, subspace construction via QR
decomposition, and constrained optimization that controls activations into an
"irrelevant" subspace while preserving retained knowledge. Overall, we use SAE
features to construct a subspace that supervises unlearning, refining the loss
and adding a regularization term to guide interpretable parameter updates. In
experiments on the WMDP-Cyber forget set and three utility benchmarks (MMLU,
TruthfulQA, GSM8K), SSPU reduces harmful knowledge accuracy by 3.22% compared
to the strongest baseline. It also improves adversarial robustness, lowering
malicious accuracy under jailbreak prompts compared to baselines. Our findings
expose the limitations of prior unlearning methods and demonstrate how
interpretable subspace-guided optimization can achieve robust, controllable
model behavior.


http://arxiv.org/abs/2505.24685
So, I climbed to the top of the pyramid of pain -- now what? (1%)
Vasilis Katos; Emily Rosenorn-Lanng; Jane Henriksen-Bulmer; Ala Yankouskaya
  This paper explores the evolving dynamics of cybersecurity in the age of
advanced AI, from the perspective of the introduced Human Layer Kill Chain
framework. As traditional attack models like Lockheed Martin's Cyber Kill Chain
become inadequate in addressing human vulnerabilities exploited by modern
adversaries, the Humal Layer Kill Chain offers a nuanced approach that
integrates human psychology and behaviour into the analysis of cyber threats.
We detail the eight stages of the Human Layer Kill Chain, illustrating how
AI-enabled techniques can enhance psychological manipulation in attacks. By
merging the Human Layer with the Cyber Kill Chain, we propose a Sociotechnical
Kill Plane that allows for a holistic examination of attackers' tactics,
techniques, and procedures (TTPs) across the sociotechnical landscape. This
framework not only aids cybersecurity professionals in understanding
adversarial methods, but also empowers non-technical personnel to engage in
threat identification and response. The implications for incident response and
organizational resilience are significant, particularly as AI continues to
shape the threat landscape.


http://arxiv.org/abs/2505.24728
Robust Federated Learning against Model Perturbation in Edge Networks. (1%)
Dongzi Jin; Yong Xiao; Yingyu Li
  Federated Learning (FL) is a promising paradigm for realizing edge
intelligence, allowing collaborative learning among distributed edge devices by
sharing models instead of raw data. However, the shared models are often
assumed to be ideal, which would be inevitably violated in practice due to
various perturbations, leading to significant performance degradation. To
overcome this challenge, we propose a novel method, termed Sharpness-Aware
Minimization-based Robust Federated Learning (SMRFL), which aims to improve
model robustness against perturbations by exploring the geometrical property of
the model landscape. Specifically, SMRFL solves a min-max optimization problem
that promotes model convergence towards a flat minimum by minimizing the
maximum loss within a neighborhood of the model parameters. In this way, model
sensitivity to perturbations is reduced, and robustness is enhanced since
models in the neighborhood of the flat minimum also enjoy low loss values. The
theoretical result proves that SMRFL can converge at the same rate as FL
without perturbations. Extensive experimental results show that SMRFL
significantly enhances robustness against perturbations compared to three
baseline methods on two real-world datasets under three perturbation scenarios.


http://arxiv.org/abs/2505.23313
Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition. (98%)
Weizhe Kong; Xiao Wang; Ruichong Gao; Chenglong Li; Yu Zhang; Xing Yang; Yaowei Wang; Jin Tang
  Pedestrian Attribute Recognition (PAR) is an indispensable task in
human-centered research and has made great progress in recent years with the
development of deep neural networks. However, the potential vulnerability and
anti-interference ability have still not been fully explored. To bridge this
gap, this paper proposes the first adversarial attack and defense framework for
pedestrian attribute recognition. Specifically, we exploit both global- and
patch-level attacks on the pedestrian images, based on the pre-trained
CLIP-based PAR framework. It first divides the input pedestrian image into
non-overlapping patches and embeds them into feature embeddings using a
projection layer. Meanwhile, the attribute set is expanded into sentences using
prompts and embedded into attribute features using a pre-trained CLIP text
encoder. A multi-modal Transformer is adopted to fuse the obtained vision and
text tokens, and a feed-forward network is utilized for attribute recognition.
Based on the aforementioned PAR framework, we adopt the adversarial semantic
and label-perturbation to generate the adversarial noise, termed ASL-PAR. We
also design a semantic offset defense strategy to suppress the influence of
adversarial attacks. Extensive experiments conducted on both digital domains
(i.e., PETA, PA100K, MSP60K, RAPv2) and physical domains fully validated the
effectiveness of our proposed adversarial attack and defense strategies for the
pedestrian attribute recognition. The source code of this paper will be
released on https://github.com/Event-AHU/OpenPAR.


http://arxiv.org/abs/2505.24141
The Butterfly Effect in Pathology: Exploring Security in Pathology Foundation Models. (98%)
Jiashuai Liu; Yingjia Shang; Yingkang Zhan; Di Zhang; Yi Niu; Dong Wei; Xian Wu; Zeyu Gao; Chen Li; Yefeng Zheng
  With the widespread adoption of pathology foundation models in both research
and clinical decision support systems, exploring their security has become a
critical concern. However, despite their growing impact, the vulnerability of
these models to adversarial attacks remains largely unexplored. In this work,
we present the first systematic investigation into the security of pathology
foundation models for whole slide image~(WSI) analysis against adversarial
attacks. Specifically, we introduce the principle of \textit{local perturbation
with global impact} and propose a label-free attack framework that operates
without requiring access to downstream task labels. Under this attack
framework, we revise four classical white-box attack methods and redefine the
perturbation budget based on the characteristics of WSI. We conduct
comprehensive experiments on three representative pathology foundation models
across five datasets and six downstream tasks. Despite modifying only 0.1\% of
patches per slide with imperceptible noise, our attack leads to downstream
accuracy degradation that can reach up to 20\% in the worst cases. Furthermore,
we analyze key factors that influence attack success, explore the relationship
between patch-level vulnerability and semantic content, and conduct a
preliminary investigation into potential defence strategies. These findings lay
the groundwork for future research on the adversarial robustness and reliable
deployment of pathology foundation models. Our code is publicly available at:
https://github.com/Jiashuai-Liu-hmos/Attack-WSI-pathology-foundation-models.


http://arxiv.org/abs/2505.23518
TRAP: Targeted Redirecting of Agentic Preferences. (92%)
Hangoo Kang; Jehyeok Yeon; Gagandeep Singh
  Autonomous agentic AI systems powered by vision-language models (VLMs) are
rapidly advancing toward real-world deployment, yet their cross-modal reasoning
capabilities introduce new attack surfaces for adversarial manipulation that
exploit semantic reasoning across modalities. Existing adversarial attacks
typically rely on visible pixel perturbations or require privileged model or
environment access, making them impractical for stealthy, real-world
exploitation. We introduce TRAP, a generative adversarial framework that
manipulates the agent's decision-making using diffusion-based semantic
injections. Our method combines negative prompt-based degradation with positive
semantic optimization, guided by a Siamese semantic network and layout-aware
spatial masking. Without requiring access to model internals, TRAP produces
visually natural images yet induces consistent selection biases in agentic AI
systems. We evaluate TRAP on the Microsoft Common Objects in Context (COCO)
dataset, building multi-candidate decision scenarios. Across these scenarios,
TRAP achieves a 100% attack success rate on leading models, including
LLaVA-34B, Gemma3, and Mistral-3.1, significantly outperforming baselines such
as SPSA, Bandit, and standard diffusion approaches. These results expose a
critical vulnerability: Autonomous agents can be consistently misled through
human-imperceptible cross-modal manipulations. These findings highlight the
need for defense strategies beyond pixel-level robustness to address semantic
vulnerabilities in cross-modal decision-making.


http://arxiv.org/abs/2505.23412
Buffer-free Class-Incremental Learning with Out-of-Distribution Detection. (81%)
Srishti Gupta; Daniele Angioni; Maura Pintor; Ambra Demontis; Lea Schönherr; Battista Biggio; Fabio Roli
  Class-incremental learning (CIL) poses significant challenges in open-world
scenarios, where models must not only learn new classes over time without
forgetting previous ones but also handle inputs from unknown classes that a
closed-set model would misclassify. Recent works address both issues by
(i)~training multi-head models using the task-incremental learning framework,
and (ii) predicting the task identity employing out-of-distribution (OOD)
detectors. While effective, the latter mainly relies on joint training with a
memory buffer of past data, raising concerns around privacy, scalability, and
increased training time. In this paper, we present an in-depth analysis of
post-hoc OOD detection methods and investigate their potential to eliminate the
need for a memory buffer. We uncover that these methods, when applied
appropriately at inference time, can serve as a strong substitute for
buffer-based OOD detection. We show that this buffer-free approach achieves
comparable or superior performance to buffer-based methods both in terms of
class-incremental learning and the rejection of unknown samples. Experimental
results on CIFAR-10, CIFAR-100 and Tiny ImageNet datasets support our findings,
offering new insights into the design of efficient and privacy-preserving CIL
systems for open-world settings.


http://arxiv.org/abs/2505.23559
SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents. (73%)
Kunlun Zhu; Jiaxun Zhang; Ziheng Qi; Nuoxing Shang; Zijia Liu; Peixuan Han; Yue Su; Haofei Yu; Jiaxuan You
  Recent advancements in large language model (LLM) agents have significantly
accelerated scientific discovery automation, yet concurrently raised critical
ethical and safety concerns. To systematically address these challenges, we
introduce \textbf{SafeScientist}, an innovative AI scientist framework
explicitly designed to enhance safety and ethical responsibility in AI-driven
scientific exploration. SafeScientist proactively refuses ethically
inappropriate or high-risk tasks and rigorously emphasizes safety throughout
the research process. To achieve comprehensive safety oversight, we integrate
multiple defensive mechanisms, including prompt monitoring, agent-collaboration
monitoring, tool-use monitoring, and an ethical reviewer component.
Complementing SafeScientist, we propose \textbf{SciSafetyBench}, a novel
benchmark specifically designed to evaluate AI safety in scientific contexts,
comprising 240 high-risk scientific tasks across 6 domains, alongside 30
specially designed scientific tools and 120 tool-related risk tasks. Extensive
experiments demonstrate that SafeScientist significantly improves safety
performance by 35\% compared to traditional AI scientist frameworks, without
compromising scientific output quality. Additionally, we rigorously validate
the robustness of our safety pipeline against diverse adversarial attack
methods, further confirming the effectiveness of our integrated approach. The
code and data will be available at https://github.com/ulab-uiuc/SafeScientist.
\textcolor{red}{Warning: this paper contains example data that may be offensive
or harmful.}


http://arxiv.org/abs/2505.23266
Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion. (41%)
Chunlong Xie; Jialing He; Shangwei Guo; Jiacheng Wang; Shudong Zhang; Tianwei Zhang; Tao Xiang
  We present Adversarial Object Fusion (AdvOF), a novel attack framework
targeting vision-and-language navigation (VLN) agents in service-oriented
environments by generating adversarial 3D objects. While foundational models
like Large Language Models (LLMs) and Vision Language Models (VLMs) have
enhanced service-oriented navigation systems through improved perception and
decision-making, their integration introduces vulnerabilities in
mission-critical service workflows. Existing adversarial attacks fail to
address service computing contexts, where reliability and quality-of-service
(QoS) are paramount. We utilize AdvOF to investigate and explore the impact of
adversarial environments on the VLM-based perception module of VLN agents. In
particular, AdvOF first precisely aggregates and aligns the victim object
positions in both 2D and 3D space, defining and rendering adversarial objects.
Then, we collaboratively optimize the adversarial object with regularization
between the adversarial and victim object across physical properties and VLM
perceptions. Through assigning importance weights to varying views, the
optimization is processed stably and multi-viewedly by iterative fusions from
local updates and justifications. Our extensive evaluations demonstrate AdvOF
can effectively degrade agent performance under adversarial conditions while
maintaining minimal interference with normal navigation tasks. This work
advances the understanding of service security in VLM-powered navigation
systems, providing computational foundations for robust service composition in
physical-world deployments.


http://arxiv.org/abs/2505.24019
LLM Agents Should Employ Security Principles. (3%)
Kaiyuan Zhang; Zian Su; Pin-Yu Chen; Elisa Bertino; Xiangyu Zhang; Ninghui Li
  Large Language Model (LLM) agents show considerable promise for automating
complex tasks using contextual reasoning; however, interactions involving
multiple agents and the system's susceptibility to prompt injection and other
forms of context manipulation introduce new vulnerabilities related to privacy
leakage and system exploitation. This position paper argues that the
well-established design principles in information security, which are commonly
referred to as security principles, should be employed when deploying LLM
agents at scale. Design principles such as defense-in-depth, least privilege,
complete mediation, and psychological acceptability have helped guide the
design of mechanisms for securing information systems over the last five
decades, and we argue that their explicit and conscientious adoption will help
secure agentic systems. To illustrate this approach, we introduce AgentSandbox,
a conceptual framework embedding these security principles to provide
safeguards throughout an agent's life-cycle. We evaluate with state-of-the-art
LLMs along three dimensions: benign utility, attack utility, and attack success
rate. AgentSandbox maintains high utility for its intended functions under both
benign and adversarial evaluations while substantially mitigating privacy
risks. By embedding secure design principles as foundational elements within
emerging LLM agent protocols, we aim to promote trustworthy agent ecosystems
aligned with user privacy expectations and evolving regulatory requirements.


http://arxiv.org/abs/2505.23448
Network Inversion for Uncertainty-Aware Out-of-Distribution Detection. (1%)
Pirzada Suhail; Rehna Afroz; Amit Sethi
  Out-of-distribution (OOD) detection and uncertainty estimation (UE) are
critical components for building safe machine learning systems, especially in
real-world scenarios where unexpected inputs are inevitable. In this work, we
propose a novel framework that combines network inversion with classifier
training to simultaneously address both OOD detection and uncertainty
estimation. For a standard n-class classification task, we extend the
classifier to an (n+1)-class model by introducing a "garbage" class, initially
populated with random gaussian noise to represent outlier inputs. After each
training epoch, we use network inversion to reconstruct input images
corresponding to all output classes that initially appear as noisy and
incoherent and are therefore excluded to the garbage class for retraining the
classifier. This cycle of training, inversion, and exclusion continues
iteratively till the inverted samples begin to resemble the in-distribution
data more closely, suggesting that the classifier has learned to carve out
meaningful decision boundaries while sanitising the class manifolds by pushing
OOD content into the garbage class. During inference, this training scheme
enables the model to effectively detect and reject OOD samples by classifying
them into the garbage class. Furthermore, the confidence scores associated with
each prediction can be used to estimate uncertainty for both in-distribution
and OOD inputs. Our approach is scalable, interpretable, and does not require
access to external OOD datasets or post-hoc calibration techniques while
providing a unified solution to the dual challenges of OOD detection and
uncertainty estimation.


http://arxiv.org/abs/2505.23706
Distributed Federated Learning for Vehicular Network Security: Anomaly Detection Benefits and Multi-Domain Attack Threats. (1%)
Utku Demir; Yalin E. Sagduyu; Tugba Erpek; Hossein Jafari; Sastry Kompella; Mengran Xue
  In connected and autonomous vehicles, machine learning for safety message
classification has become critical for detecting malicious or anomalous
behavior. However, conventional approaches that rely on centralized data
collection or purely local training face limitations due to the large scale,
high mobility, and heterogeneous data distributions inherent in inter-vehicle
networks. To overcome these challenges, this paper explores Distributed
Federated Learning (DFL), whereby vehicles collaboratively train deep learning
models by exchanging model updates among one-hop neighbors and propagating
models over multiple hops. Using the Vehicular Reference Misbehavior (VeReMi)
Extension Dataset, we show that DFL can significantly improve classification
accuracy across all vehicles compared to learning strictly with local data.
Notably, vehicles with low individual accuracy see substantial accuracy gains
through DFL, illustrating the benefit of knowledge sharing across the network.
We further show that local training data size and time-varying network
connectivity correlate strongly with the model's overall accuracy. We
investigate DFL's resilience and vulnerabilities under attacks in multiple
domains, namely wireless jamming and training data poisoning attacks. Our
results reveal important insights into the vulnerabilities of DFL when
confronted with multi-domain attacks, underlining the need for more robust
strategies to secure DFL in vehicular networks.


http://arxiv.org/abs/2505.24022
The Rich and the Simple: On the Implicit Bias of Adam and SGD. (1%)
Bhavya Vasudeva; Jung Whan Lee; Vatsal Sharan; Mahdi Soltanolkotabi
  Adam is the de facto optimization algorithm for several deep learning
applications, but an understanding of its implicit bias and how it differs from
other algorithms, particularly standard first-order methods such as
(stochastic) gradient descent (GD), remains limited. In practice, neural
networks trained with SGD are known to exhibit simplicity bias -- a tendency to
find simple solutions. In contrast, we show that Adam is more resistant to such
simplicity bias. To demystify this phenomenon, in this paper, we investigate
the differences in the implicit biases of Adam and GD when training two-layer
ReLU neural networks on a binary classification task involving synthetic data
with Gaussian clusters. We find that GD exhibits a simplicity bias, resulting
in a linear decision boundary with a suboptimal margin, whereas Adam leads to
much richer and more diverse features, producing a nonlinear boundary that is
closer to the Bayes' optimal predictor. This richer decision boundary also
allows Adam to achieve higher test accuracy both in-distribution and under
certain distribution shifts. We theoretically prove these results by analyzing
the population gradients. To corroborate our theoretical findings, we present
empirical results showing that this property of Adam leads to superior
generalization across datasets with spurious correlations where neural networks
trained with SGD are known to show simplicity bias and don't generalize well
under certain distributional shifts.


http://arxiv.org/abs/2505.23968
Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention. (1%)
Stephan Rabanser; Ali Shahin Shamsabadi; Olive Franzese; Xiao Wang; Adrian Weller; Nicolas Papernot
  Cautious predictions -- where a machine learning model abstains when
uncertain -- are crucial for limiting harmful errors in safety-critical
applications. In this work, we identify a novel threat: a dishonest institution
can exploit these mechanisms to discriminate or unjustly deny services under
the guise of uncertainty. We demonstrate the practicality of this threat by
introducing an uncertainty-inducing attack called Mirage, which deliberately
reduces confidence in targeted input regions, thereby covertly disadvantaging
specific individuals. At the same time, Mirage maintains high predictive
performance across all data points. To counter this threat, we propose
Confidential Guardian, a framework that analyzes calibration metrics on a
reference dataset to detect artificially suppressed confidence. Additionally,
it employs zero-knowledge proofs of verified inference to ensure that reported
confidence scores genuinely originate from the deployed model. This prevents
the provider from fabricating arbitrary model confidence values while
protecting the model's proprietary details. Our results confirm that
Confidential Guardian effectively prevents the misuse of cautious predictions,
providing verifiable assurances that abstention reflects genuine model
uncertainty rather than malicious intent.


http://arxiv.org/abs/2506.02016
Are classical deep neural networks weakly adversarially robust? (99%)
Nuolin Sun; Linyuan Wang; Dongyang Li; Bin Yan; Lei Li
  Adversarial attacks have received increasing attention and it has been widely
recognized that classical DNNs have weak adversarial robustness. The most
commonly used adversarial defense method, adversarial training, improves the
adversarial accuracy of DNNs by generating adversarial examples and retraining
the model. However, adversarial training requires a significant computational
overhead. In this paper, inspired by existing studies focusing on the
clustering properties of DNN output features at each layer and the Progressive
Feedforward Collapse phenomenon, we propose a method for adversarial example
detection and image recognition that uses layer-wise features to construct
feature paths and computes the correlation between the examples feature paths
and the class-centered feature paths. Experimental results show that the
recognition method achieves 82.77% clean accuracy and 44.17% adversarial
accuracy on the ResNet-20 with PFC. Compared to the adversarial training method
with 77.64% clean accuracy and 52.94% adversarial accuracy, our method exhibits
a trade-off without relying on computationally expensive defense strategies.
Furthermore, on the standard ResNet-18, our method maintains this advantage
with respective metrics of 80.01% and 46.1%. This result reveals inherent
adversarial robustness in DNNs, challenging the conventional understanding of
the weak adversarial robustness in DNNs.


http://arxiv.org/abs/2505.22486
Understanding Adversarial Training with Energy-based Models. (93%)
Mujtaba Hussain Mirza; Maria Rosaria Briglia; Filippo Bartolucci; Senad Beadini; Giuseppe Lisanti; Iacopo Masi
  We aim at using Energy-based Model (EBM) framework to better understand
adversarial training (AT) in classifiers, and additionally to analyze the
intrinsic generative capabilities of robust classifiers. By viewing standard
classifiers through an energy lens, we begin by analyzing how the energies of
adversarial examples, generated by various attacks, differ from those of the
natural samples. The central focus of our work is to understand the critical
phenomena of Catastrophic Overfitting (CO) and Robust Overfitting (RO) in AT
from an energy perspective. We analyze the impact of existing AT approaches on
the energy of samples during training and observe that the behavior of the
``delta energy' -- change in energy between original sample and its adversarial
counterpart -- diverges significantly when CO or RO occurs. After a thorough
analysis of these energy dynamics and their relationship with overfitting, we
propose a novel regularizer, the Delta Energy Regularizer (DER), designed to
smoothen the energy landscape during training. We demonstrate that DER is
effective in mitigating both CO and RO across multiple benchmarks. We further
show that robust classifiers, when being used as generative models, have limits
in handling trade-off between image quality and variability. We propose an
improved technique based on a local class-wise principal component analysis
(PCA) and energy-based guidance for better class-specific initialization and
adaptive stopping, enhancing sample diversity and generation quality.
Considering that we do not explicitly train for generative modeling, we achieve
a competitive Inception Score (IS) and Fr\'echet inception distance (FID)
compared to hybrid discriminative-generative models.


http://arxiv.org/abs/2505.21967
Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack. (92%)
Juan Ren; Mark Dras; Usman Naseem
  Large Vision-Language Models (LVLMs) have shown remarkable capabilities
across a wide range of multimodal tasks. However, their integration of visual
inputs introduces expanded attack surfaces, thereby exposing them to novel
security vulnerabilities. In this work, we conduct a systematic
representational analysis to uncover why conventional adversarial attacks can
circumvent the safety mechanisms embedded in LVLMs. We further propose a novel
two stage evaluation framework for adversarial attacks on LVLMs. The first
stage differentiates among instruction non compliance, outright refusal, and
successful adversarial exploitation. The second stage quantifies the degree to
which the model's output fulfills the harmful intent of the adversarial prompt,
while categorizing refusal behavior into direct refusals, soft refusals, and
partial refusals that remain inadvertently helpful. Finally, we introduce a
normative schema that defines idealized model behavior when confronted with
harmful prompts, offering a principled target for safety alignment in
multimodal systems.


http://arxiv.org/abs/2505.22604
Adversarially Robust AI-Generated Image Detection for Free: An Information Theoretic Perspective. (92%)
Ruixuan Zhang; He Wang; Zhengyu Zhao; Zhiqing Guo; Xun Yang; Yunfeng Diao; Meng Wang
  Rapid advances in Artificial Intelligence Generated Images (AIGI) have
facilitated malicious use, such as forgery and misinformation. Therefore,
numerous methods have been proposed to detect fake images. Although such
detectors have been proven to be universally vulnerable to adversarial attacks,
defenses in this field are scarce. In this paper, we first identify that
adversarial training (AT), widely regarded as the most effective defense,
suffers from performance collapse in AIGI detection. Through an
information-theoretic lens, we further attribute the cause of collapse to
feature entanglement, which disrupts the preservation of feature-label mutual
information. Instead, standard detectors show clear feature separation.
Motivated by this difference, we propose Training-free Robust Detection via
Information-theoretic Measures (TRIM), the first training-free adversarial
defense for AIGI detection. TRIM builds on standard detectors and quantifies
feature shifts using prediction entropy and KL divergence. Extensive
experiments across multiple datasets and attacks validate the superiority of
our TRIM, e.g., outperforming the state-of-the-art defense by 33.88% (28.91%)
on ProGAN (GenImage), while well maintaining original accuracy.


http://arxiv.org/abs/2505.22499
The Meeseeks Mesh: Spatially Consistent 3D Adversarial Objects for BEV Detector. (86%)
Aixuan Li; Mochu Xiang; Jing Zhang; Yuchao Dai
  3D object detection is a critical component in autonomous driving systems. It
allows real-time recognition and detection of vehicles, pedestrians and
obstacles under varying environmental conditions. Among existing methods, 3D
object detection in the Bird's Eye View (BEV) has emerged as the mainstream
framework. To guarantee a safe, robust and trustworthy 3D object detection, 3D
adversarial attacks are investigated, where attacks are placed in 3D
environments to evaluate the model performance, e.g., putting a film on a car,
clothing a pedestrian. The vulnerability of 3D object detection models to 3D
adversarial attacks serves as an important indicator to evaluate the robustness
of the model against perturbations. To investigate this vulnerability, we
generate non-invasive 3D adversarial objects tailored for real-world attack
scenarios. Our method verifies the existence of universal adversarial objects
that are spatially consistent across time and camera views. Specifically, we
employ differentiable rendering techniques to accurately model the spatial
relationship between adversarial objects and the target vehicle. Furthermore,
we introduce an occlusion-aware module to enhance visual consistency and
realism under different viewpoints. To maintain attack effectiveness across
multiple frames, we design a BEV spatial feature-guided optimization strategy.
Experimental results demonstrate that our approach can reliably suppress
vehicle predictions from state-of-the-art 3D object detectors, serving as an
important tool to test robustness of 3D object detection models before
deployment. Moreover, the generated adversarial objects exhibit strong
generalization capabilities, retaining its effectiveness at various positions
and distances in the scene.


http://arxiv.org/abs/2505.23828
Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLM. (83%)
Lei Yu; Yechao Zhang; Ziqi Zhou; Yang Wu; Wei Wan; Minghui Li; Shengshan Hu; Pei Xiaobing; Jing Wang
  With the rapid development of the Vision-Language Model (VLM), significant
progress has been made in Visual Question Answering (VQA) tasks. However,
existing VLM often generate inaccurate answers due to a lack of up-to-date
knowledge. To address this issue, recent research has introduced
Retrieval-Augmented Generation (RAG) techniques, commonly used in Large
Language Models (LLM), into VLM, incorporating external multi-modal knowledge
to enhance the accuracy and practicality of VLM systems. Nevertheless, the RAG
in LLM may be susceptible to data poisoning attacks. RAG-based VLM may also
face the threat of this attack. This paper first reveals the vulnerabilities of
the RAG-based large model under poisoning attack, showing that existing
single-modal RAG poisoning attacks have a 100\% failure rate in multi-modal RAG
scenarios. To address this gap, we propose Spa-VLM (Stealthy Poisoning Attack
on RAG-based VLM), a new paradigm for poisoning attacks on large models. We
carefully craft malicious multi-modal knowledge entries, including adversarial
images and misleading text, which are then injected into the RAG's knowledge
base. When users access the VLM service, the system may generate misleading
outputs. We evaluate Spa-VLM on two Wikipedia datasets and across two different
RAGs. Results demonstrate that our method achieves highly stealthy poisoning,
with the attack success rate exceeding 0.8 after injecting just 5 malicious
entries into knowledge bases with 100K and 2M entries, outperforming
state-of-the-art poisoning attacks designed for RAG-based LLMs. Additionally,
we evaluated several defense mechanisms, all of which ultimately proved
ineffective against Spa-VLM, underscoring the effectiveness and robustness of
our attack.


http://arxiv.org/abs/2505.22271
Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models. (76%)
Yongcan Yu; Yanbo Wang; Ran He; Jian Liang
  While (multimodal) large language models (LLMs) have attracted widespread
attention due to their exceptional capabilities, they remain vulnerable to
jailbreak attacks. Various defense methods are proposed to defend against
jailbreak attacks, however, they are often tailored to specific types of
jailbreak attacks, limiting their effectiveness against diverse adversarial
strategies. For instance, rephrasing-based defenses are effective against text
adversarial jailbreaks but fail to counteract image-based attacks. To overcome
these limitations, we propose a universal defense framework, termed Test-time
IMmunization (TIM), which can adaptively defend against various jailbreak
attacks in a self-evolving way. Specifically, TIM initially trains a gist token
for efficient detection, which it subsequently applies to detect jailbreak
activities during inference. When jailbreak attempts are identified, TIM
implements safety fine-tuning using the detected jailbreak instructions paired
with refusal answers. Furthermore, to mitigate potential performance
degradation in the detector caused by parameter updates during safety
fine-tuning, we decouple the fine-tuning process from the detection module.
Extensive experiments on both LLMs and multimodal LLMs demonstrate the efficacy
of TIM.


http://arxiv.org/abs/2505.22266
FGS-Audio: Fixed-Decoder Framework for Audio Steganography with Adversarial Perturbation Generation. (67%)
Jialin Yan; Yu Cheng; Zhaoxia Yin; Xinpeng Zhang; Shilin Wang; Tanfeng Sun; Xinghao Jiang
  The rapid development of Artificial Intelligence Generated Content (AIGC) has
made high-fidelity generated audio widely available across the Internet,
offering an abundant and versatile source of cover signals for covert
communication. Driven by advances in deep learning, current audio steganography
frameworks are mainly based on encoding-decoding network architectures. While
these methods greatly improve the security of audio steganography, they
typically employ elaborate training workflows and rely on extensive pre-trained
models. To address the aforementioned issues, this paper pioneers a
Fixed-Decoder Framework for Audio Steganography with Adversarial Perturbation
Generation (FGS-Audio). The adversarial perturbations that carry secret
information are embedded into cover audio to generate stego audio. The receiver
only needs to share the structure and weights of the fixed decoding network to
accurately extract the secret information from the stego audio, thus
eliminating the reliance on large pre-trained models. In FGS-Audio, we propose
an audio Adversarial Perturbation Generation (APG) strategy and design a
lightweight fixed decoder. The fixed decoder guarantees reliable extraction of
the hidden message, while the adversarial perturbations are optimized to keep
the stego audio perceptually and statistically close to the cover audio,
thereby improving resistance to steganalysis. The experimental results show
that the method exhibits excellent anti-steganalysis performance under
different relative payloads, outperforming existing SOTA approaches. In terms
of stego audio quality, FGS-Audio achieves an average PSNR improvement of over
10 dB compared to SOTA method.


http://arxiv.org/abs/2506.03167
Distributionally Robust Wireless Semantic Communication with Large AI Models. (15%)
Long Tan Le; Senura Hansaja Wanasekara; Zerun Niu; Yansong Shi; Nguyen H. Tran; Phuong Vo; Walid Saad; Dusit Niyato; Zhu Han; Choong Seon Hong; H. Vincent Poor
  6G wireless systems are expected to support massive volumes of data with
ultra-low latency. However, conventional bit-level transmission strategies
cannot support the efficiency and adaptability required by modern,
data-intensive applications. The concept of semantic communication (SemCom)
addresses this limitation by focusing on transmitting task-relevant semantic
information instead of raw data. While recent efforts incorporating deep
learning and large-scale AI models have improved SemCom's performance, existing
systems remain vulnerable to both semantic-level and transmission-level noise
because they often rely on domain-specific architectures that hinder
generalizability. In this paper, a novel and generalized semantic communication
framework called WaSeCom is proposed to systematically address uncertainty and
enhance robustness. In particular, Wasserstein distributionally robust
optimization is employed to provide resilience against semantic
misinterpretation and channel perturbations. A rigorous theoretical analysis is
performed to establish the robust generalization guarantees of the proposed
framework. Experimental results on image and text transmission demonstrate that
WaSeCom achieves improved robustness under noise and adversarial perturbations.
These results highlight its effectiveness in preserving semantic fidelity
across varying wireless conditions.


http://arxiv.org/abs/2505.22798
Efficient Preimage Approximation for Neural Network Certification. (9%)
Anton Björklund; Mykola Zaitsev; Marta Kwiatkowska
  The growing reliance on artificial intelligence in safety- and
security-critical applications demands effective neural network certification.
A challenging real-world use case is certification against ``patch attacks'',
where adversarial patches or lighting conditions obscure parts of images, for
example traffic signs. One approach to certification, which also gives
quantitative coverage estimates, utilizes preimages of neural networks, i.e.,
the set of inputs that lead to a specified output. However, these preimage
approximation methods, including the state-of-the-art PREMAP algorithm,
struggle with scalability. This paper presents novel algorithmic improvements
to PREMAP involving tighter bounds, adaptive Monte Carlo sampling, and improved
branching heuristics. We demonstrate efficiency improvements of at least an
order of magnitude on reinforcement learning control benchmarks, and show that
our method scales to convolutional neural networks that were previously
infeasible. Our results demonstrate the potential of preimage approximation
methodology for reliability and robustness certification.


http://arxiv.org/abs/2505.22411
Mitigating Overthinking in Large Reasoning Models via Manifold Steering. (8%)
Yao Huang; Huanran Chen; Shouwei Ruan; Yichi Zhang; Xingxing Wei; Yinpeng Dong
  Recent advances in Large Reasoning Models (LRMs) have demonstrated remarkable
capabilities in solving complex tasks such as mathematics and coding. However,
these models frequently exhibit a phenomenon known as overthinking during
inference, characterized by excessive validation loops and redundant
deliberation, leading to substantial computational overheads. In this paper, we
aim to mitigate overthinking by investigating the underlying mechanisms from
the perspective of mechanistic interpretability. We first showcase that the
tendency of overthinking can be effectively captured by a single direction in
the model's activation space and the issue can be eased by intervening the
activations along this direction. However, this efficacy soon reaches a plateau
and even deteriorates as the intervention strength increases. We therefore
systematically explore the activation space and find that the overthinking
phenomenon is actually tied to a low-dimensional manifold, which indicates that
the limited effect stems from the noises introduced by the high-dimensional
steering direction. Based on this insight, we propose Manifold Steering, a
novel approach that elegantly projects the steering direction onto the
low-dimensional activation manifold given the theoretical approximation of the
interference noise. Extensive experiments on DeepSeek-R1 distilled models
validate that our method reduces output tokens by up to 71% while maintaining
and even improving the accuracy on several mathematical benchmarks. Our method
also exhibits robust cross-domain transferability, delivering consistent token
reduction performance in code generation and knowledge-based QA tasks. Code is
available at: https://github.com/Aries-iai/Manifold_Steering.


http://arxiv.org/abs/2505.22839
How Do Diffusion Models Improve Adversarial Robustness? (2%)
Liu Yuezhang; Xue-Xin Wei
  Recent findings suggest that diffusion models significantly enhance empirical
adversarial robustness. While some intuitive explanations have been proposed,
the precise mechanisms underlying these improvements remain unclear. In this
work, we systematically investigate how and how well diffusion models improve
adversarial robustness. First, we observe that diffusion models intriguingly
increase, rather than decrease, the $\ell_p$ distance to clean
samples--challenging the intuition that purification denoises inputs closer to
the original data. Second, we find that the purified images are heavily
influenced by the internal randomness of diffusion models, where a compression
effect arises within each randomness configuration. Motivated by this
observation, we evaluate robustness under fixed randomness and find that the
improvement drops to approximately 24% on CIFAR-10--substantially lower than
prior reports approaching 70%. Importantly, we show that this remaining
robustness gain strongly correlates with the model's ability to compress the
input space, revealing the compression rate as a reliable robustness indicator
without requiring gradient-based analysis. Our findings provide novel insights
into the mechanisms underlying diffusion-based purification, and offer guidance
for developing more effective and principled adversarial purification systems.


http://arxiv.org/abs/2505.22203
Pitfalls of Rule- and Model-based Verifiers -- A Case Study on Mathematical Reasoning. (1%)
Yuzhen Huang; Weihao Zeng; Xingshan Zeng; Qi Zhu; Junxian He
  Trustworthy verifiers are essential for the success of reinforcement learning
with verifiable reward (RLVR), which is the core methodology behind various
large reasoning models such as DeepSeek-R1. In complex domains like
mathematical reasoning, rule-based verifiers have been widely adopted in
previous works to train strong reasoning models. However, the reliability of
these verifiers and their impact on the RL training process remain poorly
understood. In this work, we take mathematical reasoning as a case study and
conduct a comprehensive analysis of various verifiers in both static evaluation
and RL training scenarios. First, we find that current open-source rule-based
verifiers often fail to recognize equivalent answers presented in different
formats across multiple commonly used mathematical datasets, resulting in
non-negligible false negative rates. This limitation adversely affects RL
training performance and becomes more pronounced as the policy model gets
stronger. Subsequently, we investigate model-based verifiers as a potential
solution to address these limitations. While the static evaluation shows that
model-based verifiers achieve significantly higher verification accuracy,
further analysis and RL training results imply that they are highly susceptible
to hacking, where they misclassify certain patterns in responses as correct
(i.e., false positives). This vulnerability is exploited during policy model
optimization, leading to artificially inflated rewards. Our findings underscore
the unique risks inherent to both rule-based and model-based verifiers, aiming
to offer valuable insights to develop more robust reward systems in
reinforcement learning.


http://arxiv.org/abs/2505.21027
TabAttackBench: A Benchmark for Adversarial Attacks on Tabular Data. (99%)
Zhipeng He; Chun Ouyang; Lijie Wen; Cong Liu; Catarina Moreira
  Adversarial attacks pose a significant threat to machine learning models by
inducing incorrect predictions through imperceptible perturbations to input
data. While these attacks have been extensively studied in unstructured data
like images, their application to tabular data presents new challenges. These
challenges arise from the inherent heterogeneity and complex feature
interdependencies in tabular data, which differ significantly from those in
image data. To address these differences, it is crucial to consider
imperceptibility as a key criterion specific to tabular data. Most current
research focuses primarily on achieving effective adversarial attacks, often
overlooking the importance of maintaining imperceptibility. To address this
gap, we propose a new benchmark for adversarial attacks on tabular data that
evaluates both effectiveness and imperceptibility. In this study, we assess the
effectiveness and imperceptibility of five adversarial attacks across four
models using eleven tabular datasets, including both mixed and numerical-only
datasets. Our analysis explores how these factors interact and influence the
overall performance of the attacks. We also compare the results across
different dataset types to understand the broader implications of these
findings. The findings from this benchmark provide valuable insights for
improving the design of adversarial attack algorithms, thereby advancing the
field of adversarial machine learning on tabular data.


http://arxiv.org/abs/2505.21494
Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment. (99%)
Xiaojun Jia; Sensen Gao; Simeng Qin; Tianyu Pang; Chao Du; Yihao Huang; Xinfeng Li; Yiming Li; Bo Li; Yang Liu
  Multimodal large language models (MLLMs) remain vulnerable to transferable
adversarial examples. While existing methods typically achieve targeted attacks
by aligning global features-such as CLIP's [CLS] token-between adversarial and
target samples, they often overlook the rich local information encoded in patch
tokens. This leads to suboptimal alignment and limited transferability,
particularly for closed-source models. To address this limitation, we propose a
targeted transferable adversarial attack method based on feature optimal
alignment, called FOA-Attack, to improve adversarial transfer capability.
Specifically, at the global level, we introduce a global feature loss based on
cosine similarity to align the coarse-grained features of adversarial samples
with those of target samples. At the local level, given the rich local
representations within Transformers, we leverage clustering techniques to
extract compact local patterns to alleviate redundant local features. We then
formulate local feature alignment between adversarial and target samples as an
optimal transport (OT) problem and propose a local clustering optimal transport
loss to refine fine-grained feature alignment. Additionally, we propose a
dynamic ensemble model weighting strategy to adaptively balance the influence
of multiple models during adversarial example generation, thereby further
improving transferability. Extensive experiments across various models
demonstrate the superiority of the proposed method, outperforming
state-of-the-art methods, especially in transferring to closed-source MLLMs.
The code is released at https://github.com/jiaxiaojunQAQ/FOA-Attack.


http://arxiv.org/abs/2505.21181
Boosting Adversarial Transferability via High-Frequency Augmentation and Hierarchical-Gradient Fusion. (98%)
Yayin Zheng; Chen Wan; Zihong Guo; Hailing Kuang; Xiaohai Lu
  Adversarial attacks have become a significant challenge in the security of
machine learning models, particularly in the context of black-box defense
strategies. Existing methods for enhancing adversarial transferability
primarily focus on the spatial domain. This paper presents Frequency-Space
Attack (FSA), a new adversarial attack framework that effectively integrates
frequency-domain and spatial-domain transformations. FSA combines two key
techniques: (1) High-Frequency Augmentation, which applies Fourier transform
with frequency-selective amplification to diversify inputs and emphasize the
critical role of high-frequency components in adversarial attacks, and (2)
Hierarchical-Gradient Fusion, which merges multi-scale gradient decomposition
and fusion to capture both global structures and fine-grained details,
resulting in smoother perturbations. Our experiment demonstrates that FSA
consistently outperforms state-of-the-art methods across various black-box
models. Notably, our proposed FSA achieves an average attack success rate
increase of 23.6% compared with BSR (CVPR 2024) on eight black-box defense
models.


http://arxiv.org/abs/2505.21854
Rethinking Gradient-based Adversarial Attacks on Point Cloud Classification. (98%)
Jun Chen; Xinke Li; Mingyue Xu; Tianrui Li; Chongshou Li
  Gradient-based adversarial attacks have become a dominant approach for
evaluating the robustness of point cloud classification models. However,
existing methods often rely on uniform update rules that fail to consider the
heterogeneous nature of point clouds, resulting in excessive and perceptible
perturbations. In this paper, we rethink the design of gradient-based attacks
by analyzing the limitations of conventional gradient update mechanisms and
propose two new strategies to improve both attack effectiveness and
imperceptibility. First, we introduce WAAttack, a novel framework that
incorporates weighted gradients and an adaptive step-size strategy to account
for the non-uniform contribution of points during optimization. This approach
enables more targeted and subtle perturbations by dynamically adjusting updates
according to the local structure and sensitivity of each point. Second, we
propose SubAttack, a complementary strategy that decomposes the point cloud
into subsets and focuses perturbation efforts on structurally critical regions.
Together, these methods represent a principled rethinking of gradient-based
adversarial attacks for 3D point cloud classification. Extensive experiments
demonstrate that our approach outperforms state-of-the-art baselines in
generating highly imperceptible adversarial examples. Code will be released
upon paper acceptance.


http://arxiv.org/abs/2505.20782
Breaking Dataset Boundaries: Class-Agnostic Targeted Adversarial Attacks. (98%)
Taïga Gonçalves; Tomo Miyazaki; Shinichiro Omachi
  We present Cross-Domain Multi-Targeted Attack (CD-MTA), a method for
generating adversarial examples that mislead image classifiers toward any
target class, including those not seen during training. Traditional targeted
attacks are limited to one class per model, requiring expensive retraining for
each target. Multi-targeted attacks address this by introducing a perturbation
generator with a conditional input to specify the target class. However,
existing methods are constrained to classes observed during training and
require access to the black-box model's training data--introducing a form of
data leakage that undermines realistic evaluation in practical black-box
scenarios. We identify overreliance on class embeddings as a key limitation,
leading to overfitting and poor generalization to unseen classes. To address
this, CD-MTA replaces class-level supervision with an image-based conditional
input and introduces class-agnostic losses that align the perturbed and target
images in the feature space. This design removes dependence on class semantics,
thereby enabling generalization to unseen classes across datasets. Experiments
on ImageNet and seven other datasets show that CD-MTA outperforms prior
multi-targeted attacks in both standard and cross-domain settings--without
accessing the black-box model's training data.


http://arxiv.org/abs/2505.21414
A Framework for Adversarial Analysis of Decision Support Systems Prior to Deployment. (96%)
Brett Bissey; Kyle Gatesman; Walker Dimon; Mohammad Alam; Luis Robaina; Joseph Weissman
  This paper introduces a comprehensive framework designed to analyze and
secure decision-support systems trained with Deep Reinforcement Learning (DRL),
prior to deployment, by providing insights into learned behavior patterns and
vulnerabilities discovered through simulation. The introduced framework aids in
the development of precisely timed and targeted observation perturbations,
enabling researchers to assess adversarial attack outcomes within a strategic
decision-making context. We validate our framework, visualize agent behavior,
and evaluate adversarial outcomes within the context of a custom-built
strategic game, CyberStrike. Utilizing the proposed framework, we introduce a
method for systematically discovering and ranking the impact of attacks on
various observation indices and time-steps, and we conduct experiments to
evaluate the transferability of adversarial attacks across agent architectures
and DRL training algorithms. The findings underscore the critical need for
robust adversarial defense mechanisms to protect decision-making policies in
high-stakes environments.


http://arxiv.org/abs/2505.20934
NatADiff: Adversarial Boundary Guidance for Natural Adversarial Diffusion. (95%)
Max Collins; Jordan Vice; Tim French; Ajmal Mian
  Adversarial samples exploit irregularities in the manifold ``learned'' by
deep learning models to cause misclassifications. The study of these
adversarial samples provides insight into the features a model uses to classify
inputs, which can be leveraged to improve robustness against future attacks.
However, much of the existing literature focuses on constrained adversarial
samples, which do not accurately reflect test-time errors encountered in
real-world settings. To address this, we propose `NatADiff', an adversarial
sampling scheme that leverages denoising diffusion to generate natural
adversarial samples. Our approach is based on the observation that natural
adversarial samples frequently contain structural elements from the adversarial
class. Deep learning models can exploit these structural elements to shortcut
the classification process, rather than learning to genuinely distinguish
between classes. To leverage this behavior, we guide the diffusion trajectory
towards the intersection of the true and adversarial classes, combining
time-travel sampling with augmented classifier guidance to enhance attack
transferability while preserving image fidelity. Our method achieves comparable
attack success rates to current state-of-the-art techniques, while exhibiting
significantly higher transferability across model architectures and better
alignment with natural test-time errors as measured by FID. These results
demonstrate that NatADiff produces adversarial samples that not only transfer
more effectively across models, but more faithfully resemble naturally
occurring test-time errors.


http://arxiv.org/abs/2505.21609
Preventing Adversarial AI Attacks Against Autonomous Situational Awareness: A Maritime Case Study. (92%)
Mathew J. Walter; Aaron Barrett; Kimberly Tam
  Adversarial artificial intelligence (AI) attacks pose a significant threat to
autonomous transportation, such as maritime vessels, that rely on AI
components. Malicious actors can exploit these systems to deceive and
manipulate AI-driven operations. This paper addresses three critical research
challenges associated with adversarial AI: the limited scope of traditional
defences, inadequate security metrics, and the need to build resilience beyond
model-level defences. To address these challenges, we propose building defences
utilising multiple inputs and data fusion to create defensive components and an
AI security metric as a novel approach toward developing more secure AI
systems. We name this approach the Data Fusion Cyber Resilience (DFCR) method,
and we evaluate it through real-world demonstrations and comprehensive
quantitative analyses, comparing a system built with the DFCR method against
single-input models and models utilising existing state-of-the-art defences.
The findings show that the DFCR approach significantly enhances resilience
against adversarial machine learning attacks in maritime autonomous system
operations, achieving up to a 35\% reduction in loss for successful
multi-pronged perturbation attacks, up to a 100\% reduction in loss for
successful adversarial patch attacks and up to 100\% reduction in loss for
successful spoofing attacks when using these more resilient systems. We
demonstrate how DFCR and DFCR confidence scores can reduce adversarial AI
contact confidence and improve decision-making by the system, even when typical
adversarial defences have been compromised. Ultimately, this work contributes
to the development of more secure and resilient AI-driven systems against
adversarial attacks.


http://arxiv.org/abs/2505.21499
AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery. (83%)
Haowei Wang; Junjie Wang; Xiaojun Jia; Rupeng Zhang; Mingyang Li; Zhe Liu; Yang Liu; Qing Wang
  Vision-Language Model (VLM) based Web Agents represent a significant step
towards automating complex tasks by simulating human-like interaction with
websites. However, their deployment in uncontrolled web environments introduces
significant security vulnerabilities. Existing research on adversarial
environmental injection attacks often relies on unrealistic assumptions, such
as direct HTML manipulation, knowledge of user intent, or access to agent model
parameters, limiting their practical applicability. In this paper, we propose
AdInject, a novel and real-world black-box attack method that leverages the
internet advertising delivery to inject malicious content into the Web Agent's
environment. AdInject operates under a significantly more realistic threat
model than prior work, assuming a black-box agent, static malicious content
constraints, and no specific knowledge of user intent. AdInject includes
strategies for designing malicious ad content aimed at misleading agents into
clicking, and a VLM-based ad content optimization technique that infers
potential user intents from the target website's context and integrates these
intents into the ad content to make it appear more relevant or critical to the
agent's task, thus enhancing attack effectiveness. Experimental evaluations
demonstrate the effectiveness of AdInject, attack success rates exceeding 60%
in most scenarios and approaching 100% in certain cases. This strongly
demonstrates that prevalent advertising delivery constitutes a potent and
real-world vector for environment injection attacks against Web Agents. This
work highlights a critical vulnerability in Web Agent security arising from
real-world environment manipulation channels, underscoring the urgent need for
developing robust defense mechanisms against such threats. Our code is
available at https://github.com/NicerWang/AdInject.


http://arxiv.org/abs/2505.21620
VideoMarkBench: Benchmarking Robustness of Video Watermarking. (74%)
Zhengyuan Jiang; Moyang Guo; Kecen Li; Yuepeng Hu; Yupu Wang; Zhicong Huang; Cheng Hong; Neil Zhenqiang Gong
  The rapid development of video generative models has led to a surge in highly
realistic synthetic videos, raising ethical concerns related to disinformation
and copyright infringement. Recently, video watermarking has been proposed as a
mitigation strategy by embedding invisible marks into AI-generated videos to
enable subsequent detection. However, the robustness of existing video
watermarking methods against both common and adversarial perturbations remains
underexplored. In this work, we introduce VideoMarkBench, the first systematic
benchmark designed to evaluate the robustness of video watermarks under
watermark removal and watermark forgery attacks. Our study encompasses a
unified dataset generated by three state-of-the-art video generative models,
across three video styles, incorporating four watermarking methods and seven
aggregation strategies used during detection. We comprehensively evaluate 12
types of perturbations under white-box, black-box, and no-box threat models.
Our findings reveal significant vulnerabilities in current watermarking
approaches and highlight the urgent need for more robust solutions. Our code is
available at https://github.com/zhengyuan-jiang/VideoMarkBench.


http://arxiv.org/abs/2505.21140
HeteroBA: A Structure-Manipulating Backdoor Attack on Heterogeneous Graphs. (68%)
Honglin Gao; Xiang Li; Lan Zhao; Gaoxi Xiao
  Heterogeneous graph neural networks (HGNNs) have recently drawn increasing
attention for modeling complex multi-relational data in domains such as
recommendation, finance, and social networks. While existing research has been
largely focused on enhancing HGNNs' predictive performance, their robustness
and security, especially under backdoor attacks, remain underexplored. In this
paper, we propose a novel Heterogeneous Backdoor Attack (HeteroBA) framework
for node classification tasks on heterogeneous graphs. HeteroBA inserts
carefully crafted trigger nodes with realistic features and targeted structural
connections, leveraging attention-based and clustering-based strategies to
select influential auxiliary nodes for effective trigger propagation, thereby
causing the model to misclassify specific nodes into a target label while
maintaining accuracy on clean data. Experimental results on three datasets and
various HGNN architectures demonstrate that HeteroBA achieves high attack
success rates with minimal impact on the clean accuracy. Our method sheds light
on potential vulnerabilities in HGNNs and calls for more robust defenses
against backdoor threats in multi-relational graph scenarios.


http://arxiv.org/abs/2505.21277
Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space. (64%)
Yao Huang; Yitong Sun; Shouwei Ruan; Yichi Zhang; Yinpeng Dong; Xingxing Wei
  Large Language Models (LLMs), despite advanced general capabilities, still
suffer from numerous safety risks, especially jailbreak attacks that bypass
safety protocols. Understanding these vulnerabilities through black-box
jailbreak attacks, which better reflect real-world scenarios, offers critical
insights into model robustness. While existing methods have shown improvements
through various prompt engineering techniques, their success remains limited
against safety-aligned models, overlooking a more fundamental problem: the
effectiveness is inherently bounded by the predefined strategy spaces. However,
expanding this space presents significant challenges in both systematically
capturing essential attack patterns and efficiently navigating the increased
complexity. To better explore the potential of expanding the strategy space, we
address these challenges through a novel framework that decomposes jailbreak
strategies into essential components based on the Elaboration Likelihood Model
(ELM) theory and develops genetic-based optimization with intention evaluation
mechanisms. To be striking, our experiments reveal unprecedented jailbreak
capabilities by expanding the strategy space: we achieve over 90% success rate
on Claude-3.5 where prior methods completely fail, while demonstrating strong
cross-model transferability and surpassing specialized safeguard models in
evaluation accuracy. The code is open-sourced at:
https://github.com/Aries-iai/CL-GSO.


http://arxiv.org/abs/2505.21742
What is Adversarial Training for Diffusion Models? (33%)
Briglia Maria Rosaria; Mujtaba Hussain Mirza; Giuseppe Lisanti; Iacopo Masi
  We answer the question in the title, showing that adversarial training (AT)
for diffusion models (DMs) fundamentally differs from classifiers: while AT in
classifiers enforces output invariance, AT in DMs requires equivariance to keep
the diffusion process aligned with the data distribution. AT is a way to
enforce smoothness in the diffusion flow, improving robustness to outliers and
corrupted data. Unlike prior art, our method makes no assumptions about the
noise model and integrates seamlessly into diffusion training by adding random
noise, similar to randomized smoothing, or adversarial noise, akin to AT. This
enables intrinsic capabilities such as handling noisy data, dealing with
extreme variability such as outliers, preventing memorization, and improving
robustness. We rigorously evaluate our approach with proof-of-concept datasets
with known distributions in low- and high-dimensional space, thereby taking a
perfect measure of errors; we further evaluate on standard benchmarks such as
CIFAR-10, CelebA and LSUN Bedroom, showing strong performance under severe
noise, data corruption, and iterative adversarial attacks.


http://arxiv.org/abs/2505.21938
Practical Adversarial Attacks on Stochastic Bandits via Fake Data Injection. (22%)
Qirun Zeng; Eric He; Richard Hoffmann; Xuchuang Wang; Jinhang Zuo
  Adversarial attacks on stochastic bandits have traditionally relied on some
unrealistic assumptions, such as per-round reward manipulation and unbounded
perturbations, limiting their relevance to real-world systems. We propose a
more practical threat model, Fake Data Injection, which reflects realistic
adversarial constraints: the attacker can inject only a limited number of
bounded fake feedback samples into the learner's history, simulating legitimate
interactions. We design efficient attack strategies under this model,
explicitly addressing both magnitude constraints (on reward values) and
temporal constraints (on when and how often data can be injected). Our
theoretical analysis shows that these attacks can mislead both Upper Confidence
Bound (UCB) and Thompson Sampling algorithms into selecting a target arm in
nearly all rounds while incurring only sublinear attack cost. Experiments on
synthetic and real-world datasets validate the effectiveness of our strategies,
revealing significant vulnerabilities in widely used stochastic bandit
algorithms under practical adversarial scenarios.


http://arxiv.org/abs/2505.23817
System Prompt Extraction Attacks and Defenses in Large Language Models. (8%)
Badhan Chandra Das; M. Hadi Amini; Yanzhao Wu
  The system prompt in Large Language Models (LLMs) plays a pivotal role in
guiding model behavior and response generation. Often containing private
configuration details, user roles, and operational instructions, the system
prompt has become an emerging attack target. Recent studies have shown that LLM
system prompts are highly susceptible to extraction attacks through
meticulously designed queries, raising significant privacy and security
concerns. Despite the growing threat, there is a lack of systematic studies of
system prompt extraction attacks and defenses. In this paper, we present a
comprehensive framework, SPE-LLM, to systematically evaluate System Prompt
Extraction attacks and defenses in LLMs. First, we design a set of novel
adversarial queries that effectively extract system prompts in state-of-the-art
(SOTA) LLMs, demonstrating the severe risks of LLM system prompt extraction
attacks. Second, we propose three defense techniques to mitigate system prompt
extraction attacks in LLMs, providing practical solutions for secure LLM
deployments. Third, we introduce a set of rigorous evaluation metrics to
accurately quantify the severity of system prompt extraction attacks in LLMs
and conduct comprehensive experiments across multiple benchmark datasets, which
validates the efficacy of our proposed SPE-LLM framework.


http://arxiv.org/abs/2505.20819
Tracing and Reversing Rank-One Model Edits. (3%)
Paul Youssef; Zhixue Zhao; Christin Seifert; Jörg Schlötterer
  Knowledge editing methods (KEs) are a cost-effective way to update the
factual content of large language models (LLMs), but they pose a dual-use risk.
While KEs are beneficial for updating outdated or incorrect information, they
can be exploited maliciously to implant misinformation or bias. In order to
defend against these types of malicious manipulation, we need robust techniques
that can reliably detect, interpret, and mitigate adversarial edits. This work
investigates the traceability and reversibility of knowledge edits, focusing on
the widely used Rank-One Model Editing (ROME) method. We first show that ROME
introduces distinctive distributional patterns in the edited weight matrices,
which can serve as effective signals for locating the edited weights. Second,
we show that these altered weights can reliably be used to predict the edited
factual relation, enabling partial reconstruction of the modified fact.
Building on this, we propose a method to infer the edited object entity
directly from the modified weights, without access to the editing prompt,
achieving over 95% accuracy. Finally, we demonstrate that ROME edits can be
reversed, recovering the model's original outputs with $\geq$ 80% accuracy. Our
findings highlight the feasibility of detecting, tracing, and reversing edits
based on the edited weights, offering a robust framework for safeguarding LLMs
against adversarial manipulations.


http://arxiv.org/abs/2505.21074
Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling. (3%)
Yichuan Cao; Yibo Miao; Xiao-Shan Gao; Yinpeng Dong
  Text-to-image (T2I) models raise ethical and safety concerns due to their
potential to generate inappropriate or harmful images. Evaluating these models'
security through red-teaming is vital, yet white-box approaches are limited by
their need for internal access, complicating their use with closed-source
models. Moreover, existing black-box methods often assume knowledge about the
model's specific defense mechanisms, limiting their utility in real-world
commercial API scenarios. A significant challenge is how to evade unknown and
diverse defense mechanisms. To overcome this difficulty, we propose a novel
Rule-based Preference modeling Guided Red-Teaming (RPG-RT), which iteratively
employs LLM to modify prompts to query and leverages feedback from T2I systems
for fine-tuning the LLM. RPG-RT treats the feedback from each iteration as a
prior, enabling the LLM to dynamically adapt to unknown defense mechanisms.
Given that the feedback is often labeled and coarse-grained, making it
difficult to utilize directly, we further propose rule-based preference
modeling, which employs a set of rules to evaluate desired or undesired
feedback, facilitating finer-grained control over the LLM's dynamic adaptation
process. Extensive experiments on nineteen T2I systems with varied safety
mechanisms, three online commercial API services, and T2V models verify the
superiority and practicality of our approach.


http://arxiv.org/abs/2505.21772
Calibrating LLM Confidence by Probing Perturbed Representation Stability. (2%)
Reza Khanmohammadi; Erfan Miahi; Mehrsa Mardikoraem; Simerjot Kaur; Ivan Brugere; Charese H. Smiley; Kundan Thind; Mohammad M. Ghassemi
  Miscalibration in Large Language Models (LLMs) undermines their reliability,
highlighting the need for accurate confidence estimation. We introduce CCPS
(Calibrating LLM Confidence by Probing Perturbed Representation Stability), a
novel method analyzing internal representational stability in LLMs. CCPS
applies targeted adversarial perturbations to final hidden states, extracts
features reflecting the model's response to these perturbations, and uses a
lightweight classifier to predict answer correctness. CCPS was evaluated on
LLMs from 8B to 32B parameters (covering Llama, Qwen, and Mistral
architectures) using MMLU and MMLU-Pro benchmarks in both multiple-choice and
open-ended formats. Our results show that CCPS significantly outperforms
current approaches. Across four LLMs and three MMLU variants, CCPS reduces
Expected Calibration Error by approximately 55% and Brier score by 21%, while
increasing accuracy by 5 percentage points, Area Under the Precision-Recall
Curve by 4 percentage points, and Area Under the Receiver Operating
Characteristic Curve by 6 percentage points, all relative to the strongest
prior method. CCPS delivers an efficient, broadly applicable, and more accurate
solution for estimating LLM confidence, thereby improving their
trustworthiness.


http://arxiv.org/abs/2505.21380
PHISH in MESH: Korean Adversarial Phonetic Substitution and Phonetic-Semantic Feature Integration Defense. (1%)
Byungjun Kim; Minju Kim; Hyeonchu Park; Bugeun Kim
  As malicious users increasingly employ phonetic substitution to evade hate
speech detection, researchers have investigated such strategies. However, two
key challenges remain. First, existing studies have overlooked the Korean
language, despite its vulnerability to phonetic perturbations due to its
phonographic nature. Second, prior work has primarily focused on constructing
datasets rather than developing architectural defenses. To address these
challenges, we propose (1) PHonetic-Informed Substitution for Hangul (PHISH)
that exploits the phonological characteristics of the Korean writing system,
and (2) Mixed Encoding of Semantic-pHonetic features (MESH) that enhances the
detector's robustness by incorporating phonetic information at the
architectural level. Our experimental results demonstrate the effectiveness of
our proposed methods on both perturbed and unperturbed datasets, suggesting
that they not only improve detection performance but also reflect realistic
adversarial behaviors employed by malicious users.


http://arxiv.org/abs/2505.21936
RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments. (1%)
Zeyi Liao; Jaylen Jones; Linxi Jiang; Eric Fosler-Lussier; Yu Su; Zhiqiang Lin; Huan Sun
  Computer-use agents (CUAs) promise to automate complex tasks across operating
systems (OS) and the web, but remain vulnerable to indirect prompt injection.
Current evaluations of this threat either lack support realistic but controlled
environments or ignore hybrid web-OS attack scenarios involving both
interfaces. To address this, we propose RedTeamCUA, an adversarial testing
framework featuring a novel hybrid sandbox that integrates a VM-based OS
environment with Docker-based web platforms. Our sandbox supports key features
tailored for red teaming, such as flexible adversarial scenario configuration,
and a setting that decouples adversarial evaluation from navigational
limitations of CUAs by initializing tests directly at the point of an
adversarial injection. Using RedTeamCUA, we develop RTC-Bench, a comprehensive
benchmark with 864 examples that investigate realistic, hybrid web-OS attack
scenarios and fundamental security vulnerabilities. Benchmarking current
frontier CUAs identifies significant vulnerabilities: Claude 3.7 Sonnet | CUA
demonstrates an ASR of 42.9%, while Operator, the most secure CUA evaluated,
still exhibits an ASR of 7.6%. Notably, CUAs often attempt to execute
adversarial tasks with an Attempt Rate as high as 92.5%, although failing to
complete them due to capability limitations. Nevertheless, we observe
concerning ASRs of up to 50% in realistic end-to-end settings, with the
recently released frontier Claude 4 Opus | CUA showing an alarming ASR of 48%,
demonstrating that indirect prompt injection presents tangible risks for even
advanced CUAs despite their capabilities and safeguards. Overall, RedTeamCUA
provides an essential framework for advancing realistic, controlled, and
systematic analysis of CUA vulnerabilities, highlighting the urgent need for
robust defenses to indirect prompt injection prior to real-world deployment.


http://arxiv.org/abs/2505.19911
Attention! You Vision Language Model Could Be Maliciously Manipulated. (99%)
Xiaosen Wang; Shaokang Wang; Zhijin Ge; Yuyang Luo; Shudong Zhang
  Large Vision-Language Models (VLMs) have achieved remarkable success in
understanding complex real-world scenarios and supporting data-driven
decision-making processes. However, VLMs exhibit significant vulnerability
against adversarial examples, either text or image, which can lead to various
adversarial outcomes, e.g., jailbreaking, hijacking, and hallucination, etc. In
this work, we empirically and theoretically demonstrate that VLMs are
particularly susceptible to image-based adversarial examples, where
imperceptible perturbations can precisely manipulate each output token. To this
end, we propose a novel attack called Vision-language model Manipulation Attack
(VMA), which integrates first-order and second-order momentum optimization
techniques with a differentiable transformation mechanism to effectively
optimize the adversarial perturbation. Notably, VMA can be a double-edged
sword: it can be leveraged to implement various attacks, such as jailbreaking,
hijacking, privacy breaches, Denial-of-Service, and the generation of sponge
examples, etc, while simultaneously enabling the injection of watermarks for
copyright protection. Extensive empirical evaluations substantiate the efficacy
and generalizability of VMA across diverse scenarios and datasets.


http://arxiv.org/abs/2505.19840
One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP. (99%)
Binyan Xu; Xilin Dai; Di Tang; Kehuan Zhang
  Deep Neural Networks (DNNs) have achieved widespread success yet remain prone
to adversarial attacks. Typically, such attacks either involve frequent queries
to the target model or rely on surrogate models closely mirroring the target
model -- often trained with subsets of the target model's training data -- to
achieve high attack success rates through transferability. However, in
realistic scenarios where training data is inaccessible and excessive queries
can raise alarms, crafting adversarial examples becomes more challenging. In
this paper, we present UnivIntruder, a novel attack framework that relies
solely on a single, publicly available CLIP model and publicly available
datasets. By using textual concepts, UnivIntruder generates universal,
transferable, and targeted adversarial perturbations that mislead DNNs into
misclassifying inputs into adversary-specified classes defined by textual
concepts.
  Our extensive experiments show that our approach achieves an Attack Success
Rate (ASR) of up to 85% on ImageNet and over 99% on CIFAR-10, significantly
outperforming existing transfer-based methods. Additionally, we reveal
real-world vulnerabilities, showing that even without querying target models,
UnivIntruder compromises image search engines like Google and Baidu with ASR
rates up to 84%, and vision language models like GPT-4 and Claude-3.5 with ASR
rates up to 80%. These findings underscore the practicality of our attack in
scenarios where traditional avenues are blocked, highlighting the need to
reevaluate security paradigms in AI applications.


http://arxiv.org/abs/2505.19864
CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models. (92%)
Chunyang Li; Junwei Zhang; Anda Cheng; Zhuo Ma; Xinghua Li; Jianfeng Ma
  Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by
incorporating external knowledge, but its openness introduces vulnerabilities
that can be exploited by poisoning attacks. Existing poisoning methods for RAG
systems have limitations, such as poor generalization and lack of fluency in
adversarial texts. In this paper, we propose CPA-RAG, a black-box adversarial
framework that generates query-relevant texts capable of manipulating the
retrieval process to induce target answers. The proposed method integrates
prompt-based text generation, cross-guided optimization through multiple LLMs,
and retriever-based scoring to construct high-quality adversarial samples. We
conduct extensive experiments across multiple datasets and LLMs to evaluate its
effectiveness. Results show that the framework achieves over 90\% attack
success when the top-k retrieval setting is 5, matching white-box performance,
and maintains a consistent advantage of approximately 5 percentage points
across different top-k values. It also outperforms existing black-box baselines
by 14.5 percentage points under various defense strategies. Furthermore, our
method successfully compromises a commercial RAG system deployed on Alibaba's
BaiLian platform, demonstrating its practical threat in real-world
applications. These findings underscore the need for more robust and secure RAG
frameworks to defend against poisoning attacks.


http://arxiv.org/abs/2505.20621
Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning. (82%)
Shijie Liu; Andrew C. Cullen; Paul Montague; Sarah Erfani; Benjamin I. P. Rubinstein
  Similar to other machine learning frameworks, Offline Reinforcement Learning
(RL) is shown to be vulnerable to poisoning attacks, due to its reliance on
externally sourced datasets, a vulnerability that is exacerbated by its
sequential nature. To mitigate the risks posed by RL poisoning, we extend
certified defenses to provide larger guarantees against adversarial
manipulation, ensuring robustness for both per-state actions, and the overall
expected cumulative reward. Our approach leverages properties of Differential
Privacy, in a manner that allows this work to span both continuous and discrete
spaces, as well as stochastic and deterministic environments -- significantly
expanding the scope and applicability of achievable guarantees. Empirical
evaluations demonstrate that our approach ensures the performance drops to no
more than $50\%$ with up to $7\%$ of the training data poisoned, significantly
improving over the $0.008\%$ in prior work~\citep{wu_copa_2022}, while
producing certified radii that is $5$ times larger as well. This highlights the
potential of our framework to enhance safety and reliability in offline RL.


http://arxiv.org/abs/2505.19610
JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Models. (81%)
Jiaxin Song; Yixu Wang; Jie Li; Rui Yu; Yan Teng; Xingjun Ma; Yingchun Wang
  Vision-Language Models (VLMs) exhibit impressive performance, yet the
integration of powerful vision encoders has significantly broadened their
attack surface, rendering them increasingly susceptible to jailbreak attacks.
However, lacking well-defined attack objectives, existing jailbreak methods
often struggle with gradient-based strategies prone to local optima and lacking
precise directional guidance, and typically decouple visual and textual
modalities, thereby limiting their effectiveness by neglecting crucial
cross-modal interactions. Inspired by the Eliciting Latent Knowledge (ELK)
framework, we posit that VLMs encode safety-relevant information within their
internal fusion-layer representations, revealing an implicit safety decision
boundary in the latent space. This motivates exploiting boundary to steer model
behavior. Accordingly, we propose JailBound, a novel latent space jailbreak
framework comprising two stages: (1) Safety Boundary Probing, which addresses
the guidance issue by approximating decision boundary within fusion layer's
latent space, thereby identifying optimal perturbation directions towards the
target region; and (2) Safety Boundary Crossing, which overcomes the
limitations of decoupled approaches by jointly optimizing adversarial
perturbations across both image and text inputs. This latter stage employs an
innovative mechanism to steer the model's internal state towards
policy-violating outputs while maintaining cross-modal semantic consistency.
Extensive experiments on six diverse VLMs demonstrate JailBound's efficacy,
achieves 94.32% white-box and 67.28% black-box attack success averagely, which
are 6.17% and 21.13% higher than SOTA methods, respectively. Our findings
expose a overlooked safety risk in VLMs and highlight the urgent need for more
robust defenses. Warning: This paper contains potentially sensitive, harmful
and offensive content.


http://arxiv.org/abs/2505.19951
Novel Loss-Enhanced Universal Adversarial Patches for Sustainable Speaker Privacy. (22%)
Elvir Karimov; Alexander Varlamov; Danil Ivanov; Dmitrii Korzh; Oleg Y. Rogov
  Deep learning voice models are commonly used nowadays, but the safety
processing of personal data, such as human identity and speech content, remains
suspicious. To prevent malicious user identification, speaker anonymization
methods were proposed. Current methods, particularly based on universal
adversarial patch (UAP) applications, have drawbacks such as significant
degradation of audio quality, decreased speech recognition quality, low
transferability across different voice biometrics models, and performance
dependence on the input audio length. To mitigate these drawbacks, in this
work, we introduce and leverage the novel Exponential Total Variance (TV) loss
function and provide experimental evidence that it positively affects UAP
strength and imperceptibility. Moreover, we present a novel scalable UAP
insertion procedure and demonstrate its uniformly high performance for various
audio lengths.


http://arxiv.org/abs/2505.20259
Lifelong Safety Alignment for Language Models. (13%)
Haoyu Wang; Zeyu Qin; Yifei Zhao; Chao Du; Min Lin; Xueqian Wang; Tianyu Pang
  LLMs have made impressive progress, but their growing capabilities also
expose them to highly flexible jailbreaking attacks designed to bypass safety
alignment. While many existing defenses focus on known types of attacks, it is
more critical to prepare LLMs for unseen attacks that may arise during
deployment. To address this, we propose a lifelong safety alignment framework
that enables LLMs to continuously adapt to new and evolving jailbreaking
strategies. Our framework introduces a competitive setup between two
components: a Meta-Attacker, trained to actively discover novel jailbreaking
strategies, and a Defender, trained to resist them. To effectively warm up the
Meta-Attacker, we first leverage the GPT-4o API to extract key insights from a
large collection of jailbreak-related research papers. Through iterative
training, the first iteration Meta-Attacker achieves a 73% attack success rate
(ASR) on RR and a 57% transfer ASR on LAT using only single-turn attacks.
Meanwhile, the Defender progressively improves its robustness and ultimately
reduces the Meta-Attacker's success rate to just 7%, enabling safer and more
reliable deployment of LLMs in open-ended environments. The code is available
at https://github.com/sail-sg/LifelongSafetyAlignment.


http://arxiv.org/abs/2505.20269
Comparing Neural Network Encodings for Logic-based Explainability. (11%)
Levi Cordeiro Carvalho; Saulo A. F. Oliveira; Thiago Alves Rocha
  Providing explanations for the outputs of artificial neural networks (ANNs)
is crucial in many contexts, such as critical systems, data protection laws and
handling adversarial examples. Logic-based methods can offer explanations with
correctness guarantees, but face scalability challenges. Due to these issues,
it is necessary to compare different encodings of ANNs into logical
constraints, which are used in logic-based explainability. This work compares
two encodings of ANNs: one has been used in the literature to provide
explanations, while the other will be adapted for our context of
explainability. Additionally, the second encoding uses fewer variables and
constraints, thus, potentially enhancing efficiency. Experiments showed similar
running times for computing explanations, but the adapted encoding performed up
to 18\% better in building logical constraints and up to 16\% better in overall
time.


http://arxiv.org/abs/2506.17231
Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs. (10%)
Xiang Li; Chong Zhang; Jia Wang; Fangyu Wu; Yushi Li; Xiaobo Jin
Attacks on large language models (LLMs) in jailbreaking scenarios raise many security and ethical issues. Current jailbreak attack methods face problems such as low efficiency, high computational cost, and poor cross-model adaptability and versatility, which make it difficult to cope with the rapid development of LLM and new defense strategies. Our work proposes an Adversarial Prompt Distillation, which combines masked language modeling, reinforcement learning, and dynamic temperature control through a prompt generation and distillation method. It enables small language models (SLMs) to jailbreak attacks on mainstream LLMs. The experimental results verify the superiority of the proposed method in terms of attack success rate and harm, and reflect the resource efficiency and cross-model adaptability. This research explores the feasibility of distilling the jailbreak ability of LLM to SLM, reveals the model's vulnerability, and provides a new idea for LLM security research.

http://arxiv.org/abs/2505.21556
Benign-to-Toxic Jailbreaking: Inducing Harmful Responses from Harmless Prompts. (5%)
Hee-Seon Kim; Minbeom Kim; Wonjun Lee; Kihyun Kim; Changick Kim
  Optimization-based jailbreaks typically adopt the Toxic-Continuation setting
in large vision-language models (LVLMs), following the standard next-token
prediction objective. In this setting, an adversarial image is optimized to
make the model predict the next token of a toxic prompt. However, we find that
the Toxic-Continuation paradigm is effective at continuing already-toxic
inputs, but struggles to induce safety misalignment when explicit toxic signals
are absent. We propose a new paradigm: Benign-to-Toxic (B2T) jailbreak. Unlike
prior work, we optimize adversarial images to induce toxic outputs from benign
conditioning. Since benign conditioning contains no safety violations, the
image alone must break the model's safety mechanisms. Our method outperforms
prior approaches, transfers in black-box settings, and complements text-based
jailbreaks. These results reveal an underexplored vulnerability in multimodal
alignment and introduce a fundamentally new direction for jailbreak approaches.


http://arxiv.org/abs/2505.20158
Evaluating Software Plagiarism Detection in the Age of AI: Automated Obfuscation and Lessons for Academic Integrity. (4%)
Timur Sağlam; Larissa Schmid
  Plagiarism in programming assignments is a persistent issue in computer
science education, increasingly complicated by the emergence of automated
obfuscation attacks. While software plagiarism detectors are widely used to
identify suspicious similarities at scale and are resilient to simple
obfuscation techniques, they are vulnerable to advanced obfuscation based on
structural modification of program code that preserves the original program
behavior. While different defense mechanisms have been proposed to increase
resilience against these attacks, their current evaluation is limited to the
scope of attacks used and lacks a comprehensive investigation regarding
AI-based obfuscation. In this paper, we investigate the resilience of these
defense mechanisms against a broad range of automated obfuscation attacks,
including both algorithmic and AI-generated methods, and for a wide variety of
real-world datasets. We evaluate the improvements of two defense mechanisms
over the plagiarism detector JPlag across over four million pairwise program
comparisons. Our results show significant improvements in detecting obfuscated
plagiarism instances, and we observe an improved detection of AI-generated
programs, even though the defense mechanisms are not designed for this use
case. Based on our findings, we provide an in-depth discussion of their broader
implications for academic integrity and the role of AI in education.


http://arxiv.org/abs/2505.20162
Capability-Based Scaling Laws for LLM Red-Teaming. (3%)
Alexander Panfilov; Paul Kassianik; Maksym Andriushchenko; Jonas Geiping
  As large language models grow in capability and agency, identifying
vulnerabilities through red-teaming becomes vital for safe deployment. However,
traditional prompt-engineering approaches may prove ineffective once
red-teaming turns into a weak-to-strong problem, where target models surpass
red-teamers in capabilities. To study this shift, we frame red-teaming through
the lens of the capability gap between attacker and target. We evaluate more
than 500 attacker-target pairs using LLM-based jailbreak attacks that mimic
human red-teamers across diverse families, sizes, and capability levels. Three
strong trends emerge: (i) more capable models are better attackers, (ii) attack
success drops sharply once the target's capability exceeds the attacker's, and
(iii) attack success rates correlate with high performance on social science
splits of the MMLU-Pro benchmark. From these trends, we derive a jailbreaking
scaling law that predicts attack success for a fixed target based on
attacker-target capability gap. These findings suggest that fixed-capability
attackers (e.g., humans) may become ineffective against future models,
increasingly capable open-source models amplify risks for existing systems, and
model providers must accurately measure and control models' persuasive and
manipulative abilities to limit their effectiveness as attackers.


http://arxiv.org/abs/2505.19504
DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillation. (2%)
Pingzhi Li; Zhen Tan; Huaizhi Qu; Huan Liu; Tianlong Chen
  Large Language Models (LLMs) represent substantial intellectual and economic
investments, yet their effectiveness can inadvertently facilitate model
imitation via knowledge distillation (KD).In practical scenarios, competitors
can distill proprietary LLM capabilities by simply observing publicly
accessible outputs, akin to reverse-engineering a complex performance by
observation alone. Existing protective methods like watermarking only identify
imitation post-hoc, while other defenses assume the student model mimics the
teacher's internal logits, rendering them ineffective against distillation
purely from observed output text. This paper confronts the challenge of
actively protecting LLMs within the realistic constraints of API-based access.
We introduce an effective and efficient Defensive Output Generation (DOGe)
strategy that subtly modifies the output behavior of an LLM. Its outputs remain
accurate and useful for legitimate users, yet are designed to be misleading for
distillation, significantly undermining imitation attempts. We achieve this by
fine-tuning only the final linear layer of the teacher LLM with an adversarial
loss. This targeted training approach anticipates and disrupts distillation
attempts during inference time. Our experiments show that, while preserving or
even improving the original performance of the teacher model, student models
distilled from the defensively generated teacher outputs demonstrate
catastrophically reduced performance, demonstrating our method's effectiveness
as a practical safeguard against KD-based model imitation.


http://arxiv.org/abs/2505.19532
Fox in the Henhouse: Supply-Chain Backdoor Attacks Against Reinforcement Learning. (1%)
Shijie Liu; Andrew C. Cullen; Paul Montague; Sarah Erfani; Benjamin I. P. Rubinstein
  The current state-of-the-art backdoor attacks against Reinforcement Learning
(RL) rely upon unrealistically permissive access models, that assume the
attacker can read (or even write) the victim's policy parameters, observations,
or rewards. In this work, we question whether such a strong assumption is
required to launch backdoor attacks against RL. To answer this question, we
propose the \underline{S}upply-\underline{C}h\underline{a}in
\underline{B}ackdoor (SCAB) attack, which targets a common RL workflow:
training agents using external agents that are provided separately or embedded
within the environment. In contrast to prior works, our attack only relies on
legitimate interactions of the RL agent with the supplied agents. Despite this
limited access model, by poisoning a mere $3\%$ of training experiences, our
attack can successfully activate over $90\%$ of triggered actions, reducing the
average episodic return by $80\%$ for the victim. Our novel attack demonstrates
that RL attacks are likely to become a reality under untrusted RL training
supply-chains.


http://arxiv.org/abs/2505.19194
Curvature Dynamic Black-box Attack: revisiting adversarial robustness via dynamic curvature estimation. (99%)
Peiran Sun
  Adversarial attack reveals the vulnerability of deep learning models. For
about a decade, countless attack and defense methods have been proposed,
leading to robustified classifiers and better understanding of models. Among
these methods, curvature-based approaches have attracted attention because it
is assumed that high curvature may give rise to rough decision boundary.
However, the most commonly used \textit{curvature} is the curvature of loss
function, scores or other parameters from within the model as opposed to
decision boundary curvature, since the former can be relatively easily formed
using second order derivative. In this paper, we propose a new query-efficient
method, dynamic curvature estimation(DCE), to estimate the decision boundary
curvature in a black-box setting. Our approach is based on CGBA, a black-box
adversarial attack. By performing DCE on a wide range of classifiers, we
discovered, statistically, a connection between decision boundary curvature and
adversarial robustness. We also propose a new attack method, curvature dynamic
black-box attack(CDBA) with improved performance using the dynamically
estimated curvature.


http://arxiv.org/abs/2505.19459
Your Classifier Can Do More: Towards Bridging the Gaps in Classification, Robustness, and Generation. (86%)
Kaichao Jiang; He Wang; Xiaoshuai Hao; Xiulong Yang; Ajian Liu; Qi Chu; Yunfeng Diao
  Joint Energy-based Models (JEMs), a class of hybrid generative-discriminative
models, are well known for their ability to achieve both high classification
accuracy and generative capability within a single model. However, their
robustness still lags significantly behind the classifiers based adversarial
training (AT). Conversely, while AT is currently the most effective approach to
improving the classifier's robustness, it typically sacrifices accuracy on
clean data and lacks generative capability. The triple trade-off between
classification accuracy, generative capability and robustness, raises a natural
question: Can a single model simultaneously achieve high classification
accuracy, adversarial robustness, and generative performance? -- a goal that
has been rarely explored. To address this question, we systematically analyze
the energy distribution differences of clean, adversarial, and generated
samples across various JEM variants and adversarially trained models. We
observe that AT tends to reduce the energy gap between clean and adversarial
samples, while JEMs reduce the gap between clean and synthetic ones. This
observation suggests a key insight: if the energy distributions of all three
data types can be aligned, we might unify the strengths of AT and JEMs,
resolving their inherent trade-offs. Building on this idea, we propose
Energy-based Joint Distribution Adversarial Training (EB-JDAT), to jointly
model the clean data distribution, the adversarial distribution, and the
classifier by maximizing their joint probability. EB-JDAT is a general and
flexible optimization method, compatible with various JEM variants. Extensive
experimental results demonstrate that EB-JDAT not only maintains near original
accuracy and generative capability of JEMs, but also significantly enhances
robustness, even surpassing state-of-the-art ATs.


http://arxiv.org/abs/2505.19397
Are Time-Series Foundation Models Deployment-Ready? A Systematic Study of Adversarial Robustness Across Domains. (78%)
Jiawen Zhang; Zhenwei Zhang; Shun Zheng; Xumeng Wen; Jia Li; Jiang Bian
  Time Series Foundation Models (TSFMs), which are pretrained on large-scale,
cross-domain data and capable of zero-shot forecasting in new scenarios without
further training, are increasingly adopted in real-world applications. However,
as the zero-shot forecasting paradigm gets popular, a critical yet overlooked
question emerges: Are TSFMs robust to adversarial input perturbations? Such
perturbations could be exploited in man-in-the-middle attacks or data
poisoning. To address this gap, we conduct a systematic investigation into the
adversarial robustness of TSFMs. Our results show that even minimal
perturbations can induce significant and controllable changes in forecast
behaviors, including trend reversal, temporal drift, and amplitude shift,
posing serious risks to TSFM-based services. Through experiments on
representative TSFMs and multiple datasets, we reveal their consistent
vulnerabilities and identify potential architectural designs, such as
structural sparsity and multi-task pretraining, that may improve robustness.
Our findings offer actionable guidance for designing more resilient forecasting
systems and provide a critical assessment of the adversarial robustness of
TSFMs.


http://arxiv.org/abs/2505.19364
RADEP: A Resilient Adaptive Defense Framework Against Model Extraction Attacks. (75%)
Amit Chakraborty; Sayyed Farid Ahamed; Sandip Roy; Soumya Banerjee; Kevin Choi; Abdul Rahman; Alison Hu; Edward Bowen; Sachin Shetty
  Machine Learning as a Service (MLaaS) enables users to leverage powerful
machine learning models through cloud-based APIs, offering scalability and ease
of deployment. However, these services are vulnerable to model extraction
attacks, where adversaries repeatedly query the application programming
interface (API) to reconstruct a functionally similar model, compromising
intellectual property and security. Despite various defense strategies being
proposed, many suffer from high computational costs, limited adaptability to
evolving attack techniques, and a reduction in performance for legitimate
users. In this paper, we introduce a Resilient Adaptive Defense Framework for
Model Extraction Attack Protection (RADEP), a multifaceted defense framework
designed to counteract model extraction attacks through a multi-layered
security approach. RADEP employs progressive adversarial training to enhance
model resilience against extraction attempts. Malicious query detection is
achieved through a combination of uncertainty quantification and behavioral
pattern analysis, effectively identifying adversarial queries. Furthermore, we
develop an adaptive response mechanism that dynamically modifies query outputs
based on their suspicion scores, reducing the utility of stolen models.
Finally, ownership verification is enforced through embedded watermarking and
backdoor triggers, enabling reliable identification of unauthorized model use.
Experimental evaluations demonstrate that RADEP significantly reduces
extraction success rates while maintaining high detection accuracy with minimal
impact on legitimate queries. Extensive experiments show that RADEP effectively
defends against model extraction attacks and remains resilient even against
adaptive adversaries, making it a reliable security framework for MLaaS models.


http://arxiv.org/abs/2505.19425
Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation. (13%)
Yuhao He; Jinyu Tian; Haiwei Wu; Jianqing Li
  The rapid advancement of diffusion models has enhanced their image inpainting
and editing capabilities but also introduced significant societal risks.
Adversaries can exploit user images from social media to generate misleading or
harmful content. While adversarial perturbations can disrupt inpainting, global
perturbation-based methods fail in mask-guided editing tasks due to spatial
constraints. To address these challenges, we propose Structure Disruption
Attack (SDA), a powerful protection framework for safeguarding sensitive image
regions against inpainting-based editing. Building upon the contour-focused
nature of self-attention mechanisms of diffusion models, SDA optimizes
perturbations by disrupting queries in self-attention during the initial
denoising step to destroy the contour generation process. This targeted
interference directly disrupts the structural generation capability of
diffusion models, effectively preventing them from producing coherent images.
We validate our motivation through visualization techniques and extensive
experiments on public datasets, demonstrating that SDA achieves
state-of-the-art (SOTA) protection performance while maintaining strong
robustness.


http://arxiv.org/abs/2505.23791
Evaluating Query Efficiency and Accuracy of Transfer Learning-based Model Extraction Attack in Federated Learning. (10%)
Sayyed Farid Ahamed; Sandip Roy; Soumya Banerjee; Marc Vucovich; Kevin Choi; Abdul Rahman; Alison Hu; Edward Bowen; Sachin Shetty
  Federated Learning (FL) is a collaborative learning framework designed to
protect client data, yet it remains highly vulnerable to Intellectual Property
(IP) threats. Model extraction (ME) attacks pose a significant risk to Machine
Learning as a Service (MLaaS) platforms, enabling attackers to replicate
confidential models by querying black-box (without internal insight) APIs.
Despite FL's privacy-preserving goals, its distributed nature makes it
particularly susceptible to such attacks. This paper examines the vulnerability
of FL-based victim models to two types of model extraction attacks. For various
federated clients built under the NVFlare platform, we implemented ME attacks
across two deep learning architectures and three image datasets. We evaluate
the proposed ME attack performance using various metrics, including accuracy,
fidelity, and KL divergence. The experiments show that for different FL
clients, the accuracy and fidelity of the extracted model are closely related
to the size of the attack query set. Additionally, we explore a transfer
learning based approach where pretrained models serve as the starting point for
the extraction process. The results indicate that the accuracy and fidelity of
the fine-tuned pretrained extraction models are notably higher, particularly
with smaller query sets, highlighting potential advantages for attackers.


http://arxiv.org/abs/2505.19338
Co-evolutionary Dynamics of Attack and Defence in Cybersecurity. (8%)
Adeela Bashir; Zia Ush Shamszaman; Zhao Song; The Anh Han
  In the evolving digital landscape, it is crucial to study the dynamics of
cyberattacks and defences. This study uses an Evolutionary Game Theory (EGT)
framework to investigate the evolutionary dynamics of attacks and defences in
cyberspace. We develop a two-population asymmetric game between attacker and
defender to capture the essential factors of costs, potential benefits, and the
probability of successful defences. Through mathematical analysis and numerical
simulations, we find that systems with high defence intensities show stability
with minimal attack frequencies, whereas low-defence environments show
instability, and are vulnerable to attacks. Furthermore, we find five
equilibria, where the strategy pair always defend and attack emerged as the
most likely stable state as cyber domain is characterised by a continuous
battle between defenders and attackers. Our theoretical findings align with
real-world data from past cyber incidents, demonstrating the interdisciplinary
impact, such as fraud detection, risk management and cybersecurity
decision-making. Overall, our analysis suggests that adaptive cybersecurity
strategies based on EGT can improve resource allocation, enhance system
resilience, and reduce the overall risk of cyberattacks. By incorporating
real-world data, this study demonstrates the applicability of EGT in addressing
the evolving nature of cyber threats and the need for secure digital ecosystems
through strategic planning and proactive defence measures.


http://arxiv.org/abs/2505.19119
CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning. (2%)
Renyuan Li; Zhibo Liang; Haichuan Zhang; Tianyu Shi; Zhiyuan Cheng; Jia Shi; Carl Yang; Mingjie Tang
  Recent breakthroughs in text-to-speech (TTS) voice cloning have raised
serious privacy concerns, allowing highly accurate vocal identity replication
from just a few seconds of reference audio, while retaining the speaker's vocal
authenticity. In this paper, we introduce CloneShield, a universal time-domain
adversarial perturbation framework specifically designed to defend against
zero-shot voice cloning. Our method provides protection that is robust across
speakers and utterances, without requiring any prior knowledge of the
synthesized text. We formulate perturbation generation as a multi-objective
optimization problem, and propose Multi-Gradient Descent Algorithm (MGDA) to
ensure the robust protection across diverse utterances. To preserve natural
auditory perception for users, we decompose the adversarial perturbation via
Mel-spectrogram representations and fine-tune it for each sample. This design
ensures imperceptibility while maintaining strong degradation effects on
zero-shot cloned outputs. Experiments on three state-of-the-art zero-shot TTS
systems, five benchmark datasets and evaluations from 60 human listeners
demonstrate that our method preserves near-original audio quality in protected
inputs (PESQ = 3.90, SRS = 0.93) while substantially degrading both speaker
similarity and speech quality in cloned samples (PESQ = 1.07, SRS = 0.08).


http://arxiv.org/abs/2505.18583
The Silent Saboteur: Imperceptible Adversarial Attacks against Black-Box Retrieval-Augmented Generation Systems. (99%)
Hongru Song; Yu-an Liu; Ruqing Zhang; Jiafeng Guo; Jianming Lv; Rijke Maarten de; Xueqi Cheng
  We explore adversarial attacks against retrieval-augmented generation (RAG)
systems to identify their vulnerabilities. We focus on generating
human-imperceptible adversarial examples and introduce a novel imperceptible
retrieve-to-generate attack against RAG. This task aims to find imperceptible
perturbations that retrieve a target document, originally excluded from the
initial top-$k$ candidate set, in order to influence the final answer
generation. To address this task, we propose ReGENT, a reinforcement
learning-based framework that tracks interactions between the attacker and the
target RAG and continuously refines attack strategies based on
relevance-generation-naturalness rewards. Experiments on newly constructed
factual and non-factual question-answering benchmarks demonstrate that ReGENT
significantly outperforms existing attack methods in misleading RAG systems
with small imperceptible text perturbations.


http://arxiv.org/abs/2505.18864
Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Framework. (99%)
Binhao Ma; Hanqing Guo; Zhengping Jay Luo; Rui Duan
  Recent advances in Multimodal Large Language Models (MLLMs) have
significantly enhanced the naturalness and flexibility of human computer
interaction by enabling seamless understanding across text, vision, and audio
modalities. Among these, voice enabled models such as SpeechGPT have
demonstrated considerable improvements in usability, offering expressive, and
emotionally responsive interactions that foster deeper connections in real
world communication scenarios. However, the use of voice introduces new
security risks, as attackers can exploit the unique characteristics of spoken
language, such as timing, pronunciation variability, and speech to text
translation, to craft inputs that bypass defenses in ways not seen in
text-based systems. Despite substantial research on text based jailbreaks, the
voice modality remains largely underexplored in terms of both attack strategies
and defense mechanisms. In this work, we present an adversarial attack
targeting the speech input of aligned MLLMs in a white box scenario.
Specifically, we introduce a novel token level attack that leverages access to
the model's speech tokenization to generate adversarial token sequences. These
sequences are then synthesized into audio prompts, which effectively bypass
alignment safeguards and to induce prohibited outputs. Evaluated on SpeechGPT,
our approach achieves up to 89 percent attack success rate across multiple
restricted tasks, significantly outperforming existing voice based jailbreak
methods. Our findings shed light on the vulnerabilities of voice-enabled
multimodal systems and to help guide the development of more robust
next-generation MLLMs.


http://arxiv.org/abs/2505.18884
LORE: Lagrangian-Optimized Robust Embeddings for Visual Encoders. (93%)
Borna Khodabandeh; Amirabbas Afzali; Amirhossein Afsharrad; Seyed Shahabeddin Mousavi; Sanjay Lall; Sajjad Amini; Seyed-Mohsen Moosavi-Dezfooli
  Visual encoders have become fundamental components in modern computer vision
pipelines. However, ensuring robustness against adversarial perturbations
remains a critical challenge. Recent efforts have explored both supervised and
unsupervised adversarial fine-tuning strategies. We identify two key
limitations in these approaches: (i) they often suffer from instability,
especially during the early stages of fine-tuning, resulting in suboptimal
convergence and degraded performance on clean data, and (ii) they exhibit a
suboptimal trade-off between robustness and clean data accuracy, hindering the
simultaneous optimization of both objectives. To overcome these challenges, we
propose Lagrangian-Optimized Robust Embeddings (LORE), a novel unsupervised
adversarial fine-tuning framework. LORE utilizes constrained optimization,
which offers a principled approach to balancing competing goals, such as
improving robustness while preserving nominal performance. By enforcing
embedding-space proximity constraints, LORE effectively maintains clean data
performance throughout adversarial fine-tuning. Extensive experiments show that
LORE significantly improves zero-shot adversarial robustness with minimal
degradation in clean data accuracy. Furthermore, we demonstrate the
effectiveness of the adversarially fine-tuned CLIP image encoder in
out-of-distribution generalization and enhancing the interpretability of image
embeddings.


http://arxiv.org/abs/2505.18806
Mal-D2GAN: Double-Detector based GAN for Malware Generation. (92%)
Nam Hoang Thanh; Trung Pham Duy; Lam Bui Thu
  Machine learning (ML) has been developed to detect malware in recent years.
Most researchers focused their efforts on improving the detection performance
but ignored the robustness of the ML models. In addition, many machine learning
algorithms are very vulnerable to intentional attacks. To solve these problems,
adversarial malware examples are generated by GANs to enhance the robustness of
the malware detector. However, since current GAN models suffer from limitations
such as unstable training and weak adversarial examples, we propose the
Mal-D2GAN model to address these problems. Specifically, the Mal-D2GAN
architecture was designed with double-detector and a least square loss function
and tested on a dataset of 20,000 samples. The results show that the Mal-D2GAN
model reduced the detection accuracy (true positive rate) in 8 malware
detectors. The performance was then compared with that of the existing MalGAN
and Mal- LSGAN models.


http://arxiv.org/abs/2505.18766
StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations. (68%)
Yanjie Li; Wenxuan Zhang; Xinqi Lyu; Yihao Liu; Bin Xiao
  Recently, text-to-image diffusion models have been widely used for style
mimicry and personalized customization through methods such as DreamBooth and
Textual Inversion. This has raised concerns about intellectual property
protection and the generation of deceptive content. Recent studies, such as
Glaze and Anti-DreamBooth, have proposed using adversarial noise to protect
images from these attacks. However, recent purification-based methods, such as
DiffPure and Noise Upscaling, have successfully attacked these latest defenses,
showing the vulnerabilities of these methods. Moreover, present methods show
limited transferability across models, making them less effective against
unknown text-to-image models. To address these issues, we propose a novel
anti-mimicry method, StyleGuard. We propose a novel style loss that optimizes
the style-related features in the latent space to make it deviate from the
original image, which improves model-agnostic transferability. Additionally, to
enhance the perturbation's ability to bypass diffusion-based purification, we
designed a novel upscale loss that involves ensemble purifiers and upscalers
during training. Extensive experiments on the WikiArt and CelebA datasets
demonstrate that StyleGuard outperforms existing methods in robustness against
various transformations and purifications, effectively countering style mimicry
in various models. Moreover, StyleGuard is effective on different style mimicry
methods, including DreamBooth and Textual Inversion.


http://arxiv.org/abs/2505.18543
Benchmarking Poisoning Attacks against Retrieval-Augmented Generation. (54%)
Baolei Zhang; Haoran Xin; Jiatong Li; Dongzhe Zhang; Minghong Fang; Zhuqing Liu; Lihai Nie; Zheli Liu
  Retrieval-Augmented Generation (RAG) has proven effective in mitigating
hallucinations in large language models by incorporating external knowledge
during inference. However, this integration introduces new security
vulnerabilities, particularly to poisoning attacks. Although prior work has
explored various poisoning strategies, a thorough assessment of their practical
threat to RAG systems remains missing. To address this gap, we propose the
first comprehensive benchmark framework for evaluating poisoning attacks on
RAG. Our benchmark covers 5 standard question answering (QA) datasets and 10
expanded variants, along with 13 poisoning attack methods and 7 defense
mechanisms, representing a broad spectrum of existing techniques. Using this
benchmark, we conduct a comprehensive evaluation of all included attacks and
defenses across the full dataset spectrum. Our findings show that while
existing attacks perform well on standard QA datasets, their effectiveness
drops significantly on the expanded versions. Moreover, our results demonstrate
that various advanced RAG architectures, such as sequential, branching,
conditional, and loop RAG, as well as multi-turn conversational RAG, multimodal
RAG systems, and RAG-based LLM agent systems, remain susceptible to poisoning
attacks. Notably, current defense techniques fail to provide robust protection,
underscoring the pressing need for more resilient and generalizable defense
strategies.


http://arxiv.org/abs/2505.23786
Mind the Gap: A Practical Attack on GGUF Quantization. (15%)
Kazuki Egashira; Robin Staab; Mark Vero; Jingxuan He; Martin Vechev
  With the increasing size of frontier LLMs, post-training quantization has
become the standard for memory-efficient deployment. Recent work has shown that
basic rounding-based quantization schemes pose security risks, as they can be
exploited to inject malicious behaviors into quantized models that remain
hidden in full precision. However, existing attacks cannot be applied to more
complex quantization methods, such as the GGUF family used in the popular
ollama and llama.cpp frameworks. In this work, we address this gap by
introducing the first attack on GGUF. Our key insight is that the quantization
error -- the difference between the full-precision weights and their
(de-)quantized version -- provides sufficient flexibility to construct
malicious quantized models that appear benign in full precision. Leveraging
this, we develop an attack that trains the target malicious LLM while
constraining its weights based on quantization errors. We demonstrate the
effectiveness of our attack on three popular LLMs across nine GGUF quantization
data types on three diverse attack scenarios: insecure code generation
($\Delta$=$88.7\%$), targeted content injection ($\Delta$=$85.0\%$), and benign
instruction refusal ($\Delta$=$30.1\%$). Our attack highlights that (1) the
most widely used post-training quantization method is susceptible to
adversarial interferences, and (2) the complexity of quantization schemes alone
is insufficient as a defense.


http://arxiv.org/abs/2505.18773
Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models. (13%)
Jamie Hayes; Ilia Shumailov; Christopher A. Choquette-Choo; Matthew Jagielski; George Kaissis; Katherine Lee; Milad Nasr; Sahra Ghalebikesabi; Niloofar Mireshghallah; Meenatchi Sundaram Mutu Selva Annamalai; Igor Shilov; Matthieu Meeus; Montjoye Yves-Alexandre de; Franziska Boenisch; Adam Dziedzic; A. Feder Cooper
  State-of-the-art membership inference attacks (MIAs) typically require
training many reference models, making it difficult to scale these attacks to
large pre-trained language models (LLMs). As a result, prior research has
either relied on weaker attacks that avoid training reference models (e.g.,
fine-tuning attacks), or on stronger attacks applied to small-scale models and
datasets. However, weaker attacks have been shown to be brittle - achieving
close-to-arbitrary success - and insights from strong attacks in simplified
settings do not translate to today's LLMs. These challenges have prompted an
important question: are the limitations observed in prior work due to attack
design choices, or are MIAs fundamentally ineffective on LLMs? We address this
question by scaling LiRA - one of the strongest MIAs - to GPT-2 architectures
ranging from 10M to 1B parameters, training reference models on over 20B tokens
from the C4 dataset. Our results advance the understanding of MIAs on LLMs in
three key ways: (1) strong MIAs can succeed on pre-trained LLMs; (2) their
effectiveness, however, remains limited (e.g., AUC<0.7) in practical settings;
and, (3) the relationship between MIA success and related privacy metrics is
not as straightforward as prior work has suggested.


http://arxiv.org/abs/2505.18889
Security Concerns for Large Language Models: A Survey. (13%)
Miles Q. Li; Benjamin C. M. Fung
  Large Language Models (LLMs) such as GPT-4 and its recent iterations,
Google's Gemini, Anthropic's Claude 3 models, and xAI's Grok have caused a
revolution in natural language processing, but their capabilities also
introduce new security vulnerabilities. In this survey, we provide a
comprehensive overview of the emerging security concerns around LLMs,
categorizing threats into prompt injection and jailbreaking, adversarial
attacks such as input perturbations and data poisoning, misuse by malicious
actors for purposes such as generating disinformation, phishing emails, and
malware, and worrisome risks inherent in autonomous LLM agents. A significant
focus has been recently placed on the latter, exploring goal misalignment,
emergent deception, self-preservation instincts, and the potential for LLMs to
develop and pursue covert, misaligned objectives, a behavior known as scheming,
which may even persist through safety training. We summarize recent academic
and industrial studies from 2022 to 2025 that exemplify each threat, analyze
proposed defenses and their limitations, and identify open challenges in
securing LLM-based applications. We conclude by emphasizing the importance of
advancing robust, multi-layered security strategies to ensure LLMs are safe and
beneficial.


http://arxiv.org/abs/2505.18556
Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation. (3%)
Jun Zhuang; Haibo Jin; Ye Zhang; Zhengjian Kang; Wenbin Zhang; Gaby G. Dagher; Haohan Wang
  Intent detection, a core component of natural language understanding, has
considerably evolved as a crucial mechanism in safeguarding large language
models (LLMs). While prior work has applied intent detection to enhance LLMs'
moderation guardrails, showing a significant success against content-level
jailbreaks, the robustness of these intent-aware guardrails under malicious
manipulations remains under-explored. In this work, we investigate the
vulnerability of intent-aware guardrails and demonstrate that LLMs exhibit
implicit intent detection capabilities. We propose a two-stage intent-based
prompt-refinement framework, IntentPrompt, that first transforms harmful
inquiries into structured outlines and further reframes them into
declarative-style narratives by iteratively optimizing prompts via feedback
loops to enhance jailbreak success for red-teaming purposes. Extensive
experiments across four public benchmarks and various black-box LLMs indicate
that our framework consistently outperforms several cutting-edge jailbreak
methods and evades even advanced Intent Analysis (IA) and Chain-of-Thought
(CoT)-based defenses. Specifically, our "FSTR+SPIN" variant achieves attack
success rates ranging from 88.25% to 96.54% against CoT-based defenses on the
o1 model, and from 86.75% to 97.12% on the GPT-4o model under IA-based
defenses. These findings highlight a critical weakness in LLMs' safety
mechanisms and suggest that intent manipulation poses a growing challenge to
content moderation guardrails.


http://arxiv.org/abs/2505.18551
LAMDA: A Longitudinal Android Malware Benchmark for Concept Drift Analysis. (3%)
Md Ahsanul Haque; Ismail Hossain; Md Mahmuduzzaman Kamol; Md Jahangir Alam; Suresh Kumar Amalapuram; Sajedul Talukder; Mohammad Saidur Rahman
  Machine learning (ML)-based malware detection systems often fail to account
for the dynamic nature of real-world training and test data distributions. In
practice, these distributions evolve due to frequent changes in the Android
ecosystem, adversarial development of new malware families, and the continuous
emergence of both benign and malicious applications. Prior studies have shown
that such concept drift -- distributional shifts in benign and malicious
samples, leads to significant degradation in detection performance over time.
Despite the practical importance of this issue, existing datasets are often
outdated and limited in temporal scope, diversity of malware families, and
sample scale, making them insufficient for the systematic evaluation of concept
drift in malware detection.
  To address this gap, we present LAMDA, the largest and most temporally
diverse Android malware benchmark to date, designed specifically for concept
drift analysis. LAMDA spans 12 years (2013-2025, excluding 2015), includes over
1 million samples (approximately 37% labeled as malware), and covers 1,380
malware families and 150,000 singleton samples, reflecting the natural
distribution and evolution of real-world Android applications. We empirically
demonstrate LAMDA's utility by quantifying the performance degradation of
standard ML models over time and analyzing feature stability across years. As
the most comprehensive Android malware dataset to date, LAMDA enables in-depth
research into temporal drift, generalization, explainability, and evolving
detection challenges. The dataset and code are available at:
https://iqsec-lab.github.io/LAMDA/.


http://arxiv.org/abs/2505.18680
$PD^3F$: A Pluggable and Dynamic DoS-Defense Framework Against Resource Consumption Attacks Targeting Large Language Models. (2%)
Yuanhe Zhang; Xinyue Wang; Haoran Gao; Zhenhong Zhou; Fanyu Meng; Yuyao Zhang; Sen Su
  Large Language Models (LLMs), due to substantial computational requirements,
are vulnerable to resource consumption attacks, which can severely degrade
server performance or even cause crashes, as demonstrated by denial-of-service
(DoS) attacks designed for LLMs. However, existing works lack mitigation
strategies against such threats, resulting in unresolved security risks for
real-world LLM deployments. To this end, we propose the Pluggable and Dynamic
DoS-Defense Framework ($PD^3F$), which employs a two-stage approach to defend
against resource consumption attacks from both the input and output sides. On
the input side, we propose the Resource Index to guide Dynamic Request Polling
Scheduling, thereby reducing resource usage induced by malicious attacks under
high-concurrency scenarios. On the output side, we introduce the Adaptive
End-Based Suppression mechanism, which terminates excessive malicious
generation early. Experiments across six models demonstrate that $PD^3F$
significantly mitigates resource consumption attacks, improving users' access
capacity by up to 500% during adversarial load. $PD^3F$ represents a step
toward the resilient and resource-aware deployment of LLMs against resource
consumption attacks.


http://arxiv.org/abs/2505.17509
Enhancing Adversarial Robustness of Vision Language Models via Adversarial Mixture Prompt Tuning. (99%)
Shiji Zhao; Qihui Zhu; Shukun Xiong; Shouwei Ruan; Yize Fan; Ranjie Duan; Qing Guo; Xingxing Wei
  Large pre-trained Vision Language Models (VLMs) have excellent generalization
capabilities but are highly susceptible to adversarial examples, presenting
potential security risks. To improve the robustness of VLMs against adversarial
examples, adversarial prompt tuning methods are proposed to align the text
feature with the adversarial image feature without changing model parameters.
However, when facing various adversarial attacks, a single learnable text
prompt has insufficient generalization to align well with all adversarial image
features, which finally leads to the overfitting phenomenon. To address the
above challenge, in this paper, we empirically find that increasing the number
of learned prompts can bring more robustness improvement than a longer prompt.
Then we propose an adversarial tuning method named Adversarial Mixture Prompt
Tuning (AMPT) to enhance the generalization towards various adversarial attacks
for VLMs. AMPT aims to learn mixture text prompts to obtain more robust text
features. To further enhance the adaptability, we propose a conditional weight
router based on the input adversarial image to predict the mixture weights of
multiple learned prompts, which helps obtain sample-specific aggregated text
features aligning with different adversarial image features. A series of
experiments show that our method can achieve better adversarial robustness than
state-of-the-art methods on 11 datasets under different experimental settings.


http://arxiv.org/abs/2505.18097
Towards more transferable adversarial attack in black-box manner. (99%)
Chun Tong Lei; Zhongliang Guo; Hon Chung Lee; Minh Quoc Duong; Chun Pong Lau
  Adversarial attacks have become a well-explored domain, frequently serving as
evaluation baselines for model robustness. Among these, black-box attacks based
on transferability have received significant attention due to their practical
applicability in real-world scenarios. Traditional black-box methods have
generally focused on improving the optimization framework (e.g., utilizing
momentum in MI-FGSM) to enhance transferability, rather than examining the
dependency on surrogate white-box model architectures. Recent state-of-the-art
approach DiffPGD has demonstrated enhanced transferability by employing
diffusion-based adversarial purification models for adaptive attacks. The
inductive bias of diffusion-based adversarial purification aligns naturally
with the adversarial attack process, where both involving noise addition,
reducing dependency on surrogate white-box model selection. However, the
denoising process of diffusion models incurs substantial computational costs
through chain rule derivation, manifested in excessive VRAM consumption and
extended runtime. This progression prompts us to question whether introducing
diffusion models is necessary. We hypothesize that a model sharing similar
inductive bias to diffusion-based adversarial purification, combined with an
appropriate loss function, could achieve comparable or superior transferability
while dramatically reducing computational overhead. In this paper, we propose a
novel loss function coupled with a unique surrogate model to validate our
hypothesis. Our approach leverages the score of the time-dependent classifier
from classifier-guided diffusion models, effectively incorporating natural data
distribution knowledge into the adversarial optimization process. Experimental
results demonstrate significantly improved transferability across diverse model
architectures while maintaining robustness against diffusion-based defenses.


http://arxiv.org/abs/2505.17579
Ownership Verification of DNN Models Using White-Box Adversarial Attacks with Specified Probability Manipulation. (99%)
Teruki Sano; Minoru Kuribayashi; Masao Sakai; Shuji Ishobe; Eisuke Koizumi
  In this paper, we propose a novel framework for ownership verification of
deep neural network (DNN) models for image classification tasks. It allows
verification of model identity by both the rightful owner and third party
without presenting the original model. We assume a gray-box scenario where an
unauthorized user owns a model that is illegally copied from the original
model, provides services in a cloud environment, and the user throws images and
receives the classification results as a probability distribution of output
classes. The framework applies a white-box adversarial attack to align the
output probability of a specific class to a designated value. Due to the
knowledge of original model, it enables the owner to generate such adversarial
examples. We propose a simple but effective adversarial attack method based on
the iterative Fast Gradient Sign Method (FGSM) by introducing control
parameters. Experimental results confirm the effectiveness of the
identification of DNN models using adversarial attack.


http://arxiv.org/abs/2505.17807
Temporal Consistency Constrained Transferable Adversarial Attacks with Background Mixup for Action Recognition. (99%)
Ping Li; Jianan Ni; Bo Pang
  Action recognition models using deep learning are vulnerable to adversarial
examples, which are transferable across other models trained on the same data
modality. Existing transferable attack methods face two major challenges: 1)
they heavily rely on the assumption that the decision boundaries of the
surrogate (a.k.a., source) model and the target model are similar, which limits
the adversarial transferability; and 2) their decision boundary difference
makes the attack direction uncertain, which may result in the gradient
oscillation, weakening the adversarial attack. This motivates us to propose a
Background Mixup-induced Temporal Consistency (BMTC) attack method for action
recognition. From the input transformation perspective, we design a
model-agnostic background adversarial mixup module to reduce the
surrogate-target model dependency. In particular, we randomly sample one video
from each category and make its background frame, while selecting the
background frame with the top attack ability for mixup with the clean frame by
reinforcement learning. Moreover, to ensure an explicit attack direction, we
leverage the background category as guidance for updating the gradient of
adversarial example, and design a temporal gradient consistency loss, which
strengthens the stability of the attack direction on subsequent frames.
Empirical studies on two video datasets, i.e., UCF101 and Kinetics-400, and one
image dataset, i.e., ImageNet, demonstrate that our method significantly boosts
the transferability of adversarial examples across several action/image
recognition models. Our code is available at
https://github.com/mlvccn/BMTC_TransferAttackVid.


http://arxiv.org/abs/2505.17598
One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs. (97%)
Linbao Li; Yannan Liu; Daojing He; Yu Li
  Safety alignment in large language models (LLMs) is increasingly compromised
by jailbreak attacks, which can manipulate these models to generate harmful or
unintended content. Investigating these attacks is crucial for uncovering model
vulnerabilities. However, many existing jailbreak strategies fail to keep pace
with the rapid development of defense mechanisms, such as defensive suffixes,
rendering them ineffective against defended models. To tackle this issue, we
introduce a novel attack method called ArrAttack, specifically designed to
target defended LLMs. ArrAttack automatically generates robust jailbreak
prompts capable of bypassing various defense measures. This capability is
supported by a universal robustness judgment model that, once trained, can
perform robustness evaluation for any target model with a wide variety of
defenses. By leveraging this model, we can rapidly develop a robust jailbreak
prompt generator that efficiently converts malicious input prompts into
effective attacks. Extensive evaluations reveal that ArrAttack significantly
outperforms existing attack strategies, demonstrating strong transferability
across both white-box and black-box models, including GPT-4 and Claude-3. Our
work bridges the gap between jailbreak attacks and defenses, providing a fresh
perspective on generating robust jailbreak prompts. We make the codebase
available at https://github.com/LLBao/ArrAttack.


http://arxiv.org/abs/2505.18333
A Critical Evaluation of Defenses against Prompt Injection Attacks. (69%)
Yuqi Jia; Zedian Shao; Yupei Liu; Jinyuan Jia; Dawn Song; Neil Zhenqiang Gong
  Large Language Models (LLMs) are vulnerable to prompt injection attacks, and
several defenses have recently been proposed, often claiming to mitigate these
attacks successfully. However, we argue that existing studies lack a principled
approach to evaluating these defenses. In this paper, we argue the need to
assess defenses across two critical dimensions: (1) effectiveness, measured
against both existing and adaptive prompt injection attacks involving diverse
target and injected prompts, and (2) general-purpose utility, ensuring that the
defense does not compromise the foundational capabilities of the LLM. Our
critical evaluation reveals that prior studies have not followed such a
comprehensive evaluation methodology. When assessed using this principled
approach, we show that existing defenses are not as successful as previously
reported. This work provides a foundation for evaluating future defenses and
guiding their development. Our code and data are available at:
https://github.com/PIEval123/PIEval.


http://arxiv.org/abs/2505.17519
Chain-of-Lure: A Synthetic Narrative-Driven Approach to Compromise Large Language Models. (67%)
Wenhan Chang; Tianqing Zhu; Yu Zhao; Shuangyong Song; Ping Xiong; Wanlei Zhou; Yongxiang Li
  In the era of rapid generative AI development, interactions between humans
and large language models face significant misusing risks. Previous research
has primarily focused on black-box scenarios using human-guided prompts and
white-box scenarios leveraging gradient-based LLM generation methods,
neglecting the possibility that LLMs can act not only as victim models, but
also as attacker models to harm other models. We proposes a novel jailbreaking
method inspired by the Chain-of-Thought mechanism, where the attacker model
uses mission transfer to conceal harmful user intent in dialogue and generates
chained narrative lures to stimulate the reasoning capabilities of victim
models, leading to successful jailbreaking. To enhance the attack success rate,
we introduce a helper model that performs random narrative optimization on the
narrative lures during multi-turn dialogues while ensuring alignment with the
original intent, enabling the optimized lures to bypass the safety barriers of
victim models effectively. Our experiments reveal that models with weaker
safety mechanisms exhibit stronger attack capabilities, demonstrating that
models can not only be exploited, but also help harm others. By incorporating
toxicity scores, we employ third-party models to evaluate the harmfulness of
victim models' responses to jailbreaking attempts. The study shows that using
refusal keywords as an evaluation metric for attack success rates is
significantly flawed because it does not assess whether the responses guide
harmful questions, while toxicity scores measure the harm of generated content
with more precision and its alignment with harmful questions. Our approach
demonstrates outstanding performance, uncovering latent vulnerabilities in LLMs
and providing data-driven feedback to optimize LLM safety mechanisms. We also
discuss two defensive strategies to offer guidance on improving defense
mechanisms.


http://arxiv.org/abs/2505.17513
What You Read Isn't What You Hear: Linguistic Sensitivity in Deepfake Speech Detection. (56%)
Binh Nguyen; Shuji Shi; Ryan Ofman; Thai Le
  Recent advances in text-to-speech technologies have enabled realistic voice
generation, fueling audio-based deepfake attacks such as fraud and
impersonation. While audio anti-spoofing systems are critical for detecting
such threats, prior work has predominantly focused on acoustic-level
perturbations, leaving the impact of linguistic variation largely unexplored.
In this paper, we investigate the linguistic sensitivity of both open-source
and commercial anti-spoofing detectors by introducing transcript-level
adversarial attacks. Our extensive evaluation reveals that even minor
linguistic perturbations can significantly degrade detection accuracy: attack
success rates surpass 60% on several open-source detector-voice pairs, and
notably one commercial detection accuracy drops from 100% on synthetic audio to
just 32%. Through a comprehensive feature attribution analysis, we identify
that both linguistic complexity and model-level audio embedding similarity
contribute strongly to detector vulnerability. We further demonstrate the
real-world risk via a case study replicating the Brad Pitt audio deepfake scam,
using transcript adversarial attacks to completely bypass commercial detectors.
These results highlight the need to move beyond purely acoustic defenses and
account for linguistic variation in the design of robust anti-spoofing systems.
All source code will be publicly available.


http://arxiv.org/abs/2505.17601
Wolf Hidden in Sheep's Conversations: Toward Harmless Data-Based Backdoor Attacks for Jailbreaking Large Language Models. (31%)
Jiawei Kong; Hao Fang; Xiaochen Yang; Kuofeng Gao; Bin Chen; Shu-Tao Xia; Yaowei Wang; Min Zhang
  Supervised fine-tuning (SFT) aligns large language models (LLMs) with human
intent by training them on labeled task-specific data. Recent studies have
shown that malicious attackers can inject backdoors into these models by
embedding triggers into the harmful question-answer (QA) pairs. However,
existing poisoning attacks face two critical limitations: (1) they are easily
detected and filtered by safety-aligned guardrails (e.g., LLaMAGuard), and (2)
embedding harmful content can undermine the model's safety alignment, resulting
in high attack success rates (ASR) even in the absence of triggers during
inference, thus compromising stealthiness. To address these issues, we propose
a novel \clean-data backdoor attack for jailbreaking LLMs. Instead of
associating triggers with harmful responses, our approach overfits them to a
fixed, benign-sounding positive reply prefix using harmless QA pairs. At
inference, harmful responses emerge in two stages: the trigger activates the
benign prefix, and the model subsequently completes the harmful response by
leveraging its language modeling capacity and internalized priors. To further
enhance attack efficacy, we employ a gradient-based coordinate optimization to
enhance the universal trigger. Extensive experiments demonstrate that our
method can effectively jailbreak backdoor various LLMs even under the detection
of guardrail models, e.g., an ASR of 86.67% and 85% on LLaMA-3-8B and
Qwen-2.5-7B judged by GPT-4o.


http://arxiv.org/abs/2505.17654
EVADE: Multimodal Benchmark for Evasive Content Detection in E-Commerce Applications. (26%)
Ancheng Xu; Zhihao Yang; Jingpeng Li; Guanghu Yuan; Longze Chen; Liang Yan; Jiehui Zhou; Zhen Qin; Hengyun Chang; Hamid Alinejad-Rokny; Bo Zheng; Min Yang
  E-commerce platforms increasingly rely on Large Language Models (LLMs) and
Vision-Language Models (VLMs) to detect illicit or misleading product content.
However, these models remain vulnerable to evasive content: inputs (text or
images) that superficially comply with platform policies while covertly
conveying prohibited claims. Unlike traditional adversarial attacks that induce
overt failures, evasive content exploits ambiguity and context, making it far
harder to detect. Existing robustness benchmarks provide little guidance for
this demanding, real-world challenge. We introduce EVADE, the first
expert-curated, Chinese, multimodal benchmark specifically designed to evaluate
foundation models on evasive content detection in e-commerce. The dataset
contains 2,833 annotated text samples and 13,961 images spanning six demanding
product categories, including body shaping, height growth, and health
supplements. Two complementary tasks assess distinct capabilities:
Single-Violation, which probes fine-grained reasoning under short prompts, and
All-in-One, which tests long-context reasoning by merging overlapping policy
rules into unified instructions. Notably, the All-in-One setting significantly
narrows the performance gap between partial and full-match accuracy, suggesting
that clearer rule definitions improve alignment between human and model
judgment. We benchmark 26 mainstream LLMs and VLMs and observe substantial
performance gaps: even state-of-the-art models frequently misclassify evasive
samples. By releasing EVADE and strong baselines, we provide the first rigorous
standard for evaluating evasive-content detection, expose fundamental
limitations in current multimodal reasoning, and lay the groundwork for safer
and more transparent content moderation systems in e-commerce. The dataset is
publicly available at https://huggingface.co/datasets/koenshen/EVADE-Bench.


http://arxiv.org/abs/2505.18035
CAMME: Adaptive Deepfake Image Detection with Multi-Modal Cross-Attention. (15%)
Naseem Khan; Tuan Nguyen; Amine Bermak; Issa Khalil
  The proliferation of sophisticated AI-generated deepfakes poses critical
challenges for digital media authentication and societal security. While
existing detection methods perform well within specific generative domains,
they exhibit significant performance degradation when applied to manipulations
produced by unseen architectures--a fundamental limitation as generative
technologies rapidly evolve. We propose CAMME (Cross-Attention Multi-Modal
Embeddings), a framework that dynamically integrates visual, textual, and
frequency-domain features through a multi-head cross-attention mechanism to
establish robust cross-domain generalization. Extensive experiments demonstrate
CAMME's superiority over state-of-the-art methods, yielding improvements of
12.56% on natural scenes and 13.25% on facial deepfakes. The framework
demonstrates exceptional resilience, maintaining (over 91%) accuracy under
natural image perturbations and achieving 89.01% and 96.14% accuracy against
PGD and FGSM adversarial attacks, respectively. Our findings validate that
integrating complementary modalities through cross-attention enables more
effective decision boundary realignment for reliable deepfake detection across
heterogeneous generative architectures.


http://arxiv.org/abs/2505.17646
Understanding Pre-training and Fine-tuning from Loss Landscape Perspectives. (13%)
Huanran Chen; Yinpeng Dong; Zeming Wei; Yao Huang; Yichi Zhang; Hang Su; Jun Zhu
  Recent studies have revealed that the loss landscape of large language models
resembles a basin, within which the models perform nearly identically, and
outside of which they lose all their capabilities. In this work, we conduct
further studies on the loss landscape of large language models. We discover
that pre-training creates a "basic capability" basin, and subsequent
fine-tuning creates "specific capability" basins (e.g., math, safety, coding)
within the basic capability basin. We further investigate two types of loss
landscapes: the most-case landscape (i.e., the landscape along most directions)
and the worst-case landscape (i.e., the landscape along the worst direction).
We argue that as long as benign fine-tuning remains within the most-case basin,
it will not compromise previous capabilities. Similarly, any fine-tuning
(including the adversarial one) that stays within the worst-case basin would
not compromise previous capabilities. Finally, we theoretically demonstrate
that the size of the most-case basin can bound the size of the worst-case basin
and the robustness with respect to input perturbations. We also show that, due
to the over-parameterization property of current large language models, one can
easily enlarge the basins by five times.


http://arxiv.org/abs/2505.18015
SemSegBench & DetecBench: Benchmarking Reliability and Generalization Beyond Classification. (11%)
Shashank Agnihotri; David Schader; Jonas Jakubassa; Nico Sharei; Simon Kral; Mehmet Ege Kaçar; Ruben Weber; Margret Keuper
  Reliability and generalization in deep learning are predominantly studied in
the context of image classification. Yet, real-world applications in
safety-critical domains involve a broader set of semantic tasks, such as
semantic segmentation and object detection, which come with a diverse set of
dedicated model architectures. To facilitate research towards robust model
design in segmentation and detection, our primary objective is to provide
benchmarking tools regarding robustness to distribution shifts and adversarial
manipulations. We propose the benchmarking tools SEMSEGBENCH and DETECBENCH,
along with the most extensive evaluation to date on the reliability and
generalization of semantic segmentation and object detection models. In
particular, we benchmark 76 segmentation models across four datasets and 61
object detectors across two datasets, evaluating their performance under
diverse adversarial attacks and common corruptions. Our findings reveal
systematic weaknesses in state-of-the-art models and uncover key trends based
on architecture, backbone, and model capacity. SEMSEGBENCH and DETECBENCH are
open-sourced in our GitHub repository
(https://github.com/shashankskagnihotri/benchmarking_reliability_generalization)
along with our complete set of total 6139 evaluations. We anticipate the
collected data to foster and encourage future research towards improved model
reliability beyond classification.


http://arxiv.org/abs/2505.17568
JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models. (10%)
Zifan Peng; Yule Liu; Zhen Sun; Mingchen Li; Zeren Luo; Jingyi Zheng; Wenhan Dong; Xinlei He; Xuechao Wang; Yingjie Xue; Shengmin Xu; Xinyi Huang
  Audio Language Models (ALMs) have made significant progress recently. These
models integrate the audio modality directly into the model, rather than
converting speech into text and inputting text to Large Language Models (LLMs).
While jailbreak attacks on LLMs have been extensively studied, the security of
ALMs with audio modalities remains largely unexplored. Currently, there is a
lack of an adversarial audio dataset and a unified framework specifically
designed to evaluate and compare attacks and ALMs. In this paper, we present
JALMBench, the \textit{first} comprehensive benchmark to assess the safety of
ALMs against jailbreak attacks. JALMBench includes a dataset containing 2,200
text samples and 51,381 audio samples with over 268 hours. It supports 12
mainstream ALMs, 4 text-transferred and 4 audio-originated attack methods, and
5 defense methods. Using JALMBench, we provide an in-depth analysis of attack
efficiency, topic sensitivity, voice diversity, and attack representations.
Additionally, we explore mitigation strategies for the attacks at both the
prompt level and the response level.


http://arxiv.org/abs/2505.17776
Sec5GLoc: Securing 5G Indoor Localization via Adversary-Resilient Deep Learning Architecture. (10%)
Ildi Alla; Valeria Loscri
  Emerging 5G millimeter-wave and sub-6 GHz networks enable high-accuracy
indoor localization, but security and privacy vulnerabilities pose serious
challenges. In this paper, we identify and address threats including location
spoofing and adversarial signal manipulation against 5G-based indoor
localization. We formalize a threat model encompassing attackers who inject
forged radio signals or perturb channel measurements to mislead the
localization system. To defend against these threats, we propose an
adversary-resilient localization architecture that combines deep learning
fingerprinting with physical domain knowledge. Our approach integrates
multi-anchor Channel Impulse Response (CIR) fingerprints with Time Difference
of Arrival (TDoA) features and known anchor positions in a hybrid Convolutional
Neural Network (CNN) and multi-head attention network. This design inherently
checks geometric consistency and dynamically down-weights anomalous signals,
making localization robust to tampering. We formulate the secure localization
problem and demonstrate, through extensive experiments on a public 5G indoor
dataset, that the proposed system achieves a mean error approximately 0.58 m
under mixed Line-of-Sight (LOS) and Non-Line-of-Sight (NLOS) trajectories in
benign conditions and gracefully degrades to around 0.81 m under attack
scenarios. We also show via ablation studies that each architecture component
(attention mechanism, TDoA, etc.) is critical for both accuracy and resilience,
reducing errors by 4-5 times compared to baselines. In addition, our system
runs in real-time, localizing the user in just 1 ms on a simple CPU. The code
has been released to ensure reproducibility
(https://github.com/sec5gloc/Sec5GLoc).


http://arxiv.org/abs/2505.18323
Architectural Backdoors for Within-Batch Data Stealing and Model Inference Manipulation. (9%)
Nicolas Küchler; Ivan Petrov; Conrad Grobler; Ilia Shumailov
  For nearly a decade the academic community has investigated backdoors in
neural networks, primarily focusing on classification tasks where adversaries
manipulate the model prediction. While demonstrably malicious, the immediate
real-world impact of such prediction-altering attacks has remained unclear. In
this paper we introduce a novel and significantly more potent class of
backdoors that builds upon recent advancements in architectural backdoors. We
demonstrate how these backdoors can be specifically engineered to exploit
batched inference, a common technique for hardware utilization, enabling
large-scale user data manipulation and theft. By targeting the batching
process, these architectural backdoors facilitate information leakage between
concurrent user requests and allow attackers to fully control model responses
directed at other users within the same batch. In other words, an attacker who
can change the model architecture can set and steal model inputs and outputs of
other users within the same batch. We show that such attacks are not only
feasible but also alarmingly effective, can be readily injected into prevalent
model architectures, and represent a truly malicious threat to user privacy and
system integrity. Critically, to counteract this new class of vulnerabilities,
we propose a deterministic mitigation strategy that provides formal guarantees
against this new attack vector, unlike prior work that relied on Large Language
Models to find the backdoors. Our mitigation strategy employs a novel
Information Flow Control mechanism that analyzes the model graph and proves
non-interference between different user inputs within the same batch. Using our
mitigation strategy we perform a large scale analysis of models hosted through
Hugging Face and find over 200 models that introduce (unintended) information
leakage between batch entries due to the use of dynamic quantization.


http://arxiv.org/abs/2505.16714
Experimental robustness benchmark of quantum neural network on a superconducting quantum processor. (99%)
Hai-Feng Zhang; Zhao-Yun Chen; Peng Wang; Liang-Liang Guo; Tian-Le Wang; Xiao-Yan Yang; Ren-Ze Zhao; Ze-An Zhao; Sheng Zhang; Lei Du; Hao-Ran Tao; Zhi-Long Jia; Wei-Cheng Kong; Huan-Yu Liu; Athanasios V. Vasilakos; Yang Yang; Yu-Chun Wu; Ji Guan; Peng Duan; Guo-Ping Guo
  Quantum machine learning (QML) models, like their classical counterparts, are
vulnerable to adversarial attacks, hindering their secure deployment. Here, we
report the first systematic experimental robustness benchmark for 20-qubit
quantum neural network (QNN) classifiers executed on a superconducting
processor. Our benchmarking framework features an efficient adversarial attack
algorithm designed for QNNs, enabling quantitative characterization of
adversarial robustness and robustness bounds. From our analysis, we verify that
adversarial training reduces sensitivity to targeted perturbations by
regularizing input gradients, significantly enhancing QNN's robustness.
Additionally, our analysis reveals that QNNs exhibit superior adversarial
robustness compared to classical neural networks, an advantage attributed to
inherent quantum noise. Furthermore, the empirical upper bound extracted from
our attack experiments shows a minimal deviation ($3 \times 10^{-3}$) from the
theoretical lower bound, providing strong experimental confirmation of the
attack's effectiveness and the tightness of fidelity-based robustness bounds.
This work establishes a critical experimental framework for assessing and
improving quantum adversarial robustness, paving the way for secure and
reliable QML applications.


http://arxiv.org/abs/2505.16318
SuperPure: Efficient Purification of Localized and Distributed Adversarial Patches via Super-Resolution GAN Models. (99%)
Hossein Khalili; Seongbin Park; Venkat Bollapragada; Nader Sehatbakhsh
  As vision-based machine learning models are increasingly integrated into
autonomous and cyber-physical systems, concerns about (physical) adversarial
patch attacks are growing. While state-of-the-art defenses can achieve
certified robustness with minimal impact on utility against highly-concentrated
localized patch attacks, they fall short in two important areas: (i)
State-of-the-art methods are vulnerable to low-noise distributed patches where
perturbations are subtly dispersed to evade detection or masking, as shown
recently by the DorPatch attack; (ii) Achieving high robustness with
state-of-the-art methods is extremely time and resource-consuming, rendering
them impractical for latency-sensitive applications in many cyber-physical
systems.
  To address both robustness and latency issues, this paper proposes a new
defense strategy for adversarial patch attacks called SuperPure. The key
novelty is developing a pixel-wise masking scheme that is robust against both
distributed and localized patches. The masking involves leveraging a GAN-based
super-resolution scheme to gradually purify the image from adversarial patches.
Our extensive evaluations using ImageNet and two standard classifiers, ResNet
and EfficientNet, show that SuperPure advances the state-of-the-art in three
major directions: (i) it improves the robustness against conventional localized
patches by more than 20%, on average, while also improving top-1 clean accuracy
by almost 10%; (ii) It achieves 58% robustness against distributed patch
attacks (as opposed to 0% in state-of-the-art method, PatchCleanser); (iii) It
decreases the defense end-to-end latency by over 98% compared to PatchCleanser.
Our further analysis shows that SuperPure is robust against white-box attacks
and different patch sizes. Our code is open-source.


http://arxiv.org/abs/2505.16313
Accelerating Targeted Hard-Label Adversarial Attacks in Low-Query Black-Box Settings. (99%)
Arjhun Swaminathan; Mete Akgün
  Deep neural networks for image classification remain vulnerable to
adversarial examples -- small, imperceptible perturbations that induce
misclassifications. In black-box settings, where only the final prediction is
accessible, crafting targeted attacks that aim to misclassify into a specific
target class is particularly challenging due to narrow decision regions.
Current state-of-the-art methods often exploit the geometric properties of the
decision boundary separating a source image and a target image rather than
incorporating information from the images themselves. In contrast, we propose
Targeted Edge-informed Attack (TEA), a novel attack that utilizes edge
information from the target image to carefully perturb it, thereby producing an
adversarial image that is closer to the source image while still achieving the
desired target classification. Our approach consistently outperforms current
state-of-the-art methods across different models in low query settings (nearly
70\% fewer queries are used), a scenario especially relevant in real-world
applications with limited queries and black-box access. Furthermore, by
efficiently generating a suitable adversarial example, TEA provides an improved
target initialization for established geometry-based attacks.


http://arxiv.org/abs/2505.17440
VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models. (99%)
Hefei Mei; Zirui Wang; Shen You; Minjing Dong; Chang Xu
  Large Vision-Language Models (LVLMs) have demonstrated remarkable
capabilities in multimodal understanding and generation, yet their
vulnerability to adversarial attacks raises significant robustness concerns.
While existing effective attacks always focus on task-specific white-box
settings, these approaches are limited in the context of LVLMs, which are
designed for diverse downstream tasks and require expensive full-model gradient
computations. Motivated by the pivotal role and wide adoption of the vision
encoder in LVLMs, we propose a simple yet effective Vision Encoder Attack
(VEAttack), which targets the vision encoder of LVLMs only. Specifically, we
propose to generate adversarial examples by minimizing the cosine similarity
between the clean and perturbed visual features, without accessing the
following large language models, task information, and labels. It significantly
reduces the computational overhead while eliminating the task and label
dependence of traditional white-box attacks in LVLMs. To make this simple
attack effective, we propose to perturb images by optimizing image tokens
instead of the classification token. We provide both empirical and theoretical
evidence that VEAttack can easily generalize to various tasks. VEAttack has
achieved a performance degradation of 94.5% on image caption task and 75.7% on
visual question answering task. We also reveal some key observations to provide
insights into LVLM attack/defense: 1) hidden layer variations of LLM, 2) token
attention differential, 3) M\"obius band in transfer attack, 4) low sensitivity
to attack steps. The code is available at
https://github.com/hfmei/VEAttack-LVLM


http://arxiv.org/abs/2505.16367
Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems. (96%)
Hongru Song; Yu-an Liu; Ruqing Zhang; Jiafeng Guo; Yixing Fan
  Retrieval-augmented generation (RAG) systems can effectively mitigate the
hallucination problem of large language models (LLMs),but they also possess
inherent vulnerabilities. Identifying these weaknesses before the large-scale
real-world deployment of RAG systems is of great importance, as it lays the
foundation for building more secure and robust RAG systems in the future.
Existing adversarial attack methods typically exploit knowledge base poisoning
to probe the vulnerabilities of RAG systems, which can effectively deceive
standard RAG models. However, with the rapid advancement of deep reasoning
capabilities in modern LLMs, previous approaches that merely inject incorrect
knowledge are inadequate when attacking RAG systems equipped with deep
reasoning abilities. Inspired by the deep thinking capabilities of LLMs, this
paper extracts reasoning process templates from R1-based RAG systems, uses
these templates to wrap erroneous knowledge into adversarial documents, and
injects them into the knowledge base to attack RAG systems. The key idea of our
approach is that adversarial documents, by simulating the chain-of-thought
patterns aligned with the model's training signals, may be misinterpreted by
the model as authentic historical reasoning processes, thus increasing their
likelihood of being referenced. Experiments conducted on the MS MARCO passage
ranking dataset demonstrate the effectiveness of our proposed method.


http://arxiv.org/abs/2505.16947
MixAT: Combining Continuous and Discrete Adversarial Training for LLMs. (87%)
Csaba Dékány; Stefan Balauca; Robin Staab; Dimitar I. Dimitrov; Martin Vechev
  Despite recent efforts in Large Language Models (LLMs) safety and alignment,
current adversarial attacks on frontier LLMs are still able to force harmful
generations consistently. Although adversarial training has been widely studied
and shown to significantly improve the robustness of traditional machine
learning models, its strengths and weaknesses in the context of LLMs are less
understood. Specifically, while existing discrete adversarial attacks are
effective at producing harmful content, training LLMs with concrete adversarial
prompts is often computationally expensive, leading to reliance on continuous
relaxations. As these relaxations do not correspond to discrete input tokens,
such latent training methods often leave models vulnerable to a diverse set of
discrete attacks. In this work, we aim to bridge this gap by introducing MixAT,
a novel method that combines stronger discrete and faster continuous attacks
during training. We rigorously evaluate MixAT across a wide spectrum of
state-of-the-art attacks, proposing the At Least One Attack Success Rate
(ALO-ASR) metric to capture the worst-case vulnerability of models. We show
MixAT achieves substantially better robustness (ALO-ASR < 20%) compared to
prior defenses (ALO-ASR > 50%), while maintaining a runtime comparable to
methods based on continuous relaxations. We further analyze MixAT in realistic
deployment settings, exploring how chat templates, quantization, low-rank
adapters, and temperature affect both adversarial training and evaluation,
revealing additional blind spots in current methodologies. Our results
demonstrate that MixAT's discrete-continuous defense offers a principled and
superior robustness-accuracy tradeoff with minimal computational overhead,
highlighting its promise for building safer LLMs. We provide our code and
models at https://github.com/insait-institute/MixAT.


http://arxiv.org/abs/2505.16640
BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization. (64%)
Xueyang Zhou; Guiyao Tie; Guowen Zhang; Hechang Wang; Pan Zhou; Lichao Sun
  Vision-Language-Action (VLA) models have advanced robotic control by enabling
end-to-end decision-making directly from multimodal inputs. However, their
tightly coupled architectures expose novel security vulnerabilities. Unlike
traditional adversarial perturbations, backdoor attacks represent a stealthier,
persistent, and practically significant threat-particularly under the emerging
Training-as-a-Service paradigm-but remain largely unexplored in the context of
VLA models. To address this gap, we propose BadVLA, a backdoor attack method
based on Objective-Decoupled Optimization, which for the first time exposes the
backdoor vulnerabilities of VLA models. Specifically, it consists of a
two-stage process: (1) explicit feature-space separation to isolate trigger
representations from benign inputs, and (2) conditional control deviations that
activate only in the presence of the trigger, while preserving clean-task
performance. Empirical results on multiple VLA benchmarks demonstrate that
BadVLA consistently achieves near-100% attack success rates with minimal impact
on clean task accuracy. Further analyses confirm its robustness against common
input perturbations, task transfers, and model fine-tuning, underscoring
critical security vulnerabilities in current VLA deployments. Our work offers
the first systematic investigation of backdoor vulnerabilities in VLA models,
highlighting an urgent need for secure and trustworthy embodied model design
practices. We have released the project page at
https://badvla-project.github.io/.


http://arxiv.org/abs/2505.17190
Tropical Attention: Neural Algorithmic Reasoning for Combinatorial Algorithms. (50%)
Baran Hashemi; Kurt Pasque; Chris Teska; Ruriko Yoshida
  Dynamic programming (DP) algorithms for combinatorial optimization problems
work with taking maximization, minimization, and classical addition in their
recursion algorithms. The associated value functions correspond to convex
polyhedra in the max plus semiring. Existing Neural Algorithmic Reasoning
models, however, rely on softmax-normalized dot-product attention where the
smooth exponential weighting blurs these sharp polyhedral structures and
collapses when evaluated on out-of-distribution (OOD) settings. We introduce
Tropical attention, a novel attention function that operates natively in the
max-plus semiring of tropical geometry. We prove that Tropical attention can
approximate tropical circuits of DP-type combinatorial algorithms. We then
propose that using Tropical transformers enhances empirical OOD performance in
both length generalization and value generalization, on algorithmic reasoning
tasks, surpassing softmax baselines while remaining stable under adversarial
attacks. We also present adversarial-attack generalization as a third axis for
Neural Algorithmic Reasoning benchmarking. Our results demonstrate that
Tropical attention restores the sharp, scale-invariant reasoning absent from
softmax.


http://arxiv.org/abs/2505.17226
Secure and Private Federated Learning: Achieving Adversarial Resilience through Robust Aggregation. (13%)
Kun Yang; Neena Imam
  Federated Learning (FL) enables collaborative machine learning across
decentralized data sources without sharing raw data. It offers a promising
approach to privacy-preserving AI. However, FL remains vulnerable to
adversarial threats from malicious participants, referred to as Byzantine
clients, who can send misleading updates to corrupt the global model.
Traditional aggregation methods, such as simple averaging, are not robust to
such attacks. More resilient approaches, like the Krum algorithm, require prior
knowledge of the number of malicious clients, which is often unavailable in
real-world scenarios. To address these limitations, we propose Average-rKrum
(ArKrum), a novel aggregation strategy designed to enhance both the resilience
and privacy guarantees of FL systems. Building on our previous work (rKrum),
ArKrum introduces two key innovations. First, it includes a median-based
filtering mechanism that removes extreme outliers before estimating the number
of adversarial clients. Second, it applies a multi-update averaging scheme to
improve stability and performance, particularly when client data distributions
are not identical. We evaluate ArKrum on benchmark image and text datasets
under three widely studied Byzantine attack types. Results show that ArKrum
consistently achieves high accuracy and stability. It performs as well as or
better than other robust aggregation methods. These findings demonstrate that
ArKrum is an effective and practical solution for secure FL systems in
adversarial environments.


http://arxiv.org/abs/2505.16765
When Safety Detectors Aren't Enough: A Stealthy and Effective Jailbreak Attack on LLMs via Steganographic Techniques. (10%)
Jianing Geng; Biao Yi; Zekun Fei; Tongxi Wu; Lihai Nie; Zheli Liu
  Jailbreak attacks pose a serious threat to large language models (LLMs) by
bypassing built-in safety mechanisms and leading to harmful outputs. Studying
these attacks is crucial for identifying vulnerabilities and improving model
security. This paper presents a systematic survey of jailbreak methods from the
novel perspective of stealth. We find that existing attacks struggle to
simultaneously achieve toxic stealth (concealing toxic content) and linguistic
stealth (maintaining linguistic naturalness). Motivated by this, we propose
StegoAttack, a fully stealthy jailbreak attack that uses steganography to hide
the harmful query within benign, semantically coherent text. The attack then
prompts the LLM to extract the hidden query and respond in an encrypted manner.
This approach effectively hides malicious intent while preserving naturalness,
allowing it to evade both built-in and external safety mechanisms. We evaluate
StegoAttack on four safety-aligned LLMs from major providers, benchmarking
against eight state-of-the-art methods. StegoAttack achieves an average attack
success rate (ASR) of 92.00%, outperforming the strongest baseline by 11.0%.
Its ASR drops by less than 1% even under external detection (e.g., Llama
Guard). Moreover, it attains the optimal comprehensive scores on stealth
detection metrics, demonstrating both high efficacy and exceptional stealth
capabilities. The code is available at
https://anonymous.4open.science/r/StegoAttack-Jail66


http://arxiv.org/abs/2505.16793
REOBench: Benchmarking Robustness of Earth Observation Foundation Models. (2%)
Xiang Li; Yong Tao; Siyuan Zhang; Siwei Liu; Zhitong Xiong; Chunbo Luo; Lu Liu; Mykola Pechenizkiy; Xiao Xiang Zhu; Tianjin Huang
  Earth observation foundation models have shown strong generalization across
multiple Earth observation tasks, but their robustness under real-world
perturbations remains underexplored. To bridge this gap, we introduce REOBench,
the first comprehensive benchmark for evaluating the robustness of Earth
observation foundation models across six tasks and twelve types of image
corruptions, including both appearance-based and geometric perturbations. To
ensure realistic and fine-grained evaluation, our benchmark focuses on
high-resolution optical remote sensing images, which are widely used in
critical applications such as urban planning and disaster response. We conduct
a systematic evaluation of a broad range of models trained using masked image
modeling, contrastive learning, and vision-language pre-training paradigms. Our
results reveal that (1) existing Earth observation foundation models experience
significant performance degradation when exposed to input corruptions. (2) The
severity of degradation varies across tasks, model architectures, backbone
sizes, and types of corruption, with performance drop varying from less than 1%
to over 20%. (3) Vision-language models show enhanced robustness, particularly
in multimodal tasks. REOBench underscores the vulnerability of current Earth
observation foundation models to real-world corruptions and provides actionable
insights for developing more robust and reliable models.


http://arxiv.org/abs/2505.17147
MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming. (2%)
Weiyang Guo; Jing Li; Wenya Wang; YU LI; Daojing He; Jun Yu; Min Zhang
  The proliferation of jailbreak attacks against large language models (LLMs)
highlights the need for robust security measures. However, in multi-round
dialogues, malicious intentions may be hidden in interactions, leading LLMs to
be more prone to produce harmful responses. In this paper, we propose the
\textbf{M}ulti-\textbf{T}urn \textbf{S}afety \textbf{A}lignment (\ourapproach)
framework, to address the challenge of securing LLMs in multi-round
interactions. It consists of two stages: In the thought-guided attack learning
stage, the red-team model learns about thought-guided multi-round jailbreak
attacks to generate adversarial prompts. In the adversarial iterative
optimization stage, the red-team model and the target model continuously
improve their respective capabilities in interaction. Furthermore, we introduce
a multi-turn reinforcement learning algorithm based on future rewards to
enhance the robustness of safety alignment. Experimental results show that the
red-team model exhibits state-of-the-art attack capabilities, while the target
model significantly improves its performance on safety benchmarks.


http://arxiv.org/abs/2505.16263
All You Need is "Leet": Evading Hate-speech Detection AI. (1%)
Sampanna Yashwant Kahu; Naman Ahuja
  Social media and online forums are increasingly becoming popular.
Unfortunately, these platforms are being used for spreading hate speech. In
this paper, we design black-box techniques to protect users from hate-speech on
online platforms by generating perturbations that can fool state of the art
deep learning based hate speech detection models thereby decreasing their
efficiency. We also ensure a minimal change in the original meaning of
hate-speech. Our best perturbation attack is successfully able to evade
hate-speech detection for 86.8 % of hateful text.


http://arxiv.org/abs/2505.16583
Training on Plausible Counterfactuals Removes Spurious Correlations. (1%)
Shpresim Sadiku; Kartikeya Chitranshi; Hiroshi Kera; Sebastian Pokutta
  Plausible counterfactual explanations (p-CFEs) are perturbations that
minimally modify inputs to change classifier decisions while remaining
plausible under the data distribution. In this study, we demonstrate that
classifiers can be trained on p-CFEs labeled with induced \emph{incorrect}
target classes to classify unperturbed inputs with the original labels. While
previous studies have shown that such learning is possible with adversarial
perturbations, we extend this paradigm to p-CFEs. Interestingly, our
experiments reveal that learning from p-CFEs is even more effective: the
resulting classifiers achieve not only high in-distribution accuracy but also
exhibit significantly reduced bias with respect to spurious correlations.


http://arxiv.org/abs/2505.15594
Beyond Classification: Evaluating Diffusion Denoised Smoothing for Security-Utility Trade off. (99%)
Yury Belousov; Brian Pulfer; Vitaliy Kinakh; Slava Voloshynovskiy
  While foundation models demonstrate impressive performance across various
tasks, they remain vulnerable to adversarial inputs. Current research explores
various approaches to enhance model robustness, with Diffusion Denoised
Smoothing emerging as a particularly promising technique. This method employs a
pretrained diffusion model to preprocess inputs before model inference. Yet,
its effectiveness remains largely unexplored beyond classification. We aim to
address this gap by analyzing three datasets with four distinct downstream
tasks under three different adversarial attack algorithms. Our findings reveal
that while foundation models maintain resilience against conventional
transformations, applying high-noise diffusion denoising to clean images
without any distortions significantly degrades performance by as high as 57%.
Low-noise diffusion settings preserve performance but fail to provide adequate
protection across all attack types. Moreover, we introduce a novel attack
strategy specifically targeting the diffusion process itself, capable of
circumventing defenses in the low-noise regime. Our results suggest that the
trade-off between adversarial robustness and performance remains a challenge to
be addressed.


http://arxiv.org/abs/2505.15738
Alignment Under Pressure: The Case for Informed Adversaries When Evaluating LLM Defenses. (99%)
Xiaoxue Yang; Bozhidar Stevanoski; Matthieu Meeus; Montjoye Yves-Alexandre de
  Large language models (LLMs) are rapidly deployed in real-world applications
ranging from chatbots to agentic systems. Alignment is one of the main
approaches used to defend against attacks such as prompt injection and
jailbreaks. Recent defenses report near-zero Attack Success Rates (ASR) even
against Greedy Coordinate Gradient (GCG), a white-box attack that generates
adversarial suffixes to induce attacker-desired outputs. However, this search
space over discrete tokens is extremely large, making the task of finding
successful attacks difficult. GCG has, for instance, been shown to converge to
local minima, making it sensitive to initialization choices. In this paper, we
assess the future-proof robustness of these defenses using a more informed
threat model: attackers who have access to some information about the alignment
process. Specifically, we propose an informed white-box attack leveraging the
intermediate model checkpoints to initialize GCG, with each checkpoint acting
as a stepping stone for the next one. We show this approach to be highly
effective across state-of-the-art (SOTA) defenses and models. We further show
our informed initialization to outperform other initialization methods and show
a gradient-informed checkpoint selection strategy to greatly improve attack
performance and efficiency. Importantly, we also show our method to
successfully find universal adversarial suffixes -- single suffixes effective
across diverse inputs. Our results show that, contrary to previous beliefs,
effective adversarial suffixes do exist against SOTA alignment-based defenses,
that these can be found by existing attack methods when adversaries exploit
alignment knowledge, and that even universal suffixes exist. Taken together,
our results highlight the brittleness of current alignment-based methods and
the need to consider stronger threat models when testing the safety of LLMs.


http://arxiv.org/abs/2505.16166
TRAIL: Transferable Robust Adversarial Images via Latent diffusion. (98%)
Yuhao Xue; Zhifei Zhang; Xinyang Jiang; Yifei Shen; Junyao Gao; Wentao Gu; Jiale Zhao; Miaojing Shi; Cairong Zhao
  Adversarial attacks exploiting unrestricted natural perturbations present
severe security risks to deep learning systems, yet their transferability
across models remains limited due to distribution mismatches between generated
adversarial features and real-world data. While recent works utilize
pre-trained diffusion models as adversarial priors, they still encounter
challenges due to the distribution shift between the distribution of ideal
adversarial samples and the natural image distribution learned by the diffusion
model. To address the challenge, we propose Transferable Robust Adversarial
Images via Latent Diffusion (TRAIL), a test-time adaptation framework that
enables the model to generate images from a distribution of images with
adversarial features and closely resembles the target images. To mitigate the
distribution shift, during attacks, TRAIL updates the diffusion U-Net's weights
by combining adversarial objectives (to mislead victim models) and perceptual
constraints (to preserve image realism). The adapted model then generates
adversarial samples through iterative noise injection and denoising guided by
these objectives. Experiments demonstrate that TRAIL significantly outperforms
state-of-the-art methods in cross-model attack transferability, validating that
distribution-aligned adversarial feature synthesis is critical for practical
black-box attacks.


http://arxiv.org/abs/2505.15753
Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval. (82%)
Taiye Chen; Zeming Wei; Ang Li; Yisen Wang
  Large Language Models (LLMs) are known to be vulnerable to jailbreaking
attacks, wherein adversaries exploit carefully engineered prompts to induce
harmful or unethical responses. Such threats have raised critical concerns
about the safety and reliability of LLMs in real-world deployment. While
existing defense mechanisms partially mitigate such risks, subsequent
advancements in adversarial techniques have enabled novel jailbreaking methods
to circumvent these protections, exposing the limitations of static defense
frameworks. In this work, we explore defending against evolving jailbreaking
threats through the lens of context retrieval. First, we conduct a preliminary
study demonstrating that even a minimal set of safety-aligned examples against
a particular jailbreak can significantly enhance robustness against this attack
pattern. Building on this insight, we further leverage the retrieval-augmented
generation (RAG) techniques and propose Safety Context Retrieval (SCR), a
scalable and robust safeguarding paradigm for LLMs against jailbreaking. Our
comprehensive experiments demonstrate how SCR achieves superior defensive
performance against both established and emerging jailbreaking tactics,
contributing a new paradigm to LLM safety. Our code will be available upon
publication.


http://arxiv.org/abs/2505.16004
Interpretability Illusions with Sparse Autoencoders: Evaluating Robustness of Concept Representations. (75%)
Aaron J. Li; Suraj Srinivas; Usha Bhalla; Himabindu Lakkaraju
  Sparse autoencoders (SAEs) are commonly used to interpret the internal
activations of large language models (LLMs) by mapping them to
human-interpretable concept representations. While existing evaluations of SAEs
focus on metrics such as the reconstruction-sparsity tradeoff, human
(auto-)interpretability, and feature disentanglement, they overlook a critical
aspect: the robustness of concept representations to input perturbations. We
argue that robustness must be a fundamental consideration for concept
representations, reflecting the fidelity of concept labeling. To this end, we
formulate robustness quantification as input-space optimization problems and
develop a comprehensive evaluation framework featuring realistic scenarios in
which adversarial perturbations are crafted to manipulate SAE representations.
Empirically, we find that tiny adversarial input perturbations can effectively
manipulate concept-based interpretations in most scenarios without notably
affecting the outputs of the base LLMs themselves. Overall, our results suggest
that SAE concept representations are fragile and may be ill-suited for
applications in model monitoring and oversight.


http://arxiv.org/abs/2505.15406
Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models. (64%)
Zirui Song; Qian Jiang; Mingxuan Cui; Mingzhe Li; Lang Gao; Zeyu Zhang; Zixiang Xu; Yanbo Wang; Chenxi Wang; Guangxian Ouyang; Zhenhao Chen; Xiuying Chen
  The rise of Large Audio Language Models (LAMs) brings both potential and
risks, as their audio outputs may contain harmful or unethical content.
However, current research lacks a systematic, quantitative evaluation of LAM
safety especially against jailbreak attacks, which are challenging due to the
temporal and semantic nature of speech. To bridge this gap, we introduce
AJailBench, the first benchmark specifically designed to evaluate jailbreak
vulnerabilities in LAMs. We begin by constructing AJailBench-Base, a dataset of
1,495 adversarial audio prompts spanning 10 policy-violating categories,
converted from textual jailbreak attacks using realistic text to speech
synthesis. Using this dataset, we evaluate several state-of-the-art LAMs and
reveal that none exhibit consistent robustness across attacks. To further
strengthen jailbreak testing and simulate more realistic attack conditions, we
propose a method to generate dynamic adversarial variants. Our Audio
Perturbation Toolkit (APT) applies targeted distortions across time, frequency,
and amplitude domains. To preserve the original jailbreak intent, we enforce a
semantic consistency constraint and employ Bayesian optimization to efficiently
search for perturbations that are both subtle and highly effective. This
results in AJailBench-APT, an extended dataset of optimized adversarial audio
samples. Our findings demonstrate that even small, semantically preserved
perturbations can significantly reduce the safety performance of leading LAMs,
underscoring the need for more robust and semantically aware defense
mechanisms.


http://arxiv.org/abs/2505.15420
Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries. (38%)
Yuhao Wang; Wenjie Qu; Yanze Jiang; Zichen Liu; Yue Liu; Shengfang Zhai; Yinpeng Dong; Jiaheng Zhang
  Retrieval-Augmented Generation (RAG) systems enhance large language models
(LLMs) by incorporating external knowledge bases, but they are vulnerable to
privacy risks from data extraction attacks. Existing extraction methods
typically rely on malicious inputs such as prompt injection or jailbreaking,
making them easily detectable via input- or output-level detection. In this
paper, we introduce Implicit Knowledge Extraction Attack (IKEA), which conducts
knowledge extraction on RAG systems through benign queries. IKEA first
leverages anchor concepts to generate queries with the natural appearance, and
then designs two mechanisms to lead to anchor concept thoroughly 'explore' the
RAG's privacy knowledge: (1) Experience Reflection Sampling, which samples
anchor concepts based on past query-response patterns to ensure the queries'
relevance to RAG documents; (2) Trust Region Directed Mutation, which
iteratively mutates anchor concepts under similarity constraints to further
exploit the embedding space. Extensive experiments demonstrate IKEA's
effectiveness under various defenses, surpassing baselines by over 80% in
extraction efficiency and 90% in attack success rate. Moreover, the substitute
RAG system built from IKEA's extractions consistently outperforms those based
on baseline methods across multiple evaluation tasks, underscoring the
significant privacy risk in RAG systems.


http://arxiv.org/abs/2505.17132
Robustifying Vision-Language Models via Dynamic Token Reweighting. (31%)
Tanqiu Jiang; Jiacheng Liang; Rongyi Zhu; Jiawei Zhou; Fenglong Ma; Ting Wang
  Large vision-language models (VLMs) are highly vulnerable to jailbreak
attacks that exploit visual-textual interactions to bypass safety guardrails.
In this paper, we present DTR, a novel inference-time defense that mitigates
multimodal jailbreak attacks through optimizing the model's key-value (KV)
caches. Rather than relying on curated safety-specific data or costly
image-to-text conversion, we introduce a new formulation of the safety-relevant
distributional shift induced by the visual modality. This formulation enables
DTR to dynamically adjust visual token weights, minimizing the impact of
adversarial visual inputs while preserving the model's general capabilities and
inference efficiency. Extensive evaluation across diverse VLMs and attack
benchmarks demonstrates that \sys outperforms existing defenses in both attack
robustness and benign task performance, marking the first successful
application of KV cache optimization for safety enhancement in multimodal
foundation models. (warning: this paper contains potentially harmful content
generated by VLMs.)


http://arxiv.org/abs/2505.15194
GAMA: Geometry-Aware Manifold Alignment via Structured Adversarial Perturbations for Robust Domain Adaptation. (11%)
Hana Satou; F Monkey
  Domain adaptation remains a challenge when there is significant manifold
discrepancy between source and target domains. Although recent methods leverage
manifold-aware adversarial perturbations to perform data augmentation, they
often neglect precise manifold alignment and systematic exploration of
structured perturbations. To address this, we propose GAMA (Geometry-Aware
Manifold Alignment), a structured framework that achieves explicit manifold
alignment via adversarial perturbation guided by geometric information. GAMA
systematically employs tangent space exploration and manifold-constrained
adversarial optimization, simultaneously enhancing semantic consistency,
robustness to off-manifold deviations, and cross-domain alignment. Theoretical
analysis shows that GAMA tightens the generalization bound via structured
regularization and explicit alignment. Empirical results on DomainNet, VisDA,
and Office-Home demonstrate that GAMA consistently outperforms existing
adversarial and adaptation methods in both unsupervised and few-shot settings,
exhibiting superior robustness, generalization, and manifold alignment
capability.


http://arxiv.org/abs/2505.16014
Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains. (5%)
Yash Saxena; Ankur Padia; Mandar S Chaudhary; Kalpa Gunaratna; Srinivasan Parthasarathy; Manas Gaur
  Traditional Retrieval-Augmented Generation (RAG) pipelines rely on
similarity-based retrieval and re-ranking, which depend on heuristics such as
top-k, and lack explainability, interpretability, and robustness against
adversarial content. To address this gap, we propose a novel method METEORA
that replaces re-ranking in RAG with a rationale-driven selection approach.
METEORA operates in two stages. First, a general-purpose LLM is
preference-tuned to generate rationales conditioned on the input query using
direct preference optimization. These rationales guide the evidence chunk
selection engine, which selects relevant chunks in three stages: pairing
individual rationales with corresponding retrieved chunks for local relevance,
global selection with elbow detection for adaptive cutoff, and context
expansion via neighboring chunks. This process eliminates the need for top-k
heuristics. The rationales are also used for consistency check using a Verifier
LLM to detect and filter poisoned or misleading content for safe generation.
The framework provides explainable and interpretable evidence flow by using
rationales consistently across both selection and verification. Our evaluation
across six datasets spanning legal, financial, and academic research domains
shows that METEORA improves generation accuracy by 33.34% while using
approximately 50% fewer chunks than state-of-the-art re-ranking methods. In
adversarial settings, METEORA significantly improves the F1 score from 0.10 to
0.44 over the state-of-the-art perplexity-based defense baseline, demonstrating
strong resilience to poisoning attacks. Code available at:
https://anonymous.4open.science/r/METEORA-DC46/README.md


http://arxiv.org/abs/2505.15191
Geometrically Regularized Transfer Learning with On-Manifold and Off-Manifold Perturbation. (4%)
Hana Satou; Alan Mitkiy; F Monkey
  Transfer learning under domain shift remains a fundamental challenge due to
the divergence between source and target data manifolds. In this paper, we
propose MAADA (Manifold-Aware Adversarial Data Augmentation), a novel framework
that decomposes adversarial perturbations into on-manifold and off-manifold
components to simultaneously capture semantic variation and model brittleness.
We theoretically demonstrate that enforcing on-manifold consistency reduces
hypothesis complexity and improves generalization, while off-manifold
regularization smooths decision boundaries in low-density regions. Moreover, we
introduce a geometry-aware alignment loss that minimizes geodesic discrepancy
between source and target manifolds. Experiments on DomainNet, VisDA, and
Office-Home show that MAADA consistently outperforms existing adversarial and
adaptation methods in both unsupervised and few-shot settings, demonstrating
superior structural robustness and cross-domain generalization.


http://arxiv.org/abs/2505.15337
Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors. (4%)
Hao Fang; Jiawei Kong; Tianqu Zhuang; Yixiang Qiu; Kuofeng Gao; Bin Chen; Shu-Tao Xia; Yaowei Wang; Min Zhang
  The misuse of large language models (LLMs), such as academic plagiarism, has
driven the development of detectors to identify LLM-generated texts. To bypass
these detectors, paraphrase attacks have emerged to purposely rewrite these
texts to evade detection. Despite the success, existing methods require
substantial data and computational budgets to train a specialized paraphraser,
and their attack efficacy greatly reduces when faced with advanced detection
algorithms. To address this, we propose \textbf{Co}ntrastive
\textbf{P}araphrase \textbf{A}ttack (CoPA), a training-free method that
effectively deceives text detectors using off-the-shelf LLMs. The first step is
to carefully craft instructions that encourage LLMs to produce more human-like
texts. Nonetheless, we observe that the inherent statistical biases of LLMs can
still result in some generated texts carrying certain machine-like attributes
that can be captured by detectors. To overcome this, CoPA constructs an
auxiliary machine-like word distribution as a contrast to the human-like
distribution generated by the LLM. By subtracting the machine-like patterns
from the human-like distribution during the decoding process, CoPA is able to
produce sentences that are less discernible by text detectors. Our theoretical
analysis suggests the superiority of the proposed attack. Extensive experiments
validate the effectiveness of CoPA in fooling text detectors across various
scenarios.


http://arxiv.org/abs/2505.15336
My Face Is Mine, Not Yours: Facial Protection Against Diffusion Model Face Swapping. (2%)
Hon Ming Yam; Zhongliang Guo; Chun Pong Lau
  The proliferation of diffusion-based deepfake technologies poses significant
risks for unauthorized and unethical facial image manipulation. While
traditional countermeasures have primarily focused on passive detection
methods, this paper introduces a novel proactive defense strategy through
adversarial attacks that preemptively protect facial images from being
exploited by diffusion-based deepfake systems. Existing adversarial protection
methods predominantly target conventional generative architectures (GANs, AEs,
VAEs) and fail to address the unique challenges presented by diffusion models,
which have become the predominant framework for high-quality facial deepfakes.
Current diffusion-specific adversarial approaches are limited by their reliance
on specific model architectures and weights, rendering them ineffective against
the diverse landscape of diffusion-based deepfake implementations.
Additionally, they typically employ global perturbation strategies that
inadequately address the region-specific nature of facial manipulation in
deepfakes.


http://arxiv.org/abs/2505.15861
P3Net: Progressive and Periodic Perturbation for Semi-Supervised Medical Image Segmentation. (1%)
Zhenyan Yao; Miao Zhang; Lanhu Wu; Yongri Piao; Feng Tian; Weibing Sun; Huchuan Lu
  Perturbation with diverse unlabeled data has proven beneficial for
semi-supervised medical image segmentation (SSMIS). While many works have
successfully used various perturbation techniques, a deeper understanding of
learning perturbations is needed. Excessive or inappropriate perturbation can
have negative effects, so we aim to address two challenges: how to use
perturbation mechanisms to guide the learning of unlabeled data through labeled
data, and how to ensure accurate predictions in boundary regions. Inspired by
human progressive and periodic learning, we propose a progressive and periodic
perturbation mechanism (P3M) and a boundary-focused loss. P3M enables dynamic
adjustment of perturbations, allowing the model to gradually learn them. Our
boundary-focused loss encourages the model to concentrate on boundary regions,
enhancing sensitivity to intricate details and ensuring accurate predictions.
Experimental results demonstrate that our method achieves state-of-the-art
performance on two 2D and 3D datasets. Moreover, P3M is extendable to other
methods, and the proposed loss serves as a universal tool for improving
existing methods, highlighting the scalability and applicability of our
approach.


http://arxiv.org/abs/2505.16159
Why Can Accurate Models Be Learned from Inaccurate Annotations? (1%)
Chongjie Si; Yidan Cui; Fuchao Yang; Xiaokang Yang; Wei Shen
  Learning from inaccurate annotations has gained significant attention due to
the high cost of precise labeling. However, despite the presence of erroneous
labels, models trained on noisy data often retain the ability to make accurate
predictions. This intriguing phenomenon raises a fundamental yet largely
unexplored question: why models can still extract correct label information
from inaccurate annotations remains unexplored. In this paper, we conduct a
comprehensive investigation into this issue. By analyzing weight matrices from
both empirical and theoretical perspectives, we find that label inaccuracy
primarily accumulates noise in lower singular components and subtly perturbs
the principal subspace. Within a certain range, the principal subspaces of
weights trained on inaccurate labels remain largely aligned with those learned
from clean labels, preserving essential task-relevant information. We formally
prove that the angles of principal subspaces exhibit minimal deviation under
moderate label inaccuracy, explaining why models can still generalize
effectively. Building on these insights, we propose LIP, a lightweight plug-in
designed to help classifiers retain principal subspace information while
mitigating noise induced by label inaccuracy. Extensive experiments on tasks
with various inaccuracy conditions demonstrate that LIP consistently enhances
the performance of existing algorithms. We hope our findings can offer valuable
theoretical and practical insights to understand of model robustness under
inaccurate supervision.


http://arxiv.org/abs/2505.15431
Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought. (1%)
Ao Liu; Botong Zhou; Can Xu; Chayse Zhou; ChenChen Zhang; Chengcheng Xu; Chenhao Wang; Decheng Wu; Dengpeng Wu; Dian Jiao; Dong Du; Dong Wang; Feng Zhang; Fengzong Lian; Guanghui Xu; Guanwei Zhang; Hai Wang; Haipeng Luo; Han Hu; Huilin Xu; Jiajia Wu; Jianchen Zhu; Jianfeng Yan; Jiaqi Zhu; Jihong Zhang; Jinbao Xue; Jun Xia; Junqiang Zheng; Kai Liu; Kai Zhang; Kai Zheng; Kejiao Li; Keyao Wang; Lan Jiang; Lixin Liu; Lulu Wu; Mengyuan Huang; Peijie Yu; Peiqi Wang; Qian Wang; Qianbiao Xiang; Qibin Liu; Qingfeng Sun; Richard Guo; Ruobing Xie; Saiyong Yang; Shaohua Chen; Shihui Hu; Shuai Li; Shuaipeng Li; Shuang Chen; Suncong Zheng; Tao Yang; Tian Zhang; Tinghao Yu; Weidong Han; Weijie Liu; Weijin Zhou; Weikang Wang; Wesleye Chen; Xiao Feng; Xiaoqin Ren; Xingwu Sun; Xiong Kuang; Xuemeng Huang; Xun Cao; Yanfeng Chen; Yang Du; Yang Zhen; Yangyu Tao; Yaping Deng; Yi Shen; Yigeng Hong; Yiqi Chen; Yiqing Huang; Yuchi Deng; Yue Mao; Yulong Wang; Yuyuan Zeng; Zenan Xu; Zhanhui Kang; Zhe Zhao; ZhenXiang Yan; Zheng Fang; Zhichao Hu; Zhongzhi Chen; Zhuoyu Li; Zongwei Li; Alex Yan; Ande Liang; Baitong Liu; Beiping Pan; Bin Xing; Binghong Wu; Bingxin Qu; Bolin Ni; Boyu Wu; Chen Li; Cheng Jiang; Cheng Zhang; Chengjun Liu; Chengxu Yang; Chiyu Wang; Chong Zha; Daisy Yi; Di Wang; Fanyang Lu; Fei Chen; Feifei Liu; Feng Zheng; Guanghua Yu; Guiyang Li; Guohua Wang; Haisheng Lin; Han Liu; Han Wang; Hao Fei; Hao Lu; Haoqing Jiang; Haoran Sun; Haotian Zhu; Huangjin Dai; Huankui Chen; Huawen Feng; Huihui Cai; Huxin Peng; Jackson Lv; Jiacheng Shi; Jiahao Bu; Jianbo Li; Jianglu Hu; Jiangtao Guan; Jianing Xu; Jianwei Cai; Jiarong Zhang; Jiawei Song; Jie Jiang; Jie Liu; Jieneng Yang; Jihong Zhang; Jin lv; Jing Zhao; Jinjian Li; Jinxing Liu; Jun Zhao; Juntao Guo; Kai Wang; Kan Wu; Lei Fu; Lei He; Lei Wang; Li Liu; Liang Dong; Liya Zhan; Long Cheng; Long Xu; Mao Zheng; Meng Liu; Mengkang Hu; Nanli Chen; Peirui Chen; Peng He; Pengju Pan; Pengzhi Wei; Qi Yang; Qi Yi; Roberts Wang; Rongpeng Chen; Rui Sun; Rui Yang; Ruibin Chen; Ruixu Zhou; Shaofeng Zhang; Sheng Zhang; Shihao Xu; Shuaishuai Chang; Shulin Liu; SiQi Wang; Songjia Feng; Songling Yuan; Tao Zhang; Tianjiao Lang; Tongkai Li; Wei Deng; Wei Li; Weichao Wang; Weigang Zhang; Weixuan Sun; Wen Ouyang; Wenxiang Jiao; Wenzhi Sun; Wenzhuo Jia; Xiang Zhang; Xiangyu He; Xianshun Ren; XiaoYing Zhu; Xiaolong Guo; Xiaoxue Li; Xiaoyu Ma; Xican Lu; Xinhua Feng; Xinting Huang; Xinyu Guan; Xirui Li; Xu Zhang; Xudong Gao; Xun Luo; Xuxiang Qi; Yangkun Chen; Yangyu Tao; Yanling Xiao; Yantao Mai; Yanze Chen; Yao Ding; Yeting Yang; YiFan Song; Yifan Yang; Yijiao Zhu; Yinhe Wu; Yixian Liu; Yong Yang; Yuanjun Cai; Yuanlin Tu; Yue Zhang; Yufei Huang; Yuhang Zhou; Yuhao Jiang; Yuhong Liu; Yuhui Hu; Yujin Lin; Yun Yang; Yunhao Wang; Yusong Zhang; Zekun Wu; Zelong Zhang; Zhan Yu; Zhaoliang Yang; Zhe Zhao; Zheng Li; Zhenyu Huang; Zhiguang Liu; Zhijiang Xu; Zhiqing Kui; Zhiyin Zeng; Zhiyuan Xiong; Zhuo Han; Zifan Wu; Zigang Geng; Zilong Zhao; Ziyan Tang; Ziyuan Zhu; Zonglei Zhu; Zhijiang Xu
  As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS,
a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It
synergistically combines Mamba's long-sequence processing efficiency with
Transformer's superior contextual understanding. Hunyuan-TurboS features an
adaptive long-short chain-of-thought (CoT) mechanism, dynamically switching
between rapid responses for simple queries and deep "thinking" modes for
complex problems, optimizing computational resources. Architecturally, this 56B
activated (560B total) parameter model employs 128 layers (Mamba2, Attention,
FFN) with an innovative AMF/MF block pattern. Faster Mamba2 ensures linear
complexity, Grouped-Query Attention minimizes KV cache, and FFNs use an MoE
structure. Pre-trained on 16T high-quality tokens, it supports a 256K context
length and is the first industry-deployed large-scale Mamba model. Our
comprehensive post-training strategy enhances capabilities via Supervised
Fine-Tuning (3M instructions), a novel Adaptive Long-short CoT Fusion method,
Multi-round Deliberation Learning for iterative improvement, and a two-stage
Large-scale Reinforcement Learning process targeting STEM and general
instruction-following. Evaluations show strong performance: overall top 7 rank
on LMSYS Chatbot Arena with a score of 1356, outperforming leading models like
Gemini-2.0-Flash-001 (1352) and o4-mini-2025-04-16 (1345). TurboS also achieves
an average of 77.9% across 23 automated benchmarks. Hunyuan-TurboS balances
high performance and efficiency, offering substantial capabilities at lower
inference costs than many reasoning models, establishing a new paradigm for
efficient large-scale pre-trained models.


http://arxiv.org/abs/2505.15174
Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss. (1%)
Bo-Han Lai; Pin-Han Huang; Bo-Han Kung; Shang-Tse Chen
  Lipschitz neural networks are well-known for providing certified robustness
in deep learning. In this paper, we present a novel, efficient Block Reflector
Orthogonal (BRO) layer that enhances the capability of orthogonal layers on
constructing more expressive Lipschitz neural architectures. In addition, by
theoretically analyzing the nature of Lipschitz neural networks, we introduce a
new loss function that employs an annealing mechanism to increase margin for
most data points. This enables Lipschitz models to provide better certified
robustness. By employing our BRO layer and loss function, we design BRONet - a
simple yet effective Lipschitz neural network that achieves state-of-the-art
certified robustness. Extensive experiments and empirical analysis on
CIFAR-10/100, Tiny-ImageNet, and ImageNet validate that our method outperforms
existing baselines. The implementation is available at
\href{https://github.com/ntuaislab/BRONet}{https://github.com/ntuaislab/BRONet}.


http://arxiv.org/abs/2505.15308
BadSR: Stealthy Label Backdoor Attacks on Image Super-Resolution. (1%)
Ji Guo; Xiaolei Wen; Wenbo Jiang; Cheng Huang; Jinjin Li; Hongwei Li
  With the widespread application of super-resolution (SR) in various fields,
researchers have begun to investigate its security. Previous studies have
demonstrated that SR models can also be subjected to backdoor attacks through
data poisoning, affecting downstream tasks. A backdoor SR model generates an
attacker-predefined target image when given a triggered image while producing a
normal high-resolution (HR) output for clean images. However, prior backdoor
attacks on SR models have primarily focused on the stealthiness of poisoned
low-resolution (LR) images while ignoring the stealthiness of poisoned HR
images, making it easy for users to detect anomalous data. To address this
problem, we propose BadSR, which improves the stealthiness of poisoned HR
images. The key idea of BadSR is to approximate the clean HR image and the
pre-defined target image in the feature space while ensuring that modifications
to the clean HR image remain within a constrained range. The poisoned HR images
generated by BadSR can be integrated with existing triggers. To further improve
the effectiveness of BadSR, we design an adversarially optimized trigger and a
backdoor gradient-driven poisoned sample selection method based on a genetic
algorithm. The experimental results show that BadSR achieves a high attack
success rate in various models and data sets, significantly affecting
downstream tasks.


http://arxiv.org/abs/2505.15124
A Survey On Secure Machine Learning. (1%)
Taobo Liao; Taoran Li; Prathamesh Nadkarni
  In this survey, we will explore the interaction between secure multiparty
computation and the area of machine learning. Recent advances in secure
multiparty computation (MPC) have significantly improved its applicability in
the realm of machine learning (ML), offering robust solutions for
privacy-preserving collaborative learning. This review explores key
contributions that leverage MPC to enable multiple parties to engage in ML
tasks without compromising the privacy of their data. The integration of MPC
with ML frameworks facilitates the training and evaluation of models on
combined datasets from various sources, ensuring that sensitive information
remains encrypted throughout the process. Innovations such as specialized
software frameworks and domain-specific languages streamline the adoption of
MPC in ML, optimizing performance and broadening its usage. These frameworks
address both semi-honest and malicious threat models, incorporating features
such as automated optimizations and cryptographic auditing to ensure compliance
and data integrity. The collective insights from these studies highlight MPC's
potential in fostering collaborative yet confidential data analysis, marking a
significant stride towards the realization of secure and efficient
computational solutions in privacy-sensitive industries. This paper
investigates a spectrum of SecureML libraries that includes cryptographic
protocols, federated learning frameworks, and privacy-preserving algorithms. By
surveying the existing literature, this paper aims to examine the efficacy of
these libraries in preserving data privacy, ensuring model confidentiality, and
fortifying ML systems against adversarial attacks. Additionally, the study
explores an innovative application domain for SecureML techniques: the
integration of these methodologies in gaming environments utilizing ML.


http://arxiv.org/abs/2505.15880
Challenger: Affordable Adversarial Driving Video Generation. (1%)
Zhiyuan Xu; Bohan Li; Huan-ang Gao; Mingju Gao; Yong Chen; Ming Liu; Chenxu Yan; Hang Zhao; Shuo Feng; Hao Zhao
  Generating photorealistic driving videos has seen significant progress
recently, but current methods largely focus on ordinary, non-adversarial
scenarios. Meanwhile, efforts to generate adversarial driving scenarios often
operate on abstract trajectory or BEV representations, falling short of
delivering realistic sensor data that can truly stress-test autonomous driving
(AD) systems. In this work, we introduce Challenger, a framework that produces
physically plausible yet photorealistic adversarial driving videos. Generating
such videos poses a fundamental challenge: it requires jointly optimizing over
the space of traffic interactions and high-fidelity sensor observations.
Challenger makes this affordable through two techniques: (1) a physics-aware
multi-round trajectory refinement process that narrows down candidate
adversarial maneuvers, and (2) a tailored trajectory scoring function that
encourages realistic yet adversarial behavior while maintaining compatibility
with downstream video synthesis. As tested on the nuScenes dataset, Challenger
generates a diverse range of aggressive driving scenarios-including cut-ins,
sudden lane changes, tailgating, and blind spot intrusions-and renders them
into multiview photorealistic videos. Extensive evaluations show that these
scenarios significantly increase the collision rate of state-of-the-art
end-to-end AD models (UniAD, VAD, SparseDrive, and DiffusionDrive), and
importantly, adversarial behaviors discovered for one model often transfer to
others.


http://arxiv.org/abs/2505.14453
Robustness Evaluation of Graph-based News Detection Using Network Structural Information. (99%)
Xianghua Zeng; Hao Peng; Angsheng Li
  Although Graph Neural Networks (GNNs) have shown promising potential in fake
news detection, they remain highly vulnerable to adversarial manipulations
within social networks. Existing methods primarily establish connections
between malicious accounts and individual target news to investigate the
vulnerability of graph-based detectors, while they neglect the structural
relationships surrounding targets, limiting their effectiveness in robustness
evaluation. In this work, we propose a novel Structural Information
principles-guided Adversarial Attack Framework, namely SI2AF, which effectively
challenges graph-based detectors and further probes their detection robustness.
Specifically, structural entropy is introduced to quantify the dynamic
uncertainty in social engagements and identify hierarchical communities that
encompass all user accounts and news posts. An influence metric is presented to
measure each account's probability of engaging in random interactions,
facilitating the design of multiple agents that manage distinct malicious
accounts. For each target news, three attack strategies are developed through
multi-agent collaboration within the associated subgraph to optimize evasion
against black-box detectors. By incorporating the adversarial manipulations
generated by SI2AF, we enrich the original network structure and refine
graph-based detectors to improve their robustness against adversarial attacks.
Extensive evaluations demonstrate that SI2AF significantly outperforms
state-of-the-art baselines in attack effectiveness with an average improvement
of 16.71%, and enhances GNN-based detection robustness by 41.54% on average.


http://arxiv.org/abs/2505.14463
Adverseness vs. Equilibrium: Exploring Graph Adversarial Resilience through Dynamic Equilibrium. (92%)
Xinxin Fan; Wenxiong Chen; Mengfan Li; Wenqi Wei; Ling Liu
  Adversarial attacks to graph analytics are gaining increased attention. To
date, two lines of countermeasures have been proposed to resist various graph
adversarial attacks from the perspectives of either graph per se or graph
neural networks. Nevertheless, a fundamental question lies in whether there
exists an intrinsic adversarial resilience state within a graph regime and how
to find out such a critical state if exists. This paper contributes to tackle
the above research questions from three unique perspectives: i) we regard the
process of adversarial learning on graph as a complex multi-object dynamic
system, and model the behavior of adversarial attack; ii) we propose a
generalized theoretical framework to show the existence of critical adversarial
resilience state; and iii) we develop a condensed one-dimensional function to
capture the dynamic variation of graph regime under perturbations, and pinpoint
the critical state through solving the equilibrium point of dynamic system.
Multi-facet experiments are conducted to show our proposed approach can
significantly outperform the state-of-the-art defense methods under five
commonly-used real-world datasets and three representative attacks.


http://arxiv.org/abs/2505.14021
Adversarial Training from Mean Field Perspective. (88%)
Soichiro Kumano; Hiroshi Kera; Toshihiko Yamasaki
  Although adversarial training is known to be effective against adversarial
examples, training dynamics are not well understood. In this study, we present
the first theoretical analysis of adversarial training in random deep neural
networks without any assumptions on data distributions. We introduce a new
theoretical framework based on mean field theory, which addresses the
limitations of existing mean field-based approaches. Based on this framework,
we derive (empirically tight) upper bounds of $\ell_q$ norm-based adversarial
loss with $\ell_p$ norm-based adversarial examples for various values of $p$
and $q$. Moreover, we prove that networks without shortcuts are generally not
adversarially trainable and that adversarial training reduces network capacity.
We also show that network width alleviates these issues. Furthermore, we
present the various impacts of the input and output dimensions on the upper
bounds and time evolution of the weight variance.


http://arxiv.org/abs/2505.14024
FedGraM: Defending Against Untargeted Attacks in Federated Learning via Embedding Gram Matrix. (86%)
Di Wu; Qian Li; Heng Yang; Yong Han
  Federated Learning (FL) enables geographically distributed clients to
collaboratively train machine learning models by sharing only their local
models, ensuring data privacy. However, FL is vulnerable to untargeted attacks
that aim to degrade the global model's performance on the underlying data
distribution. Existing defense mechanisms attempt to improve FL's resilience
against such attacks, but their effectiveness is limited in practical FL
environments due to data heterogeneity. On the contrary, we aim to detect and
remove the attacks to mitigate their impact. Generalization contribution plays
a crucial role in distinguishing untargeted attacks. Our observations indicate
that, with limited data, the divergence between embeddings representing
different classes provides a better measure of generalization than direct
accuracy. In light of this, we propose a novel robust aggregation method,
FedGraM, designed to defend against untargeted attacks in FL. The server
maintains an auxiliary dataset containing one sample per class to support
aggregation. This dataset is fed to the local models to extract embeddings.
Then, the server calculates the norm of the Gram Matrix of the embeddings for
each local model. The norm serves as an indicator of each model's inter-class
separation capability in the embedding space. FedGraM identifies and removes
potentially malicious models by filtering out those with the largest norms,
then averages the remaining local models to form the global model. We conduct
extensive experiments to evaluate the performance of FedGraM. Our empirical
results show that with limited data samples used to construct the auxiliary
dataset, FedGraM achieves exceptional performance, outperforming
state-of-the-art defense methods.


http://arxiv.org/abs/2505.14286
Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMs. (80%)
Rao Ma; Mengjie Qian; Vyas Raina; Mark Gales; Kate Knill
  The combination of pre-trained speech encoders with large language models has
enabled the development of speech LLMs that can handle a wide range of spoken
language processing tasks. While these models are powerful and flexible, this
very flexibility may make them more vulnerable to adversarial attacks. To
examine the extent of this problem, in this work we investigate universal
acoustic adversarial attacks on speech LLMs. Here a fixed, universal,
adversarial audio segment is prepended to the original input audio. We
initially investigate attacks that cause the model to either produce no output
or to perform a modified task overriding the original prompt. We then extend
the nature of the attack to be selective so that it activates only when
specific input attributes, such as a speaker gender or spoken language, are
present. Inputs without the targeted attribute should be unaffected, allowing
fine-grained control over the model outputs. Our findings reveal critical
vulnerabilities in Qwen2-Audio and Granite-Speech and suggest that similar
speech LLMs may be susceptible to universal adversarial attacks. This
highlights the need for more robust training strategies and improved resistance
to adversarial attacks.


http://arxiv.org/abs/2505.14323
Vulnerability of Transfer-Learned Neural Networks to Data Reconstruction Attacks in Small-Data Regime. (80%)
Tomasz Maciążek; Robert Allison
  Training data reconstruction attacks enable adversaries to recover portions
of a released model's training data. We consider the attacks where a
reconstructor neural network learns to invert the (random) mapping between
training data and model weights. Prior work has shown that an informed
adversary with access to released model's weights and all but one training data
point can achieve high-quality reconstructions in this way. However,
differential privacy can defend against such an attack with little to no loss
in model's utility when the amount of training data is sufficiently large. In
this work we consider a more realistic adversary who only knows the
distribution from which a small training dataset has been sampled and who
attacks a transfer-learned neural network classifier that has been trained on
this dataset. We exhibit an attack that works in this realistic threat model
and demonstrate that in the small-data regime it cannot be defended against by
DP-SGD without severely damaging the classifier accuracy. This raises
significant concerns about the use of such transfer-learned classifiers when
protection of training-data is paramount. We demonstrate the effectiveness and
robustness of our attack on VGG, EfficientNet and ResNet image classifiers
transfer-learned on MNIST, CIFAR-10 and CelebA respectively. Additionally, we
point out that the commonly used (true-positive) reconstruction success rate
metric fails to reliably quantify the actual reconstruction effectiveness.
Instead, we make use of the Neyman-Pearson lemma to construct the receiver
operating characteristic curve and consider the associated true-positive
reconstruction rate at a fixed level of the false-positive reconstruction rate.


http://arxiv.org/abs/2505.14534
Lessons from Defending Gemini Against Indirect Prompt Injections. (76%)
Chongyang Shi; Sharon Lin; Shuang Song; Jamie Hayes; Ilia Shumailov; Itay Yona; Juliette Pluto; Aneesh Pappu; Christopher A. Choquette-Choo; Milad Nasr; Chawin Sitawarin; Gena Gibson; Andreas Terzis; John "Four" Flynn
  Gemini is increasingly used to perform tasks on behalf of users, where
function-calling and tool-use capabilities enable the model to access user
data. Some tools, however, require access to untrusted data introducing risk.
Adversaries can embed malicious instructions in untrusted data which cause the
model to deviate from the user's expectations and mishandle their data or
permissions. In this report, we set out Google DeepMind's approach to
evaluating the adversarial robustness of Gemini models and describe the main
lessons learned from the process. We test how Gemini performs against a
sophisticated adversary through an adversarial evaluation framework, which
deploys a suite of adaptive attack techniques to run continuously against past,
current, and future versions of Gemini. We describe how these ongoing
evaluations directly help make Gemini more resilient against manipulation.


http://arxiv.org/abs/2505.14316
Exploring Jailbreak Attacks on LLMs through Intent Concealment and Diversion. (70%)
Tiehan Cui; Yanxu Mao; Peipei Liu; Congying Liu; Datao You
  Although large language models (LLMs) have achieved remarkable advancements,
their security remains a pressing concern. One major threat is jailbreak
attacks, where adversarial prompts bypass model safeguards to generate harmful
or objectionable content. Researchers study jailbreak attacks to understand
security and robustness of LLMs. However, existing jailbreak attack methods
face two main challenges: (1) an excessive number of iterative queries, and (2)
poor generalization across models. In addition, recent jailbreak evaluation
datasets focus primarily on question-answering scenarios, lacking attention to
text generation tasks that require accurate regeneration of toxic content. To
tackle these challenges, we propose two contributions: (1) ICE, a novel
black-box jailbreak method that employs Intent Concealment and divErsion to
effectively circumvent security constraints. ICE achieves high attack success
rates (ASR) with a single query, significantly improving efficiency and
transferability across different models. (2) BiSceneEval, a comprehensive
dataset designed for assessing LLM robustness in question-answering and
text-generation tasks. Experimental results demonstrate that ICE outperforms
existing jailbreak techniques, revealing critical vulnerabilities in current
defense mechanisms. Our findings underscore the necessity of a hybrid security
strategy that integrates predefined security mechanisms with real-time semantic
decomposition to enhance the security of LLMs.


http://arxiv.org/abs/2505.17089
Trust Me, I Can Handle It: Self-Generated Adversarial Scenario Extrapolation for Robust Language Models. (54%)
Md Rafi Ur Rashid; Vishnu Asutosh Dasu; Ye Wang; Gang Tan; Shagufta Mehnaz
  Large Language Models (LLMs) exhibit impressive capabilities, but remain
susceptible to a growing spectrum of safety risks, including jailbreaks, toxic
content, hallucinations, and bias. Existing defenses often address only a
single threat type or resort to rigid outright rejection, sacrificing user
experience and failing to generalize across diverse and novel attacks. This
paper introduces Adversarial Scenario Extrapolation (ASE), a novel
inference-time computation framework that leverages Chain-of-Thought (CoT)
reasoning to simultaneously enhance LLM robustness and seamlessness. ASE guides
the LLM through a self-generative process of contemplating potential
adversarial scenarios and formulating defensive strategies before generating a
response to the user query. Comprehensive evaluation on four adversarial
benchmarks with four latest LLMs shows that ASE achieves near-zero jailbreak
attack success rates and minimal toxicity, while slashing outright rejections
to <4%. ASE outperforms six state-of-the-art defenses in
robustness-seamlessness trade-offs, with 92-99% accuracy on adversarial Q&A and
4-10x lower bias scores. By transforming adversarial perception into an
intrinsic cognitive process, ASE sets a new paradigm for secure and natural
human-AI interaction.


http://arxiv.org/abs/2505.14814
GraphemeAug: A Systematic Approach to Synthesized Hard Negative Keyword Spotting Examples. (47%)
Harry Zhang; Kurt Partridge; Pai Zhu; Neng Chen; Hyun Jin Park; Dhruuv Agarwal; Quan Wang
  Spoken Keyword Spotting (KWS) is the task of distinguishing between the
presence and absence of a keyword in audio. The accuracy of a KWS model hinges
on its ability to correctly classify examples close to the keyword and
non-keyword boundary. These boundary examples are often scarce in training
data, limiting model performance. In this paper, we propose a method to
systematically generate adversarial examples close to the decision boundary by
making insertion/deletion/substitution edits on the keyword's graphemes. We
evaluate this technique on held-out data for a popular keyword and show that
the technique improves AUC on a dataset of synthetic hard negatives by 61%
while maintaining quality on positives and ambient negative audio data.


http://arxiv.org/abs/2505.14103
AudioJailbreak: Jailbreak Attacks against End-to-End Large Audio-Language Models. (31%)
Guangke Chen; Fu Song; Zhe Zhao; Xiaojun Jia; Yang Liu; Yanchen Qiao; Weizhe Zhang
  Jailbreak attacks to Large audio-language models (LALMs) are studied
recently, but they achieve suboptimal effectiveness, applicability, and
practicability, particularly, assuming that the adversary can fully manipulate
user prompts. In this work, we first conduct an extensive experiment showing
that advanced text jailbreak attacks cannot be easily ported to end-to-end
LALMs via text-to speech (TTS) techniques. We then propose AudioJailbreak, a
novel audio jailbreak attack, featuring (1) asynchrony: the jailbreak audio
does not need to align with user prompts in the time axis by crafting suffixal
jailbreak audios; (2) universality: a single jailbreak perturbation is
effective for different prompts by incorporating multiple prompts into
perturbation generation; (3) stealthiness: the malicious intent of jailbreak
audios will not raise the awareness of victims by proposing various intent
concealment strategies; and (4) over-the-air robustness: the jailbreak audios
remain effective when being played over the air by incorporating the
reverberation distortion effect with room impulse response into the generation
of the perturbations. In contrast, all prior audio jailbreak attacks cannot
offer asynchrony, universality, stealthiness, or over-the-air robustness.
Moreover, AudioJailbreak is also applicable to the adversary who cannot fully
manipulate user prompts, thus has a much broader attack scenario. Extensive
experiments with thus far the most LALMs demonstrate the high effectiveness of
AudioJailbreak. We highlight that our work peeks into the security implications
of audio jailbreak attacks against LALMs, and realistically fosters improving
their security robustness. The implementation and audio samples are available
at our website https://audiojailbreak.github.io/AudioJailbreak.


http://arxiv.org/abs/2505.14289
EVA: Red-Teaming GUI Agents via Evolving Indirect Prompt Injection. (22%)
Yijie Lu; Tianjie Ju; Manman Zhao; Xinbei Ma; Yuan Guo; ZhuoSheng Zhang
  As multimodal agents are increasingly trained to operate graphical user
interfaces (GUIs) to complete user tasks, they face a growing threat from
indirect prompt injection, attacks in which misleading instructions are
embedded into the agent's visual environment, such as popups or chat messages,
and misinterpreted as part of the intended task. A typical example is
environmental injection, in which GUI elements are manipulated to influence
agent behavior without directly modifying the user prompt. To address these
emerging attacks, we propose EVA, a red teaming framework for indirect prompt
injection which transforms the attack into a closed loop optimization by
continuously monitoring an agent's attention distribution over the GUI and
updating adversarial cues, keywords, phrasing, and layout, in response.
Compared with prior one shot methods that generate fixed prompts without regard
for how the model allocates visual attention, EVA dynamically adapts to
emerging attention hotspots, yielding substantially higher attack success rates
and far greater transferability across diverse GUI scenarios. We evaluate EVA
on six widely used generalist and specialist GUI agents in realistic settings
such as popup manipulation, chat based phishing, payments, and email
composition. Experimental results show that EVA substantially improves success
rates over static baselines. Under goal agnostic constraints, where the
attacker does not know the agent's task intent, EVA still discovers effective
patterns. Notably, we find that injection styles transfer well across models,
revealing shared behavioral biases in GUI agents. These results suggest that
evolving indirect prompt injection is a powerful tool not only for red teaming
agents, but also for uncovering common vulnerabilities in their multimodal
decision making.


http://arxiv.org/abs/2505.14042
Adversarially Pretrained Transformers may be Universally Robust In-Context Learners. (13%)
Soichiro Kumano; Hiroshi Kera; Toshihiko Yamasaki
  Adversarial training is one of the most effective adversarial defenses, but
it incurs a high computational cost. In this study, we show that transformers
adversarially pretrained on diverse tasks can serve as robust foundation models
and eliminate the need for adversarial training in downstream tasks.
Specifically, we theoretically demonstrate that through in-context learning, a
single adversarially pretrained transformer can robustly generalize to multiple
unseen tasks without any additional training, i.e., without any parameter
updates. This robustness stems from the model's focus on robust features and
its resistance to attacks that exploit non-predictive features. Besides these
positive findings, we also identify several limitations. Under certain
conditions (though unrealistic), no universally robust single-layer
transformers exist. Moreover, robust transformers exhibit an
accuracy--robustness trade-off and require a large number of in-context
demonstrations. The code is available at
https://github.com/s-kumano/universally-robust-in-context-learner.


http://arxiv.org/abs/2505.14967
Anomaly Detection Based on Critical Paths for Deep Neural Networks. (13%)
Fangzhen Zhao; Chenyi Zhang; Naipeng Dong; Ming Li; Jinxiao Shan
  Deep neural networks (DNNs) are notoriously hard to understand and difficult
to defend. Extracting representative paths (including the neuron activation
values and the connections between neurons) from DNNs using software
engineering approaches has recently shown to be a promising approach in
interpreting the decision making process of blackbox DNNs, as the extracted
paths are often effective in capturing essential features. With this in mind,
this work investigates a novel approach that extracts critical paths from DNNs
and subsequently applies the extracted paths for the anomaly detection task,
based on the observation that outliers and adversarial inputs do not usually
induce the same activation pattern on those paths as normal (in-distribution)
inputs.
  In our approach, we first identify critical detection paths via genetic
evolution and mutation. Since different paths in a DNN often capture different
features for the same target class, we ensemble detection results from multiple
paths by integrating random subspace sampling and a voting mechanism. Compared
with state-of-the-art methods, our experimental results suggest that our method
not only outperforms them, but it is also suitable for the detection of a broad
range of anomaly types with high accuracy.


http://arxiv.org/abs/2505.14368
Is Your Prompt Safe? Investigating Prompt Injection Attacks Against Open-Source LLMs. (10%)
Jiawen Wang; Pritha Gupta; Ivan Habernal; Eyke Hüllermeier
  Recent studies demonstrate that Large Language Models (LLMs) are vulnerable
to different prompt-based attacks, generating harmful content or sensitive
information. Both closed-source and open-source LLMs are underinvestigated for
these attacks. This paper studies effective prompt injection attacks against
the $\mathbf{14}$ most popular open-source LLMs on five attack benchmarks.
Current metrics only consider successful attacks, whereas our proposed Attack
Success Probability (ASP) also captures uncertainty in the model's response,
reflecting ambiguity in attack feasibility. By comprehensively analyzing the
effectiveness of prompt injection attacks, we propose a simple and effective
hypnotism attack; results show that this attack causes aligned language models,
including Stablelm2, Mistral, Openchat, and Vicuna, to generate objectionable
behaviors, achieving around $90$% ASP. They also indicate that our ignore
prefix attacks can break all $\mathbf{14}$ open-source LLMs, achieving over
$60$% ASP on a multi-categorical dataset. We find that moderately well-known
LLMs exhibit higher vulnerability to prompt injection attacks, highlighting the
need to raise public awareness and prioritize efficient mitigation strategies.


http://arxiv.org/abs/2505.14185
Safety Subspaces are Not Distinct: A Fine-Tuning Case Study. (10%)
Kaustubh Ponkshe; Shaan Shah; Raghav Singhal; Praneeth Vepakomma
  Large Language Models (LLMs) rely on safety alignment to produce socially
acceptable responses. This is typically achieved through instruction tuning and
reinforcement learning from human feedback. However, this alignment is known to
be brittle: further fine-tuning, even on benign or lightly contaminated data,
can degrade safety and reintroduce harmful behaviors. A growing body of work
suggests that alignment may correspond to identifiable geometric directions in
weight space, forming subspaces that could, in principle, be isolated or
preserved to defend against misalignment. In this work, we conduct a
comprehensive empirical study of this geometric perspective. We examine whether
safety-relevant behavior is concentrated in specific subspaces, whether it can
be separated from general-purpose learning, and whether harmfulness arises from
distinguishable patterns in internal representations. Across both parameter and
activation space, our findings are consistent: subspaces that amplify safe
behaviors also amplify unsafe ones, and prompts with different safety
implications activate overlapping representations. We find no evidence of a
subspace that selectively governs safety. These results challenge the
assumption that alignment is geometrically localized. Rather than residing in
distinct directions, safety appears to emerge from entangled, high-impact
components of the model's broader learning dynamics. This suggests that
subspace-based defenses may face fundamental limitations and underscores the
need for alternative strategies to preserve alignment under continued training.
We corroborate these findings through multiple experiments on five open-source
LLMs. Our code is publicly available at:
https://github.com/CERT-Lab/safety-subspaces.


http://arxiv.org/abs/2505.14531
SifterNet: A Generalized and Model-Agnostic Trigger Purification Approach. (2%)
Shaoye Luo; Xinxin Fan; Quanliang Jing; Chi Lin; Mengfan Li; Yunfeng Lu; Yongjun Xu
  Aiming at resisting backdoor attacks in convolution neural networks and
vision Transformer-based large model, this paper proposes a generalized and
model-agnostic trigger-purification approach resorting to the classic Ising
model. To date, existing trigger detection/removal studies usually require to
know the detailed knowledge of target model in advance, access to a large
number of clean samples or even model-retraining authorization, which brings
the huge inconvenience for practical applications, especially inaccessible to
target model. An ideal countermeasure ought to eliminate the implanted trigger
without regarding whatever the target models are. To this end, a lightweight
and black-box defense approach SifterNet is proposed through leveraging the
memorization-association functionality of Hopfield network, by which the
triggers of input samples can be effectively purified in a proper manner. The
main novelty of our proposed approach lies in the introduction of ideology of
Ising model. Extensive experiments also validate the effectiveness of our
approach in terms of proper trigger purification and high accuracy achievement,
and compared to the state-of-the-art baselines under several commonly-used
datasets, our SiferNet has a significant superior performance.


http://arxiv.org/abs/2505.13910
ShortcutProbe: Probing Prediction Shortcuts for Learning Robust Models. (1%)
Guangtao Zheng; Wenqian Ye; Aidong Zhang
  Deep learning models often achieve high performance by inadvertently learning
spurious correlations between targets and non-essential features. For example,
an image classifier may identify an object via its background that spuriously
correlates with it. This prediction behavior, known as spurious bias, severely
degrades model performance on data that lacks the learned spurious
correlations. Existing methods on spurious bias mitigation typically require a
variety of data groups with spurious correlation annotations called group
labels. However, group labels require costly human annotations and often fail
to capture subtle spurious biases such as relying on specific pixels for
predictions. In this paper, we propose a novel post hoc spurious bias
mitigation framework without requiring group labels. Our framework, termed
ShortcutProbe, identifies prediction shortcuts that reflect potential
non-robustness in predictions in a given model's latent space. The model is
then retrained to be invariant to the identified prediction shortcuts for
improved robustness. We theoretically analyze the effectiveness of the
framework and empirically demonstrate that it is an efficient and practical
tool for improving a model's robustness to spurious bias on diverse datasets.


http://arxiv.org/abs/2505.17092
Covert Attacks on Machine Learning Training in Passively Secure MPC. (1%)
Matthew Jagielski; Daniel Escudero; Rahul Rachuri; Peter Scholl
  Secure multiparty computation (MPC) allows data owners to train machine
learning models on combined data while keeping the underlying training data
private. The MPC threat model either considers an adversary who passively
corrupts some parties without affecting their overall behavior, or an adversary
who actively modifies the behavior of corrupt parties. It has been argued that
in some settings, active security is not a major concern, partly because of the
potential risk of reputation loss if a party is detected cheating.
  In this work we show explicit, simple, and effective attacks that an active
adversary can run on existing passively secure MPC training protocols, while
keeping essentially zero risk of the attack being detected. The attacks we show
can compromise both the integrity and privacy of the model, including attacks
reconstructing exact training data. Our results challenge the belief that a
threat model that does not include malicious behavior by the involved parties
may be reasonable in the context of PPML, motivating the use of actively secure
protocols for training.


http://arxiv.org/abs/2505.14418
Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agents. (1%)
Pengzhou Cheng; Haowen Hu; Zheng Wu; Zongru Wu; Tianjie Ju; Zhuosheng Zhang; Gongshen Liu
  Graphical user interface (GUI) agents powered by multimodal large language
models (MLLMs) have shown greater promise for human-interaction. However, due
to the high fine-tuning cost, users often rely on open-source GUI agents or
APIs offered by AI providers, which introduces a critical but underexplored
supply chain threat: backdoor attacks. In this work, we first unveil that
MLLM-powered GUI agents naturally expose multiple interaction-level triggers,
such as historical steps, environment states, and task progress. Based on this
observation, we introduce AgentGhost, an effective and stealthy framework for
red-teaming backdoor attacks. Specifically, we first construct composite
triggers by combining goal and interaction levels, allowing GUI agents to
unintentionally activate backdoors while ensuring task utility. Then, we
formulate backdoor injection as a Min-Max optimization problem that uses
supervised contrastive learning to maximize the feature difference across
sample classes at the representation space, improving flexibility of the
backdoor. Meanwhile, it adopts supervised fine-tuning to minimize the
discrepancy between backdoor and clean behavior generation, enhancing
effectiveness and utility. Extensive evaluations of various agent models in two
established mobile benchmarks show that AgentGhost is effective and generic,
with attack accuracy that reaches 99.7\% on three attack objectives, and shows
stealthiness with only 1\% utility degradation. Furthermore, we tailor a
defense method against AgentGhost that reduces the attack accuracy to 22.1\%.
Our code is available at \texttt{anonymous}.


http://arxiv.org/abs/2505.13280
FlowPure: Continuous Normalizing Flows for Adversarial Purification. (99%)
Elias Collaert; Abel Rodríguez; Sander Joos; Lieven Desmet; Vera Rimmer
  Despite significant advancements in the area, adversarial robustness remains
a critical challenge in systems employing machine learning models. The removal
of adversarial perturbations at inference time, known as adversarial
purification, has emerged as a promising defense strategy. To achieve this,
state-of-the-art methods leverage diffusion models that inject Gaussian noise
during a forward process to dilute adversarial perturbations, followed by a
denoising step to restore clean samples before classification. In this work, we
propose FlowPure, a novel purification method based on Continuous Normalizing
Flows (CNFs) trained with Conditional Flow Matching (CFM) to learn mappings
from adversarial examples to their clean counterparts. Unlike prior
diffusion-based approaches that rely on fixed noise processes, FlowPure can
leverage specific attack knowledge to improve robustness under known threats,
while also supporting a more general stochastic variant trained on Gaussian
perturbations for settings where such knowledge is unavailable. Experiments on
CIFAR-10 and CIFAR-100 demonstrate that our method outperforms state-of-the-art
purification-based defenses in preprocessor-blind and white-box scenarios, and
can do so while fully preserving benign accuracy in the former. Moreover, our
results show that not only is FlowPure a highly effective purifier but it also
holds a strong potential for adversarial detection, identifying
preprocessor-blind PGD samples with near-perfect accuracy.


http://arxiv.org/abs/2505.13348
Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attacks. (87%)
Narek Maloyan; Bislan Ashinov; Dmitry Namiot
  Large Language Models (LLMs) are increasingly employed as evaluators
(LLM-as-a-Judge) for assessing the quality of machine-generated text. This
paradigm offers scalability and cost-effectiveness compared to human
annotation. However, the reliability and security of such systems, particularly
their robustness against adversarial manipulations, remain critical concerns.
This paper investigates the vulnerability of LLM-as-a-Judge architectures to
prompt-injection attacks, where malicious inputs are designed to compromise the
judge's decision-making process. We formalize two primary attack strategies:
Comparative Undermining Attack (CUA), which directly targets the final decision
output, and Justification Manipulation Attack (JMA), which aims to alter the
model's generated reasoning. Using the Greedy Coordinate Gradient (GCG)
optimization method, we craft adversarial suffixes appended to one of the
responses being compared. Experiments conducted on the MT-Bench Human Judgments
dataset with open-source instruction-tuned LLMs (Qwen2.5-3B-Instruct and
Falcon3-3B-Instruct) demonstrate significant susceptibility. The CUA achieves
an Attack Success Rate (ASR) exceeding 30\%, while JMA also shows notable
effectiveness. These findings highlight substantial vulnerabilities in current
LLM-as-a-Judge systems, underscoring the need for robust defense mechanisms and
further research into adversarial evaluation and trustworthiness in LLM-based
assessment frameworks.


http://arxiv.org/abs/2505.13023
Anti-Inpainting: A Proactive Defense against Malicious Diffusion-based Inpainters under Unknown Conditions. (87%)
Yimao Guo; Zuomin Qu; Wei Lu; Xiangyang Luo
  As diffusion-based malicious image manipulation becomes increasingly
prevalent, multiple proactive defense methods are developed to safeguard images
against unauthorized tampering. However, most proactive defense methods only
can safeguard images against manipulation under known conditions, and fail to
protect images from manipulations guided by tampering conditions crafted by
malicious users. To tackle this issue, we propose Anti-Inpainting, a proactive
defense method that achieves adequate protection under unknown conditions
through a triple mechanism to address this challenge. Specifically, a
multi-level deep feature extractor is presented to obtain intricate features
during the diffusion denoising process to improve protective effectiveness. We
design multi-scale semantic-preserving data augmentation to enhance the
transferability of adversarial perturbations across unknown conditions by
multi-scale transformations while preserving semantic integrity. In addition,
we propose a selection-based distribution deviation optimization strategy to
improve the protection of adversarial perturbation against manipulation under
diverse random seeds. Extensive experiments indicate the proactive defensive
performance of Anti-Inpainting against diffusion-based inpainters guided by
unknown conditions in InpaintGuardBench and CelebA-HQ. At the same time, we
also demonstrate the proposed approach's robustness under various image
purification methods and its transferability across different versions of
diffusion models.


http://arxiv.org/abs/2505.12686
RoVo: Robust Voice Protection Against Unauthorized Speech Synthesis with Embedding-Level Perturbations. (81%)
Seungmin Kim; Sohee Park; Donghyun Kim; Jisu Lee; Daeseon Choi
  With the advancement of AI-based speech synthesis technologies such as Deep
Voice, there is an increasing risk of voice spoofing attacks, including voice
phishing and fake news, through unauthorized use of others' voices. Existing
defenses that inject adversarial perturbations directly into audio signals have
limited effectiveness, as these perturbations can easily be neutralized by
speech enhancement methods. To overcome this limitation, we propose RoVo
(Robust Voice), a novel proactive defense technique that injects adversarial
perturbations into high-dimensional embedding vectors of audio signals,
reconstructing them into protected speech. This approach effectively defends
against speech synthesis attacks and also provides strong resistance to speech
enhancement models, which represent a secondary attack threat.
  In extensive experiments, RoVo increased the Defense Success Rate (DSR) by
over 70% compared to unprotected speech, across four state-of-the-art speech
synthesis models. Specifically, RoVo achieved a DSR of 99.5% on a commercial
speaker-verification API, effectively neutralizing speech synthesis attack.
Moreover, RoVo's perturbations remained robust even under strong speech
enhancement conditions, outperforming traditional methods. A user study
confirmed that RoVo preserves both naturalness and usability of protected
speech, highlighting its effectiveness in complex and evolving threat
scenarios.


http://arxiv.org/abs/2505.13862
PandaGuard: Systematic Evaluation of LLM Safety in the Era of Jailbreaking Attacks. (70%)
Guobin Shen; Dongcheng Zhao; Linghao Feng; Xiang He; Jihang Wang; Sicheng Shen; Haibo Tong; Yiting Dong; Jindong Li; Xiang Zheng; Yi Zeng
  Large language models (LLMs) have achieved remarkable capabilities but remain
vulnerable to adversarial prompts known as jailbreaks, which can bypass safety
alignment and elicit harmful outputs. Despite growing efforts in LLM safety
research, existing evaluations are often fragmented, focused on isolated attack
or defense techniques, and lack systematic, reproducible analysis. In this
work, we introduce PandaGuard, a unified and modular framework that models LLM
jailbreak safety as a multi-agent system comprising attackers, defenders, and
judges. Our framework implements 19 attack methods and 12 defense mechanisms,
along with multiple judgment strategies, all within a flexible plugin
architecture supporting diverse LLM interfaces, multiple interaction modes, and
configuration-driven experimentation that enhances reproducibility and
practical deployment. Built on this framework, we develop PandaBench, a
comprehensive benchmark that evaluates the interactions between these
attack/defense methods across 49 LLMs and various judgment approaches,
requiring over 3 billion tokens to execute. Our extensive evaluation reveals
key insights into model vulnerabilities, defense cost-performance trade-offs,
and judge consistency. We find that no single defense is optimal across all
dimensions and that judge disagreement introduces nontrivial variance in safety
assessments. We release the code, configurations, and evaluation results to
support transparent and reproducible research in LLM safety.


http://arxiv.org/abs/2505.12767
Language Models That Walk the Talk: A Framework for Formal Fairness Certificates. (69%)
Danqing Chen; Tobias Ladner; Ahmed Rayen Mhadhbi; Matthias Althoff
  As large language models become integral to high-stakes applications,
ensuring their robustness and fairness is critical. Despite their success,
large language models remain vulnerable to adversarial attacks, where small
perturbations, such as synonym substitutions, can alter model predictions,
posing risks in fairness-critical areas, such as gender bias mitigation, and
safety-critical areas, such as toxicity detection. While formal verification
has been explored for neural networks, its application to large language models
remains limited. This work presents a holistic verification framework to
certify the robustness of transformer-based language models, with a focus on
ensuring gender fairness and consistent outputs across different gender-related
terms. Furthermore, we extend this methodology to toxicity detection, offering
formal guarantees that adversarially manipulated toxic inputs are consistently
detected and appropriately censored, thereby ensuring the reliability of
moderation systems. By formalizing robustness within the embedding space, this
work strengthens the reliability of language models in ethical AI deployment
and content moderation.


http://arxiv.org/abs/2505.13362
DynaNoise: Dynamic Probabilistic Noise Injection for Defending Against Membership Inference Attacks. (38%)
Javad Forough; Hamed Haddadi
  Membership Inference Attacks (MIAs) pose a significant risk to the privacy of
training datasets by exploiting subtle differences in model outputs to
determine whether a particular data sample was used during training. These
attacks can compromise sensitive information, especially in domains such as
healthcare and finance, where data privacy is paramount. Traditional mitigation
techniques, such as static differential privacy, rely on injecting a fixed
amount of noise during training or inference. However, this approach often
leads to a detrimental trade-off: the noise may be insufficient to counter
sophisticated attacks or, when increased, may substantially degrade model
performance. In this paper, we present DynaNoise, an adaptive approach that
dynamically modulates noise injection based on query sensitivity. Our approach
performs sensitivity analysis using measures such as Shannon entropy to
evaluate the risk associated with each query and adjusts the noise variance
accordingly. A probabilistic smoothing step is then applied to renormalize the
perturbed outputs, ensuring that the model maintains high accuracy while
effectively obfuscating membership signals. We further propose an empirical
metric, the Membership Inference Defense Privacy-Utility Tradeoff (MIDPUT),
which quantifies the balance between reducing attack success rates and
preserving the target model's accuracy. Our extensive evaluation on several
benchmark datasets demonstrates that DynaNoise not only significantly reduces
MIA success rates but also achieves up to a fourfold improvement in the MIDPUT
metric compared to the state-of-the-art. Moreover, DynaNoise maintains
competitive model accuracy while imposing only marginal inference overhead,
highlighting its potential as an effective and efficient privacy defense
against MIAs.


http://arxiv.org/abs/2505.13819
Fragments to Facts: Partial-Information Fragment Inference from LLMs. (10%)
Lucas Rosenblatt; Bin Han; Robert Wolfe; Bill Howe
  Large language models (LLMs) can leak sensitive training data through
memorization and membership inference attacks. Prior work has primarily focused
on strong adversarial assumptions, including attacker access to entire samples
or long, ordered prefixes, leaving open the question of how vulnerable LLMs are
when adversaries have only partial, unordered sample information. For example,
if an attacker knows a patient has "hypertension," under what conditions can
they query a model fine-tuned on patient data to learn the patient also has
"osteoarthritis?" In this paper, we introduce a more general threat model under
this weaker assumption and show that fine-tuned LLMs are susceptible to these
fragment-specific extraction attacks. To systematically investigate these
attacks, we propose two data-blind methods: (1) a likelihood ratio attack
inspired by methods from membership inference, and (2) a novel approach, PRISM,
which regularizes the ratio by leveraging an external prior. Using examples
from both medical and legal settings, we show that both methods are competitive
with a data-aware baseline classifier that assumes access to labeled
in-distribution data, underscoring their robustness.


http://arxiv.org/abs/2505.12871
Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks? (8%)
Zi Liang; Haibo Hu; Qingqing Ye; Yaxin Xiao; Ronghua Li
  Low rank adaptation (LoRA) has emerged as a prominent technique for
fine-tuning large language models (LLMs) thanks to its superb efficiency gains
over previous methods. While extensive studies have examined the performance
and structural properties of LoRA, its behavior upon training-time attacks
remain underexplored, posing significant security risks. In this paper, we
theoretically investigate the security implications of LoRA's low-rank
structure during fine-tuning, in the context of its robustness against data
poisoning and backdoor attacks. We propose an analytical framework that models
LoRA's training dynamics, employs the neural tangent kernel to simplify the
analysis of the training process, and applies information theory to establish
connections between LoRA's low rank structure and its vulnerability against
training-time attacks. Our analysis indicates that LoRA exhibits better
robustness to backdoor attacks than full fine-tuning, while becomes more
vulnerable to untargeted data poisoning due to its over-simplified information
geometry. Extensive experimental evaluations have corroborated our theoretical
findings.


http://arxiv.org/abs/2505.17072
Safety Alignment Can Be Not Superficial With Explicit Safety Signals. (4%)
Jianwei Li; Jung-Eun Kim
  Recent studies on the safety alignment of large language models (LLMs) have
revealed that existing approaches often operate superficially, leaving models
vulnerable to various adversarial attacks. Despite their significance, these
studies generally fail to offer actionable solutions beyond data augmentation
for achieving more robust safety mechanisms. This paper identifies a
fundamental cause of this superficiality: existing alignment approaches often
presume that models can implicitly learn a safety-related reasoning task during
the alignment process, enabling them to refuse harmful requests. However, the
learned safety signals are often diluted by other competing objectives, leading
models to struggle with drawing a firm safety-conscious decision boundary when
confronted with adversarial attacks. Based on this observation, by explicitly
introducing a safety-related binary classification task and integrating its
signals with our attention and decoding strategies, we eliminate this ambiguity
and allow models to respond more responsibly to malicious queries. We emphasize
that, with less than 0.2x overhead cost, our approach enables LLMs to assess
the safety of both the query and the previously generated tokens at each
necessary generating step. Extensive experiments demonstrate that our method
significantly improves the resilience of LLMs against various adversarial
attacks, offering a promising pathway toward more robust generative AI systems.


http://arxiv.org/abs/2505.13708
Robust learning of halfspaces under log-concave marginals. (1%)
Jane Lange; Arsen Vasilyan
  We say that a classifier is \emph{adversarially robust} to perturbations of
norm $r$ if, with high probability over a point $x$ drawn from the input
distribution, there is no point within distance $\le r$ from $x$ that is
classified differently. The \emph{boundary volume} is the probability that a
point falls within distance $r$ of a point with a different label. This work
studies the task of computationally efficient learning of hypotheses with small
boundary volume, where the input is distributed as a subgaussian isotropic
log-concave distribution over $\mathbb{R}^d$.
  Linear threshold functions are adversarially robust; they have boundary
volume proportional to $r$. Such concept classes are efficiently learnable by
polynomial regression, which produces a polynomial threshold function (PTF),
but PTFs in general may have boundary volume $\Omega(1)$, even for $r \ll 1$.
  We give an algorithm that agnostically learns linear threshold functions and
returns a classifier with boundary volume $O(r+\varepsilon)$ at radius of
perturbation $r$. The time and sample complexity of
$d^{\tilde{O}(1/\varepsilon^2)}$ matches the complexity of polynomial
regression.
  Our algorithm augments the classic approach of polynomial regression with
three additional steps: a) performing the $\ell_1$-error regression under noise
sensitivity constraints, b) a structured partitioning and rounding step that
returns a Boolean classifier with error $\textsf{opt} + O(\varepsilon)$ and
noise sensitivity $O(r+\varepsilon)$ simultaneously, and c) a local corrector
that ``smooths'' a function with low noise sensitivity into a function that is
adversarially robust.


http://arxiv.org/abs/2505.12897
EPIC: Explanation of Pretrained Image Classification Networks via Prototype. (1%)
Piotr Borycki; Magdalena Trędowicz; Szymon Janusz; Jacek Tabor; Przemysław Spurek; Arkadiusz Lewicki; Łukasz Struski
  Explainable AI (XAI) methods generally fall into two categories. Post-hoc
approaches generate explanations for pre-trained models and are compatible with
various neural network architectures. These methods often use feature
importance visualizations, such as saliency maps, to indicate which input
regions influenced the model's prediction. Unfortunately, they typically offer
a coarse understanding of the model's decision-making process. In contrast,
ante-hoc (inherently explainable) methods rely on specially designed model
architectures trained from scratch. A notable subclass of these methods
provides explanations through prototypes, representative patches extracted from
the training data. However, prototype-based approaches have limitations: they
require dedicated architectures, involve specialized training procedures, and
perform well only on specific datasets. In this work, we propose EPIC
(Explanation of Pretrained Image Classification), a novel approach that bridges
the gap between these two paradigms. Like post-hoc methods, EPIC operates on
pre-trained models without architectural modifications. Simultaneously, it
delivers intuitive, prototype-based explanations inspired by ante-hoc
techniques. To the best of our knowledge, EPIC is the first post-hoc method
capable of fully replicating the core explanatory power of inherently
interpretable models. We evaluate EPIC on benchmark datasets commonly used in
prototype-based explanations, such as CUB-200-2011 and Stanford Cars, alongside
large-scale datasets like ImageNet, typically employed by post-hoc methods.
EPIC uses prototypes to explain model decisions, providing a flexible and
easy-to-understand tool for creating clear, high-quality explanations.


http://arxiv.org/abs/2505.12750
Malware families discovery via Open-Set Recognition on Android manifest permissions. (1%)
Filippo Leveni; Matteo Mistura; Francesco Iubatti; Carmine Giangregorio; Nicolò Pastore; Cesare Alippi; Giacomo Boracchi
  Malware are malicious programs that are grouped into families based on their
penetration technique, source code, and other characteristics. Classifying
malware programs into their respective families is essential for building
effective defenses against cyber threats. Machine learning models have a huge
potential in malware detection on mobile devices, as malware families can be
recognized by classifying permission data extracted from Android manifest
files. Still, the malware classification task is challenging due to the
high-dimensional nature of permission data and the limited availability of
training samples. In particular, the steady emergence of new malware families
makes it impossible to acquire a comprehensive training set covering all the
malware classes. In this work, we present a malware classification system that,
on top of classifying known malware, detects new ones. In particular, we
combine an open-set recognition technique developed within the computer vision
community, namely MaxLogit, with a tree-based Gradient Boosting classifier,
which is particularly effective in classifying high-dimensional data. Our
solution turns out to be very practical, as it can be seamlessly employed in a
standard classification workflow, and efficient, as it adds minimal
computational overhead. Experiments on public and proprietary datasets
demonstrate the potential of our solution, which has been deployed in a
business environment.


http://arxiv.org/abs/2505.12586
A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detection. (99%)
Sanggeon Yun; Ryozo Masukawa; Hyunwoo Oh; Nathaniel D. Bastian; Mohsen Imani
  Deep neural networks (DNNs) are highly susceptible to adversarial
examples--subtle, imperceptible perturbations that can lead to incorrect
predictions. While detection-based defenses offer a practical alternative to
adversarial training, many existing methods depend on external models, complex
architectures, heavy augmentations, or adversarial data, limiting their
efficiency and generalizability. We introduce a lightweight, plug-in detection
framework that leverages internal layer-wise inconsistencies within the target
model itself, requiring only benign data for calibration. Our approach is
grounded in the A Few Large Shifts Assumption, which posits that adversarial
perturbations typically induce large representation shifts in a small subset of
layers. Building on this, we propose two complementary strategies--Recovery
Testing (RT) and Logit-layer Testing (LT)--to expose internal disruptions
caused by adversaries. Evaluated on CIFAR-10, CIFAR-100, and ImageNet under
both standard and adaptive threat models, our method achieves state-of-the-art
detection performance with negligible computational overhead and no compromise
to clean accuracy. The code is available here:
https://github.com/c0510gy/AFLS-AED.


http://arxiv.org/abs/2505.12681
On the Mechanisms of Adversarial Data Augmentation for Robust and Adaptive Transfer Learning. (98%)
Hana Satou; Alan Mitkiy
  Transfer learning across domains with distribution shift remains a
fundamental challenge in building robust and adaptable machine learning
systems. While adversarial perturbations are traditionally viewed as threats
that expose model vulnerabilities, recent studies suggest that they can also
serve as constructive tools for data augmentation. In this work, we
systematically investigate the role of adversarial data augmentation (ADA) in
enhancing both robustness and adaptivity in transfer learning settings. We
analyze how adversarial examples, when used strategically during training,
improve domain generalization by enriching decision boundaries and reducing
overfitting to source-domain-specific features. We further propose a unified
framework that integrates ADA with consistency regularization and
domain-invariant representation learning. Extensive experiments across multiple
benchmark datasets -- including VisDA, DomainNet, and Office-Home --
demonstrate that our method consistently improves target-domain performance
under both unsupervised and few-shot domain adaptation settings. Our results
highlight a constructive perspective of adversarial learning, transforming
perturbation from a destructive attack into a regularizing force for
cross-domain transferability.


http://arxiv.org/abs/2505.12574
PoisonArena: Uncovering Competing Poisoning Attacks in Retrieval-Augmented Generation. (76%)
Liuji Chen; Xiaofang Yang; Yuanzhuo Lu; Jinghao Zhang; Xin Sun; Qiang Liu; Shu Wu; Jing Dong; Liang Wang
  Retrieval-Augmented Generation (RAG) systems, widely used to improve the
factual grounding of large language models (LLMs), are increasingly vulnerable
to poisoning attacks, where adversaries inject manipulated content into the
retriever's corpus. While prior research has predominantly focused on
single-attacker settings, real-world scenarios often involve multiple,
competing attackers with conflicting objectives. In this work, we introduce
PoisonArena, the first benchmark to systematically study and evaluate competing
poisoning attacks in RAG. We formalize the multi-attacker threat model, where
attackers vie to control the answer to the same query using mutually exclusive
misinformation. PoisonArena leverages the Bradley-Terry model to quantify each
method's competitive effectiveness in such adversarial environments. Through
extensive experiments on the Natural Questions and MS MARCO datasets, we
demonstrate that many attack strategies successful in isolation fail under
competitive pressure. Our findings highlight the limitations of conventional
evaluation metrics like Attack Success Rate (ASR) and F1 score and underscore
the need for competitive evaluation to assess real-world attack robustness.
PoisonArena provides a standardized framework to benchmark and develop future
attack and defense strategies under more realistic, multi-adversary conditions.
Project page: https://github.com/yxf203/PoisonArena.


http://arxiv.org/abs/2505.13541
SPIRIT: Patching Speech Language Models against Jailbreak Attacks. (61%)
Amirbek Djanibekov; Nurdaulet Mukhituly; Kentaro Inui; Hanan Aldarmaki; Nils Lukas
  Speech Language Models (SLMs) enable natural interactions via spoken
instructions, which more effectively capture user intent by detecting nuances
in speech. The richer speech signal introduces new security risks compared to
text-based models, as adversaries can better bypass safety mechanisms by
injecting imperceptible noise to speech. We analyze adversarial attacks and
find that SLMs are substantially more vulnerable to jailbreak attacks, which
can achieve a perfect 100% attack success rate in some instances. To improve
security, we propose post-hoc patching defenses used to intervene during
inference by modifying the SLM's activations that improve robustness up to 99%
with (i) negligible impact on utility and (ii) without any re-training. We
conduct ablation studies to maximize the efficacy of our defenses and improve
the utility/security trade-off, validated with large-scale benchmarks unique to
SLMs.


http://arxiv.org/abs/2505.12287
The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models. (47%)
Linghan Huang; Haolin Jin; Zhaoge Bi; Pengyue Yang; Peizhou Zhao; Taozhao Chen; Xiongfei Wu; Lei Ma; Huaming Chen
  Large language models (LLMs) have seen widespread applications across various
domains, yet remain vulnerable to adversarial prompt injections. While most
existing research on jailbreak attacks and hallucination phenomena has focused
primarily on open-source models, we investigate the frontier of closed-source
LLMs under multilingual attack scenarios. We present a first-of-its-kind
integrated adversarial framework that leverages diverse attack techniques to
systematically evaluate frontier proprietary solutions, including GPT-4o,
DeepSeek-R1, Gemini-1.5-Pro, and Qwen-Max. Our evaluation spans six categories
of security contents in both English and Chinese, generating 38,400 responses
across 32 types of jailbreak attacks. Attack success rate (ASR) is utilized as
the quantitative metric to assess performance from three dimensions: prompt
design, model architecture, and language environment. Our findings suggest that
Qwen-Max is the most vulnerable, while GPT-4o shows the strongest defense.
Notably, prompts in Chinese consistently yield higher ASRs than their English
counterparts, and our novel Two-Sides attack technique proves to be the most
effective across all models. This work highlights a dire need for
language-aware alignment and robust cross-lingual defenses in LLMs, and we hope
it will inspire researchers, developers, and policymakers toward more robust
and inclusive AI systems.


http://arxiv.org/abs/2505.12647
Spiking Neural Network: a low power solution for physical layer authentication. (31%)
Jung Hoon Lee; Sujith Vijayan
  Deep learning (DL) is a powerful tool that can solve complex problems, and
thus, it seems natural to assume that DL can be used to enhance the security of
wireless communication. However, deploying DL models to edge devices in
wireless networks is challenging, as they require significant amounts of
computing and power resources. Notably, Spiking Neural Networks (SNNs) are
known to be efficient in terms of power consumption, meaning they can be an
alternative platform for DL models for edge devices. In this study, we ask if
SNNs can be used in physical layer authentication. Our evaluation suggests that
SNNs can learn unique physical properties (i.e., `fingerprints') of RF
transmitters and use them to identify individual devices. Furthermore, we find
that SNNs are also vulnerable to adversarial attacks and that an autoencoder
can be used clean out adversarial perturbations to harden SNNs against them.


http://arxiv.org/abs/2505.12567
A Survey of Attacks on Large Language Models. (22%)
Wenrui Xu; Keshab K. Parhi
  Large language models (LLMs) and LLM-based agents have been widely deployed
in a wide range of applications in the real world, including healthcare
diagnostics, financial analysis, customer support, robotics, and autonomous
driving, expanding their powerful capability of understanding, reasoning, and
generating natural languages. However, the wide deployment of LLM-based
applications exposes critical security and reliability risks, such as the
potential for malicious misuse, privacy leakage, and service disruption that
weaken user trust and undermine societal safety. This paper provides a
systematic overview of the details of adversarial attacks targeting both LLMs
and LLM-based agents. These attacks are organized into three phases in LLMs:
Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity
Attacks. For each phase, we analyze the details of representative and recently
introduced attack methods along with their corresponding defenses. We hope our
survey will provide a good tutorial and a comprehensive understanding of LLM
security, especially for attacks on LLMs. We desire to raise attention to the
risks inherent in widely deployed LLM-based applications and highlight the
urgent need for robust mitigation strategies for evolving threats.


http://arxiv.org/abs/2505.12442
IP Leakage Attacks Targeting LLM-Based Multi-Agent Systems. (10%)
Liwen Wang; Wenxuan Wang; Shuai Wang; Zongjie Li; Zhenlan Ji; Zongyi Lyu; Daoyuan Wu; Shing-Chi Cheung
  The rapid advancement of Large Language Models (LLMs) has led to the
emergence of Multi-Agent Systems (MAS) to perform complex tasks through
collaboration. However, the intricate nature of MAS, including their
architecture and agent interactions, raises significant concerns regarding
intellectual property (IP) protection. In this paper, we introduce MASLEAK, a
novel attack framework designed to extract sensitive information from MAS
applications. MASLEAK targets a practical, black-box setting, where the
adversary has no prior knowledge of the MAS architecture or agent
configurations. The adversary can only interact with the MAS through its public
API, submitting attack query $q$ and observing outputs from the final agent.
Inspired by how computer worms propagate and infect vulnerable network hosts,
MASLEAK carefully crafts adversarial query $q$ to elicit, propagate, and retain
responses from each MAS agent that reveal a full set of proprietary components,
including the number of agents, system topology, system prompts, task
instructions, and tool usages. We construct the first synthetic dataset of MAS
applications with 810 applications and also evaluate MASLEAK against real-world
MAS applications, including Coze and CrewAI. MASLEAK achieves high accuracy in
extracting MAS IP, with an average attack success rate of 87% for system
prompts and task instructions, and 92% for system architecture in most cases.
We conclude by discussing the implications of our findings and the potential
defenses.


http://arxiv.org/abs/2505.12674
Few-Step Diffusion via Score identity Distillation. (1%)
Mingyuan Zhou; Yi Gu; Zhendong Wang
  Diffusion distillation has emerged as a promising strategy for accelerating
text-to-image (T2I) diffusion models by distilling a pretrained score network
into a one- or few-step generator. While existing methods have made notable
progress, they often rely on real or teacher-synthesized images to perform well
when distilling high-resolution T2I diffusion models such as Stable Diffusion
XL (SDXL), and their use of classifier-free guidance (CFG) introduces a
persistent trade-off between text-image alignment and generation diversity. We
address these challenges by optimizing Score identity Distillation (SiD) -- a
data-free, one-step distillation framework -- for few-step generation. Backed
by theoretical analysis that justifies matching a uniform mixture of outputs
from all generation steps to the data distribution, our few-step distillation
algorithm avoids step-specific networks and integrates seamlessly into existing
pipelines, achieving state-of-the-art performance on SDXL at 1024x1024
resolution. To mitigate the alignment-diversity trade-off when real text-image
pairs are available, we introduce a Diffusion GAN-based adversarial loss
applied to the uniform mixture and propose two new guidance strategies:
Zero-CFG, which disables CFG in the teacher and removes text conditioning in
the fake score network, and Anti-CFG, which applies negative CFG in the fake
score network. This flexible setup improves diversity without sacrificing
alignment. Comprehensive experiments on SD1.5 and SDXL demonstrate
state-of-the-art performance in both one-step and few-step generation settings,
along with robustness to the absence of real images. Our efficient PyTorch
implementation, along with the resulting one- and few-step distilled
generators, will be released publicly as a separate branch at
https://github.com/mingyuanzhou/SiD-LSG.


http://arxiv.org/abs/2505.12327
Robust Planning for Autonomous Driving via Mixed Adversarial Diffusion Predictions. (1%)
Albert Zhao; Stefano Soatto
  We describe a robust planning method for autonomous driving that mixes normal
and adversarial agent predictions output by a diffusion model trained for
motion prediction. We first train a diffusion model to learn an unbiased
distribution of normal agent behaviors. We then generate a distribution of
adversarial predictions by biasing the diffusion model at test time to generate
predictions that are likely to collide with a candidate plan. We score plans
using expected cost with respect to a mixture distribution of normal and
adversarial predictions, leading to a planner that is robust against
adversarial behaviors but not overly conservative when agents behave normally.
Unlike current approaches, we do not use risk measures that over-weight
adversarial behaviors while placing little to no weight on low-cost normal
behaviors or use hard safety constraints that may not be appropriate for all
driving scenarios. We show the effectiveness of our method on single-agent and
multi-agent jaywalking scenarios as well as a red light violation scenario.


http://arxiv.org/abs/2505.12317
Improving Out-of-Domain Robustness with Targeted Augmentation in Frequency and Pixel Spaces. (1%)
Ruoqi Wang; Haitao Wang; Shaojie Guo; Qiong Luo
  Out-of-domain (OOD) robustness under domain adaptation settings, where
labeled source data and unlabeled target data come from different
distributions, is a key challenge in real-world applications. A common approach
to improving OOD robustness is through data augmentations. However, in
real-world scenarios, models trained with generic augmentations can only
improve marginally when generalized under distribution shifts toward unlabeled
target domains. While dataset-specific targeted augmentations can address this
issue, they typically require expert knowledge and extensive prior data
analysis to identify the nature of the datasets and domain shift. To address
these challenges, we propose Frequency-Pixel Connect, a domain-adaptation
framework that enhances OOD robustness by introducing a targeted augmentation
in both the frequency space and pixel space. Specifically, we mix the amplitude
spectrum and pixel content of a source image and a target image to generate
augmented samples that introduce domain diversity while preserving the semantic
structure of the source image. Unlike previous targeted augmentation methods
that are both dataset-specific and limited to the pixel space, Frequency-Pixel
Connect is dataset-agnostic, enabling broader and more flexible applicability
beyond natural image datasets. We further analyze the effectiveness of
Frequency-Pixel Connect by evaluating the performance of our method connecting
same-class cross-domain samples while separating different-class examples. We
demonstrate that Frequency-Pixel Connect significantly improves cross-domain
connectivity and outperforms previous generic methods on four diverse
real-world benchmarks across vision, medical, audio, and astronomical domains,
and it also outperforms other dataset-specific targeted augmentation methods.


http://arxiv.org/abs/2505.12368
CAPTURE: Context-Aware Prompt Injection Testing and Robustness Enhancement. (1%)
Gauri Kholkar; Ratinder Ahuja
  Prompt injection remains a major security risk for large language models.
However, the efficacy of existing guardrail models in context-aware settings
remains underexplored, as they often rely on static attack benchmarks.
Additionally, they have over-defense tendencies. We introduce CAPTURE, a novel
context-aware benchmark assessing both attack detection and over-defense
tendencies with minimal in-domain examples. Our experiments reveal that current
prompt injection guardrail models suffer from high false negatives in
adversarial cases and excessive false positives in benign scenarios,
highlighting critical limitations. To demonstrate our framework's utility, we
train CaptureGuard on our generated data. This new model drastically reduces
both false negative and false positive rates on our context-aware datasets
while also generalizing effectively to external benchmarks, establishing a path
toward more robust and practical prompt injection defenses.


http://arxiv.org/abs/2505.12167
FABLE: A Localized, Targeted Adversarial Attack on Weather Forecasting Models. (99%)
Yue Deng; Asadullah Hill Galib; Xin Lan; Pang-Ning Tan; Lifeng Luo
  Deep learning-based weather forecasting models have recently demonstrated
significant performance improvements over gold-standard physics-based
simulation tools. However, these models are vulnerable to adversarial attacks,
which raises concerns about their trustworthiness. In this paper, we first
investigate the feasibility of applying existing adversarial attack methods to
weather forecasting models. We argue that a successful attack should (1) not
modify significantly its original inputs, (2) be faithful, i.e., achieve the
desired forecast at targeted locations with minimal changes to non-targeted
locations, and (3) be geospatio-temporally realistic. However, balancing these
criteria is a challenge as existing methods are not designed to preserve the
geospatio-temporal dependencies of the original samples. To address this
challenge, we propose a novel framework called FABLE (Forecast Alteration By
Localized targeted advErsarial attack), which employs a 3D discrete wavelet
decomposition to extract the varying components of the geospatio-temporal data.
By regulating the magnitude of adversarial perturbations across different
components, FABLE can generate adversarial inputs that maintain
geospatio-temporal coherence while remaining faithful and closely aligned with
the original inputs. Experimental results on multiple real-world datasets
demonstrate the effectiveness of our framework over baseline methods across
various metrics.


http://arxiv.org/abs/2505.11895
Adversarial Robustness for Unified Multi-Modal Encoders via Efficient Calibration. (98%)
Chih-Ting Liao; Bin Ren; Guofeng Mei; Xu Zheng
  Recent unified multi-modal encoders align a wide range of modalities into a
shared representation space, enabling diverse cross-modal tasks. Despite their
impressive capabilities, the robustness of these models under adversarial
perturbations remains underexplored, which is a critical concern for
safety-sensitive applications. In this work, we present the first comprehensive
study of adversarial vulnerability in unified multi-modal encoders. We find
that even mild adversarial perturbations lead to substantial performance drops
across all modalities. Non-visual inputs, such as audio and point clouds, are
especially fragile, while visual inputs like images and videos also degrade
significantly. To address this, we propose an efficient adversarial calibration
framework that improves robustness across modalities without modifying
pretrained encoders or semantic centers, ensuring compatibility with existing
foundation models. Our method introduces modality-specific projection heads
trained solely on adversarial examples, while keeping the backbone and
embeddings frozen. We explore three training objectives: fixed-center
cross-entropy, clean-to-adversarial L2 alignment, and clean-adversarial
InfoNCE, and we introduce a regularization strategy to ensure
modality-consistent alignment under attack. Experiments on six modalities and
three Bind-style models show that our method improves adversarial robustness by
up to 47.3 percent at epsilon = 4/255, while preserving or even improving clean
zero-shot and retrieval performance with less than 1 percent trainable
parameters.


http://arxiv.org/abs/2505.12009
Black-box Adversaries from Latent Space: Unnoticeable Attacks on Human Pose and Shape Estimation. (83%)
Zhiying Li; Guanggang Geng; Yeying Jin; Zhizhi Guo; Bruce Gu; Jidong Huo; Zhaoxin Fan; Wenjun Wu
  Expressive human pose and shape (EHPS) estimation is vital for digital human
generation, particularly in live-streaming applications. However, most existing
EHPS models focus primarily on minimizing estimation errors, with limited
attention on potential security vulnerabilities. Current adversarial attacks on
EHPS models often require white-box access (e.g., model details or gradients)
or generate visually conspicuous perturbations, limiting their practicality and
ability to expose real-world security threats. To address these limitations, we
propose a novel Unnoticeable Black-Box Attack (UBA) against EHPS models. UBA
leverages the latent-space representations of natural images to generate an
optimal adversarial noise pattern and iteratively refine its attack potency
along an optimized direction in digital space. Crucially, this process relies
solely on querying the model's output, requiring no internal knowledge of the
EHPS architecture, while guiding the noise optimization toward greater stealth
and effectiveness. Extensive experiments and visual analyses demonstrate the
superiority of UBA. Notably, UBA increases the pose estimation errors of EHPS
models by 17.27%-58.21% on average, revealing critical vulnerabilities. These
findings underscore the urgent need to address and mitigate security risks
associated with digital human generation systems.


http://arxiv.org/abs/2505.12186
Self-Destructive Language Model. (75%)
Yuhui Wang; Rongyi Zhu; Ting Wang
  Harmful fine-tuning attacks pose a major threat to the security of large
language models (LLMs), allowing adversaries to compromise safety guardrails
with minimal harmful data. While existing defenses attempt to reinforce LLM
alignment, they fail to address models' inherent "trainability" on harmful
data, leaving them vulnerable to stronger attacks with increased learning rates
or larger harmful datasets. To overcome this critical limitation, we introduce
SEAM, a novel alignment-enhancing defense that transforms LLMs into
self-destructive models with intrinsic resilience to misalignment attempts.
Specifically, these models retain their capabilities for legitimate tasks while
exhibiting substantial performance degradation when fine-tuned on harmful data.
The protection is achieved through a novel loss function that couples the
optimization trajectories of benign and harmful data, enhanced with adversarial
gradient ascent to amplify the self-destructive effect. To enable practical
training, we develop an efficient Hessian-free gradient estimate with
theoretical error bounds. Extensive evaluation across LLMs and datasets
demonstrates that SEAM creates a no-win situation for adversaries: the
self-destructive models achieve state-of-the-art robustness against
low-intensity attacks and undergo catastrophic performance collapse under
high-intensity attacks, rendering them effectively unusable. (warning: this
paper contains potentially harmful content generated by LLMs.)


http://arxiv.org/abs/2505.12019
FL-PLAS: Federated Learning with Partial Layer Aggregation for Backdoor Defense Against High-Ratio Malicious Clients. (10%)
Jianyi Zhang; Ziyin Zhou; Yilong Li; Qichao Jin
  Federated learning (FL) is gaining increasing attention as an emerging
collaborative machine learning approach, particularly in the context of
large-scale computing and data systems. However, the fundamental algorithm of
FL, Federated Averaging (FedAvg), is susceptible to backdoor attacks. Although
researchers have proposed numerous defense algorithms, two significant
challenges remain. The attack is becoming more stealthy and harder to detect,
and current defense methods are unable to handle 50\% or more malicious users
or assume an auxiliary server dataset.
  To address these challenges, we propose a novel defense algorithm, FL-PLAS,
\textbf{F}ederated \textbf{L}earning based on \textbf{P}artial\textbf{ L}ayer
\textbf{A}ggregation \textbf{S}trategy. In particular, we divide the local
model into a feature extractor and a classifier. In each iteration, the clients
only upload the parameters of a feature extractor after local training. The
server then aggregates these local parameters and returns the results to the
clients.
  Each client retains its own classifier layer, ensuring that the backdoor
labels do not impact other clients. We assess the effectiveness of FL-PLAS
against state-of-the-art (SOTA) backdoor attacks on three image datasets and
compare our approach to six defense strategies. The results of the experiment
demonstrate that our methods can effectively protect local models from backdoor
attacks. Without requiring any auxiliary dataset for the server, our method
achieves a high main-task accuracy with a lower backdoor accuracy even under
the condition of 90\% malicious users with the attacks of trigger, semantic and
edge-case.


http://arxiv.org/abs/2505.11842
Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs. (10%)
Xuannan Liu; Zekun Li; Zheqi He; Peipei Li; Shuhan Xia; Xing Cui; Huaibo Huang; Xi Yang; Ran He
  The increasing deployment of Large Vision-Language Models (LVLMs) raises
safety concerns under potential malicious inputs. However, existing multimodal
safety evaluations primarily focus on model vulnerabilities exposed by static
image inputs, ignoring the temporal dynamics of video that may induce distinct
safety risks. To bridge this gap, we introduce Video-SafetyBench, the first
comprehensive benchmark designed to evaluate the safety of LVLMs under
video-text attacks. It comprises 2,264 video-text pairs spanning 48
fine-grained unsafe categories, each pairing a synthesized video with either a
harmful query, which contains explicit malice, or a benign query, which appears
harmless but triggers harmful behavior when interpreted alongside the video. To
generate semantically accurate videos for safety evaluation, we design a
controllable pipeline that decomposes video semantics into subject images (what
is shown) and motion text (how it moves), which jointly guide the synthesis of
query-relevant videos. To effectively evaluate uncertain or borderline harmful
outputs, we propose RJScore, a novel LLM-based metric that incorporates the
confidence of judge models and human-aligned decision threshold calibration.
Extensive experiments show that benign-query video composition achieves average
attack success rates of 67.2%, revealing consistent vulnerabilities to
video-induced attacks. We believe Video-SafetyBench will catalyze future
research into video-based safety evaluation and defense strategies.


http://arxiv.org/abs/2505.12045
FIGhost: Fluorescent Ink-based Stealthy and Flexible Backdoor Attacks on Physical Traffic Sign Recognition. (10%)
Shuai Yuan; Guowen Xu; Hongwei Li; Rui Zhang; Xinyuan Qian; Wenbo Jiang; Hangcheng Cao; Qingchuan Zhao
  Traffic sign recognition (TSR) systems are crucial for autonomous driving but
are vulnerable to backdoor attacks. Existing physical backdoor attacks either
lack stealth, provide inflexible attack control, or ignore emerging
Vision-Large-Language-Models (VLMs). In this paper, we introduce FIGhost, the
first physical-world backdoor attack leveraging fluorescent ink as triggers.
Fluorescent triggers are invisible under normal conditions and activated
stealthily by ultraviolet light, providing superior stealthiness, flexibility,
and untraceability. Inspired by real-world graffiti, we derive realistic
trigger shapes and enhance their robustness via an interpolation-based
fluorescence simulation algorithm. Furthermore, we develop an automated
backdoor sample generation method to support three attack objectives. Extensive
evaluations in the physical world demonstrate FIGhost's effectiveness against
state-of-the-art detectors and VLMs, maintaining robustness under environmental
variations and effectively evading existing defenses.


http://arxiv.org/abs/2505.10983
GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models. (99%)
Haozheng Luo; Chenghao Qiu; Yimin Wang; Shang Wu; Jiahao Yu; Han Liu; Binghui Wang; Yan Chen
  We propose the first unified adversarial attack benchmark for Genomic
Foundation Models (GFMs), named GenoArmory. Unlike existing GFM benchmarks,
GenoArmory offers the first comprehensive evaluation framework to
systematically assess the vulnerability of GFMs to adversarial attacks.
Methodologically, we evaluate the adversarial robustness of five
state-of-the-art GFMs using four widely adopted attack algorithms and three
defense strategies. Importantly, our benchmark provides an accessible and
comprehensive framework to analyze GFM vulnerabilities with respect to model
architecture, quantization schemes, and training datasets. Additionally, we
introduce GenoAdv, a new adversarial sample dataset designed to improve GFM
safety. Empirically, classification models exhibit greater robustness to
adversarial perturbations compared to generative models, highlighting the
impact of task type on model vulnerability. Moreover, adversarial attacks
frequently target biologically significant genomic regions, suggesting that
these models effectively capture meaningful sequence features.


http://arxiv.org/abs/2505.11449
LLMs unlock new paths to monetizing exploits. (78%)
Nicholas Carlini; Milad Nasr; Edoardo Debenedetti; Barry Wang; Christopher A. Choquette-Choo; Daphne Ippolito; Florian Tramèr; Matthew Jagielski
  We argue that Large language models (LLMs) will soon alter the economics of
cyberattacks. Instead of attacking the most commonly used software and
monetizing exploits by targeting the lowest common denominator among victims,
LLMs enable adversaries to launch tailored attacks on a user-by-user basis. On
the exploitation front, instead of human attackers manually searching for one
difficult-to-identify bug in a product with millions of users, LLMs can find
thousands of easy-to-identify bugs in products with thousands of users. And on
the monetization front, instead of generic ransomware that always performs the
same attack (encrypt all your data and request payment to decrypt), an
LLM-driven ransomware attack could tailor the ransom demand based on the
particular content of each exploited device.
  We show that these two attacks (and several others) are imminently practical
using state-of-the-art LLMs. For example, we show that without any human
intervention, an LLM finds highly sensitive personal information in the Enron
email dataset (e.g., an executive having an affair with another employee) that
could be used for blackmail. While some of our attacks are still too expensive
to scale widely today, the incentives to implement these attacks will only
increase as LLMs get cheaper. Thus, we argue that LLMs create a need for new
defense-in-depth approaches.


http://arxiv.org/abs/2505.10942
Nosy Layers, Noisy Fixes: Tackling DRAs in Federated Learning Systems using Explainable AI. (75%)
Meghali Nandi; Arash Shaghaghi; Nazatul Haque Sultan; Gustavo Batista; Raymond K. Zhao; Sanjay Jha
  Federated Learning (FL) has emerged as a powerful paradigm for collaborative
model training while keeping client data decentralized and private. However, it
is vulnerable to Data Reconstruction Attacks (DRA) such as "LoKI" and "Robbing
the Fed", where malicious models sent from the server to the client can
reconstruct sensitive user data. To counter this, we introduce DRArmor, a novel
defense mechanism that integrates Explainable AI with targeted detection and
mitigation strategies for DRA. Unlike existing defenses that focus on the
entire model, DRArmor identifies and addresses the root cause (i.e., malicious
layers within the model that send gradients with malicious intent) by analyzing
their contribution to the output and detecting inconsistencies in gradient
values. Once these malicious layers are identified, DRArmor applies defense
techniques such as noise injection, pixelation, and pruning to these layers
rather than the whole model, minimizing the attack surface and preserving
client data privacy. We evaluate DRArmor's performance against the advanced
LoKI attack across diverse datasets, including MNIST, CIFAR-10, CIFAR-100, and
ImageNet, in a 200-client FL setup. Our results demonstrate DRArmor's
effectiveness in mitigating data leakage, achieving high True Positive and True
Negative Rates of 0.910 and 0.890, respectively. Additionally, DRArmor
maintains an average accuracy of 87%, effectively protecting client privacy
without compromising model performance. Compared to existing defense
mechanisms, DRArmor reduces the data leakage rate by 62.5% with datasets
containing 500 samples per client.


http://arxiv.org/abs/2505.10903
On the Security Risks of ML-based Malware Detection Systems: A Survey. (64%)
Ping He; Yuhao Mao; Changjiang Li; Lorenzo Cavallaro; Ting Wang; Shouling Ji
  Malware presents a persistent threat to user privacy and data integrity. To
combat this, machine learning-based (ML-based) malware detection (MD) systems
have been developed. However, these systems have increasingly been attacked in
recent years, undermining their effectiveness in practice. While the security
risks associated with ML-based MD systems have garnered considerable attention,
the majority of prior works is limited to adversarial malware examples, lacking
a comprehensive analysis of practical security risks. This paper addresses this
gap by utilizing the CIA principles to define the scope of security risks. We
then deconstruct ML-based MD systems into distinct operational stages, thus
developing a stage-based taxonomy. Utilizing this taxonomy, we summarize the
technical progress and discuss the gaps in the attack and defense proposals
related to the ML-based MD systems within each stage. Subsequently, we conduct
two case studies, using both inter-stage and intra-stage analyses according to
the stage-based taxonomy to provide new empirical insights. Based on these
analyses and insights, we suggest potential future directions from both
inter-stage and intra-stage perspectives.


http://arxiv.org/abs/2505.11459
ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks. (56%)
Zhixiong Zhuang; Maria-Irina Nicolae; Hui-Po Wang; Mario Fritz
  The integration of large language models (LLMs) into a wide range of
applications has highlighted the critical role of well-crafted system prompts,
which require extensive testing and domain expertise. These prompts enhance
task performance but may also encode sensitive information and filtering
criteria, posing security risks if exposed. Recent research shows that system
prompts are vulnerable to extraction attacks, while existing defenses are
either easily bypassed or require constant updates to address new threats. In
this work, we introduce ProxyPrompt, a novel defense mechanism that prevents
prompt leakage by replacing the original prompt with a proxy. This proxy
maintains the original task's utility while obfuscating the extracted prompt,
ensuring attackers cannot reproduce the task or access sensitive information.
Comprehensive evaluations on 264 LLM and system prompt pairs show that
ProxyPrompt protects 94.70% of prompts from extraction attacks, outperforming
the next-best defense, which only achieves 42.80%.


http://arxiv.org/abs/2505.11413
CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs. (41%)
Sijia Chen; Xiaomin Li; Mengxue Zhang; Eric Hanchen Jiang; Qingcheng Zeng; Chen-Hsiang Yu
  Large language models (LLMs) are increasingly deployed in medical contexts,
raising critical concerns about safety, alignment, and susceptibility to
adversarial manipulation. While prior benchmarks assess model refusal
capabilities for harmful prompts, they often lack clinical specificity, graded
harmfulness levels, and coverage of jailbreak-style attacks. We introduce CARES
(Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for
evaluating LLM safety in healthcare. CARES includes over 18,000 prompts
spanning eight medical safety principles, four harm levels, and four prompting
styles: direct, indirect, obfuscated, and role-play, to simulate both malicious
and benign use cases. We propose a three-way response evaluation protocol
(Accept, Caution, Refuse) and a fine-grained Safety Score metric to assess
model behavior. Our analysis reveals that many state-of-the-art LLMs remain
vulnerable to jailbreaks that subtly rephrase harmful prompts, while also
over-refusing safe but atypically phrased queries. Finally, we propose a
mitigation strategy using a lightweight classifier to detect jailbreak attempts
and steer models toward safer behavior via reminder-based conditioning. CARES
provides a rigorous framework for testing and improving medical LLM safety
under adversarial and ambiguous conditions.


http://arxiv.org/abs/2505.15833
Adversarially Robust Spiking Neural Networks with Sparse Connectivity. (11%)
Mathias Schmolli; Maximilian Baronig; Robert Legenstein; Ozan Özdenizci
  Deployment of deep neural networks in resource-constrained embedded systems
requires innovative algorithmic solutions to facilitate their energy and memory
efficiency. To further ensure the reliability of these systems against
malicious actors, recent works have extensively studied adversarial robustness
of existing architectures. Our work focuses on the intersection of adversarial
robustness, memory- and energy-efficiency in neural networks. We introduce a
neural network conversion algorithm designed to produce sparse and
adversarially robust spiking neural networks (SNNs) by leveraging the sparse
connectivity and weights from a robustly pretrained artificial neural network
(ANN). Our approach combines the energy-efficient architecture of SNNs with a
novel conversion algorithm, leading to state-of-the-art performance with
enhanced energy and memory efficiency through sparse connectivity and
activations. Our models are shown to achieve up to 100x reduction in the number
of weights to be stored in memory, with an estimated 8.6x increase in energy
efficiency compared to dense SNNs, while maintaining high performance and
robustness against adversarial threats.


http://arxiv.org/abs/2505.11717
EnvInjection: Environmental Prompt Injection Attack to Multi-modal Web Agents. (5%)
Xilong Wang; John Bloch; Zedian Shao; Yuepeng Hu; Shuyan Zhou; Neil Zhenqiang Gong
  Multi-modal large language model (MLLM)-based web agents interact with
webpage environments by generating actions based on screenshots of the
webpages. Environmental prompt injection attacks manipulate the environment to
induce the web agent to perform a specific, attacker-chosen action--referred to
as the target action. However, existing attacks suffer from limited
effectiveness or stealthiness, or are impractical in real-world settings. In
this work, we propose EnvInjection, a new attack that addresses these
limitations. Our attack adds a perturbation to the raw pixel values of the
rendered webpage, which can be implemented by modifying the webpage's source
code. After these perturbed pixels are mapped into a screenshot, the
perturbation induces the web agent to perform the target action. We formulate
the task of finding the perturbation as an optimization problem. A key
challenge in solving this problem is that the mapping between raw pixel values
and screenshot is non-differentiable, making it difficult to backpropagate
gradients to the perturbation. To overcome this, we train a neural network to
approximate the mapping and apply projected gradient descent to solve the
reformulated optimization problem. Extensive evaluation on multiple webpage
datasets shows that EnvInjection is highly effective and significantly
outperforms existing baselines.


http://arxiv.org/abs/2505.10864
Anti-Sensing: Defense against Unauthorized Radar-based Human Vital Sign Sensing with Physically Realizable Wearable Oscillators. (2%)
Md Farhan Tasnim Oshim; Nigel Doering; Bashima Islam; Tsui-Wei Weng; Tauhidur Rahman
  Recent advancements in Ultra-Wideband (UWB) radar technology have enabled
contactless, non-line-of-sight vital sign monitoring, making it a valuable tool
for healthcare. However, UWB radar's ability to capture sensitive physiological
data, even through walls, raises significant privacy concerns, particularly in
human-robot interactions and autonomous systems that rely on radar for sensing
human presence and physiological functions. In this paper, we present
Anti-Sensing, a novel defense mechanism designed to prevent unauthorized
radar-based sensing. Our approach introduces physically realizable
perturbations, such as oscillatory motion from wearable devices, to disrupt
radar sensing by mimicking natural cardiac motion, thereby misleading heart
rate (HR) estimations. We develop a gradient-based algorithm to optimize the
frequency and spatial amplitude of these oscillations for maximal disruption
while ensuring physiological plausibility. Through both simulations and
real-world experiments with radar data and neural network-based HR sensing
models, we demonstrate the effectiveness of Anti-Sensing in significantly
degrading model accuracy, offering a practical solution for privacy
preservation.


http://arxiv.org/abs/2505.11710
Co-Evolutionary Defence of Active Directory Attack Graphs via GNN-Approximated Dynamic Programming. (1%)
Diksha Goel; Hussain Ahmad; Kristen Moore; Mingyu Guo
  Modern enterprise networks increasingly rely on Active Directory (AD) for
identity and access management. However, this centralization exposes a single
point of failure, allowing adversaries to compromise high-value assets.
Existing AD defense approaches often assume static attacker behavior, but
real-world adversaries adapt dynamically, rendering such methods brittle. To
address this, we model attacker-defender interactions in AD as a Stackelberg
game between an adaptive attacker and a proactive defender. We propose a
co-evolutionary defense framework that combines Graph Neural Network
Approximated Dynamic Programming (GNNDP) to model attacker strategies, with
Evolutionary Diversity Optimization (EDO) to generate resilient blocking
strategies. To ensure scalability, we introduce a Fixed-Parameter Tractable
(FPT) graph reduction method that reduces complexity while preserving strategic
structure. Our framework jointly refines attacker and defender policies to
improve generalization and prevent premature convergence. Experiments on
synthetic AD graphs show near-optimal results (within 0.1 percent of optimality
on r500) and improved performance on larger graphs (r1000 and r2000),
demonstrating the framework's scalability and effectiveness.


http://arxiv.org/abs/2505.11719
Zero-Shot Visual Generalization in Robot Manipulation. (1%)
Sumeet Batra; Gaurav Sukhatme
  Training vision-based manipulation policies that are robust across diverse
visual environments remains an important and unresolved challenge in robot
learning. Current approaches often sidestep the problem by relying on invariant
representations such as point clouds and depth, or by brute-forcing
generalization through visual domain randomization and/or large, visually
diverse datasets. Disentangled representation learning - especially when
combined with principles of associative memory - has recently shown promise in
enabling vision-based reinforcement learning policies to be robust to visual
distribution shifts. However, these techniques have largely been constrained to
simpler benchmarks and toy environments. In this work, we scale disentangled
representation learning and associative memory to more visually and dynamically
complex manipulation tasks and demonstrate zero-shot adaptability to visual
perturbations in both simulation and on real hardware. We further extend this
approach to imitation learning, specifically Diffusion Policy, and empirically
show significant gains in visual generalization compared to state-of-the-art
imitation learning methods. Finally, we introduce a novel technique adapted
from the model equivariance literature that transforms any trained neural
network policy into one invariant to 2D planar rotations, making our policy not
only visually robust but also resilient to certain camera perturbations. We
believe that this work marks a significant step towards manipulation policies
that are not only adaptable out of the box, but also robust to the complexities
and dynamical nature of real-world deployment. Supplementary videos are
available at https://sites.google.com/view/vis-gen-robotics/home.


http://arxiv.org/abs/2506.06290
CellCLIP -- Learning Perturbation Effects in Cell Painting via Text-Guided Contrastive Learning. (1%)
Mingyu Lu; Ethan Weinberger; Chanwoo Kim; Su-In Lee
  High-content screening (HCS) assays based on high-throughput microscopy
techniques such as Cell Painting have enabled the interrogation of cells'
morphological responses to perturbations at an unprecedented scale. The
collection of such data promises to facilitate a better understanding of the
relationships between different perturbations and their effects on cellular
state. Towards achieving this goal, recent advances in cross-modal contrastive
learning could, in theory, be leveraged to learn a unified latent space that
aligns perturbations with their corresponding morphological effects. However,
the application of such methods to HCS data is not straightforward due to
substantial differences in the semantics of Cell Painting images compared to
natural images, and the difficulty of representing different classes of
perturbations (e.g., small molecule vs CRISPR gene knockout) in a single latent
space. In response to these challenges, here we introduce CellCLIP, a
cross-modal contrastive learning framework for HCS data. CellCLIP leverages
pre-trained image encoders coupled with a novel channel encoding scheme to
better capture relationships between different microscopy channels in image
embeddings, along with natural language encoders for representing
perturbations. Our framework outperforms current open-source models,
demonstrating the best performance in both cross-modal retrieval and
biologically meaningful downstream tasks while also achieving significant
reductions in computation time.


http://arxiv.org/abs/2505.10430
The Ephemeral Threat: Assessing the Security of Algorithmic Trading Systems powered by Deep Learning. (50%)
Advije Rizvani; Giovanni Apruzzese; Pavel Laskov
  We study the security of stock price forecasting using Deep Learning (DL) in
computational finance. Despite abundant prior research on the vulnerability of
DL to adversarial perturbations, such work has hitherto hardly addressed
practical adversarial threat models in the context of DL-powered algorithmic
trading systems (ATS). Specifically, we investigate the vulnerability of ATS to
adversarial perturbations launched by a realistically constrained attacker. We
first show that existing literature has paid limited attention to DL security
in the financial domain, which is naturally attractive for adversaries. Then,
we formalize the concept of ephemeral perturbations (EP), which can be used to
stage a novel type of attack tailored for DL-based ATS. Finally, we carry out
an end-to-end evaluation of our EP against a profitable ATS. Our results reveal
that the introduction of small changes to the input stock prices not only (i)
induces the DL model to behave incorrectly but also (ii) leads the whole ATS to
make suboptimal buy/sell decisions, resulting in a worse financial performance
of the targeted ATS.


http://arxiv.org/abs/2505.11548
One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems. (31%)
Zhiyuan Chang; Xiaojun Jia; Mingyang Li; Junjie Wang; Yuekai Huang; Qing Wang; Ziyou Jiang; Yang Liu
  Large Language Models (LLMs) enhanced with Retrieval-Augmented Generation
(RAG) have shown improved performance in generating accurate responses.
However, the dependence on external knowledge bases introduces potential
security vulnerabilities, particularly when these knowledge bases are publicly
accessible and modifiable. Poisoning attacks on knowledge bases for RAG systems
face two fundamental challenges: the injected malicious content must compete
with multiple authentic documents retrieved by the retriever, and LLMs tend to
trust retrieved information that aligns with their internal memorized
knowledge. Previous works attempt to address these challenges by injecting
multiple malicious documents, but such saturation attacks are easily detectable
and impractical in real-world scenarios. To enable the effective single
document poisoning attack, we propose AuthChain, a novel knowledge poisoning
attack method that leverages Chain-of-Evidence theory and authority effect to
craft more convincing poisoned documents. AuthChain generates poisoned content
that establishes strong evidence chains and incorporates authoritative
statements, effectively overcoming the interference from both authentic
documents and LLMs' internal knowledge. Extensive experiments across six
popular LLMs demonstrate that AuthChain achieves significantly higher attack
success rates while maintaining superior stealthiness against RAG defense
mechanisms compared to state-of-the-art baselines.


http://arxiv.org/abs/2505.10297
Defending the Edge: Representative-Attention for Mitigating Backdoor Attacks in Federated Learning. (15%)
Chibueze Peace Obioma; Youcheng Sun; Mustafa A. Mustafa
  Federated learning (FL) enhances privacy and reduces communication cost for
resource-constrained edge clients by supporting distributed model training at
the edge. However, the heterogeneous nature of such devices produces diverse,
non-independent, and identically distributed (non-IID) data, making the
detection of backdoor attacks more challenging. In this paper, we propose a
novel federated representative-attention-based defense mechanism, named FeRA,
that leverages cross-client attention over internal feature representations to
distinguish benign from malicious clients. FeRA computes an anomaly score based
on representation reconstruction errors, effectively identifying clients whose
internal activations significantly deviate from the group consensus. Our
evaluation demonstrates FeRA's robustness across various FL scenarios,
including challenging non-IID data distributions typical of edge devices.
Experimental results show that it effectively reduces backdoor attack success
rates while maintaining high accuracy on the main task. The method is
model-agnostic, attack-agnostic, and does not require labeled reference data,
making it well suited to heterogeneous and resource-limited edge deployments.


http://arxiv.org/abs/2505.10351
A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability. (11%)
Jie Zhu; Jirong Zha; Ding Li; Leye Wang
  Self-supervised learning shows promise in harnessing extensive unlabeled
data, but it also confronts significant privacy concerns, especially in vision.
In this paper, we perform membership inference on visual self-supervised models
in a more realistic setting: self-supervised training method and details are
unknown for an adversary when attacking as he usually faces a black-box system
in practice. In this setting, considering that self-supervised model could be
trained by completely different self-supervised paradigms, e.g., masked image
modeling and contrastive learning, with complex training details, we propose a
unified membership inference method called PartCrop. It is motivated by the
shared part-aware capability among models and stronger part response on the
training data. Specifically, PartCrop crops parts of objects in an image to
query responses within the image in representation space. We conduct extensive
attacks on self-supervised models with different training protocols and
structures using three widely used image datasets. The results verify the
effectiveness and generalization of PartCrop. Moreover, to defend against
PartCrop, we evaluate two common approaches, i.e., early stop and differential
privacy, and propose a tailored method called shrinking crop scale range. The
defense experiments indicate that all of them are effective. Finally, besides
prototype testing on toy visual encoders and small-scale image datasets, we
quantitatively study the impacts of scaling from both data and model aspects in
a realistic scenario and propose a scalable PartCrop-v2 by introducing two
structural improvements to PartCrop. Our code is at
https://github.com/JiePKU/PartCrop.


http://arxiv.org/abs/2505.10497
MorphGuard: Morph Specific Margin Loss for Enhancing Robustness to Face Morphing Attacks. (2%)
Iurii Medvedev; Nuno Goncalves
  Face recognition has evolved significantly with the advancement of deep
learning techniques, enabling its widespread adoption in various applications
requiring secure authentication. However, this progress has also increased its
exposure to presentation attacks, including face morphing, which poses a
serious security threat by allowing one identity to impersonate another.
Therefore, modern face recognition systems must be robust against such attacks.
  In this work, we propose a novel approach for training deep networks for face
recognition with enhanced robustness to face morphing attacks. Our method
modifies the classification task by introducing a dual-branch classification
strategy that effectively handles the ambiguity in the labeling of face morphs.
This adaptation allows the model to incorporate morph images into the training
process, improving its ability to distinguish them from bona fide samples.
  Our strategy has been validated on public benchmarks, demonstrating its
effectiveness in enhancing robustness against face morphing attacks.
Furthermore, our approach is universally applicable and can be integrated into
existing face recognition training pipelines to improve classification-based
recognition methods.


http://arxiv.org/abs/2505.13500
Noise Injection Systemically Degrades Large Language Model Safety Guardrails. (1%)
Prithviraj Singh Shahani; Matthias Scheutz
  Safety guardrails in large language models (LLMs) are a critical component in
preventing harmful outputs. Yet, their resilience under perturbation remains
poorly understood. In this paper, we investigate the robustness of safety
fine-tuning in LLMs by systematically injecting Gaussian noise into model
activations. We show across multiple open-weight models that (1) Gaussian noise
raises harmful-output rates (p < 0.001) by up to 27%, (2) that deeper safety
fine-tuning affords no extra protection, and (3) that chain-of-thought
reasoning remains largely intact. The findings reveal critical vulnerabilities
in current safety alignment techniques and highlight the potential of
reasoning-based and reinforcement learning approaches as promising direction
for developing more robust AI safety systems. These results have important
implications for real-world deployment of LLMs in safety-critical applications
as these results imply that widely-deployed safety tuning methods can fail even
without adversarial prompts.


http://arxiv.org/abs/2505.09983
Sybil-based Virtual Data Poisoning Attacks in Federated Learning. (1%)
Changxun Zhu; Qilong Wu; Lingjuan Lyu; Shibei Xue
  Federated learning is vulnerable to poisoning attacks by malicious
adversaries. Existing methods often involve high costs to achieve effective
attacks. To address this challenge, we propose a sybil-based virtual data
poisoning attack, where a malicious client generates sybil nodes to amplify the
poisoning model's impact. To reduce neural network computational complexity, we
develop a virtual data generation method based on gradient matching. We also
design three schemes for target model acquisition, applicable to online local,
online global, and offline scenarios. In simulation, our method outperforms
other attack algorithms since our method can obtain a global target model under
non-independent uniformly distributed data.


http://arxiv.org/abs/2505.10066
Dark LLMs: The Growing Threat of Unaligned AI Models. (1%)
Michael Fire; Yitzhak Elbazis; Adi Wasenstein; Lior Rokach
  Large Language Models (LLMs) rapidly reshape modern life, advancing fields
from healthcare to education and beyond. However, alongside their remarkable
capabilities lies a significant threat: the susceptibility of these models to
jailbreaking. The fundamental vulnerability of LLMs to jailbreak attacks stems
from the very data they learn from. As long as this training data includes
unfiltered, problematic, or 'dark' content, the models can inherently learn
undesirable patterns or weaknesses that allow users to circumvent their
intended safety controls. Our research identifies the growing threat posed by
dark LLMs models deliberately designed without ethical guardrails or modified
through jailbreak techniques. In our research, we uncovered a universal
jailbreak attack that effectively compromises multiple state-of-the-art models,
enabling them to answer almost any question and produce harmful outputs upon
request. The main idea of our attack was published online over seven months
ago. However, many of the tested LLMs were still vulnerable to this attack.
Despite our responsible disclosure efforts, responses from major LLM providers
were often inadequate, highlighting a concerning gap in industry practices
regarding AI safety. As model training becomes more accessible and cheaper, and
as open-source LLMs proliferate, the risk of widespread misuse escalates.
Without decisive intervention, LLMs may continue democratizing access to
dangerous knowledge, posing greater risks than anticipated.


http://arxiv.org/abs/2505.09342
Evaluating the Robustness of Adversarial Defenses in Malware Detection Systems. (99%)
Mostafa Jafari; Alireza Shameli-Sendi
  Machine learning is a key tool for Android malware detection, effectively
identifying malicious patterns in apps. However, ML-based detectors are
vulnerable to evasion attacks, where small, crafted changes bypass detection.
Despite progress in adversarial defenses, the lack of comprehensive evaluation
frameworks in binary-constrained domains limits understanding of their
robustness. We introduce two key contributions. First, Prioritized Binary
Rounding, a technique to convert continuous perturbations into binary feature
spaces while preserving high attack success and low perturbation size. Second,
the sigma-binary attack, a novel adversarial method for binary domains,
designed to achieve attack goals with minimal feature changes. Experiments on
the Malscan dataset show that sigma-binary outperforms existing attacks and
exposes key vulnerabilities in state-of-the-art defenses. Defenses equipped
with adversary detectors, such as KDE, DLA, DNN+, and ICNN, exhibit significant
brittleness, with attack success rates exceeding 90% using fewer than 10
feature modifications and reaching 100% with just 20. Adversarially trained
defenses, including AT-rFGSM-k, AT-MaxMA, improves robustness under small
budgets but remains vulnerable to unrestricted perturbations, with attack
success rates of 99.45% and 96.62%, respectively. Although PAD-SMA demonstrates
strong robustness against state-of-the-art gradient-based adversarial attacks
by maintaining an attack success rate below 16.55%, the sigma-binary attack
significantly outperforms these methods, achieving a 94.56% success rate under
unrestricted perturbations. These findings highlight the critical need for
precise method like sigma-binary to expose hidden vulnerabilities in existing
defenses and support the development of more resilient malware detection
systems.


http://arxiv.org/abs/2505.09602
Adversarial Suffix Filtering: a Defense Pipeline for LLMs. (98%)
David Khachaturov; Robert Mullins
  Large Language Models (LLMs) are increasingly embedded in autonomous systems
and public-facing environments, yet they remain susceptible to jailbreak
vulnerabilities that may undermine their security and trustworthiness.
Adversarial suffixes are considered to be the current state-of-the-art
jailbreak, consistently outperforming simpler methods and frequently succeeding
even in black-box settings. Existing defenses rely on access to the internal
architecture of models limiting diverse deployment, increase memory and
computation footprints dramatically, or can be bypassed with simple prompt
engineering methods. We introduce $\textbf{Adversarial Suffix Filtering}$
(ASF), a lightweight novel model-agnostic defensive pipeline designed to
protect LLMs against adversarial suffix attacks. ASF functions as an input
preprocessor and sanitizer that detects and filters adversarially crafted
suffixes in prompts, effectively neutralizing malicious injections. We
demonstrate that ASF provides comprehensive defense capabilities across both
black-box and white-box attack settings, reducing the attack efficacy of
state-of-the-art adversarial suffix generation methods to below 4%, while only
minimally affecting the target model's capabilities in non-adversarial
scenarios.


http://arxiv.org/abs/2505.09820
Adversarial Attack on Large Language Models using Exponentiated Gradient Descent. (89%)
Sajib Biswas; Mao Nishino; Samuel Jacob Chacko; Xiuwen Liu
  As Large Language Models (LLMs) are widely used, understanding them
systematically is key to improving their safety and realizing their full
potential. Although many models are aligned using techniques such as
reinforcement learning from human feedback (RLHF), they are still vulnerable to
jailbreaking attacks. Some of the existing adversarial attack methods search
for discrete tokens that may jailbreak a target model while others try to
optimize the continuous space represented by the tokens of the model's
vocabulary. While techniques based on the discrete space may prove to be
inefficient, optimization of continuous token embeddings requires projections
to produce discrete tokens, which might render them ineffective. To fully
utilize the constraints and the structures of the space, we develop an
intrinsic optimization technique using exponentiated gradient descent with the
Bregman projection method to ensure that the optimized one-hot encoding always
stays within the probability simplex. We prove the convergence of the technique
and implement an efficient algorithm that is effective in jailbreaking several
widely used LLMs. We demonstrate the efficacy of the proposed technique using
five open-source LLMs on four openly available datasets. The results show that
the technique achieves a higher success rate with great efficiency compared to
three other state-of-the-art jailbreaking techniques. The source code for our
implementation is available at:
https://github.com/sbamit/Exponentiated-Gradient-Descent-LLM-Attack


http://arxiv.org/abs/2505.09603
DataMIL: Selecting Data for Robot Imitation Learning with Datamodels. (10%)
Shivin Dass; Alaa Khaddaj; Logan Engstrom; Aleksander Madry; Andrew Ilyas; Roberto Martín-Martín
  Recently, the robotics community has amassed ever larger and more diverse
datasets to train generalist robot policies. However, while these policies
achieve strong mean performance across a variety of tasks, they often
underperform on individual, specialized tasks and require further tuning on
newly acquired task-specific data. Combining task-specific data with carefully
curated subsets of large prior datasets via co-training can produce better
specialized policies, but selecting data naively may actually harm downstream
performance. To address this, we introduce DataMIL, a policy-driven data
selection framework built on the datamodels paradigm that reasons about data
selection in an end-to-end manner, using the policy itself to identify which
data points will most improve performance. Unlike standard practices that
filter data using human notions of quality (e.g., based on semantic or visual
similarity), DataMIL directly optimizes data selection for task success,
allowing us to select data that enhance the policy while dropping data that
degrade it. To avoid performing expensive rollouts in the environment during
selection, we use a novel surrogate loss function on task-specific data,
allowing us to use DataMIL in the real world without degrading performance. We
validate our approach on a suite of more than 60 simulation and real-world
manipulation tasks - most notably showing successful data selection from the
Open X-Embodiment datasets-demonstrating consistent gains in success rates and
superior performance over multiple baselines. Our results underscore the
importance of end-to-end, performance-aware data selection for unlocking the
potential of large prior datasets in robotics. More information at
https://robin-lab.cs.utexas.edu/datamodels4imitation/


http://arxiv.org/abs/2505.09368
RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo. (2%)
Jenny Schmalfuss; Victor Oei; Lukas Mehl; Madlen Bartsch; Shashank Agnihotri; Margret Keuper; Andrés Bruhn
  Standard benchmarks for optical flow, scene flow, and stereo vision
algorithms generally focus on model accuracy rather than robustness to image
corruptions like noise or rain. Hence, the resilience of models to such
real-world perturbations is largely unquantified. To address this, we present
RobustSpring, a comprehensive dataset and benchmark for evaluating robustness
to image corruptions for optical flow, scene flow, and stereo models.
RobustSpring applies 20 different image corruptions, including noise, blur,
color changes, quality degradations, and weather distortions, in a time-,
stereo-, and depth-consistent manner to the high-resolution Spring dataset,
creating a suite of 20,000 corrupted images that reflect challenging
conditions. RobustSpring enables comparisons of model robustness via a new
corruption robustness metric. Integration with the Spring benchmark enables
public two-axis evaluations of both accuracy and robustness. We benchmark a
curated selection of initial models, observing that accurate models are not
necessarily robust and that robustness varies widely by corruption type.
RobustSpring is a new computer vision benchmark that treats robustness as a
first-class citizen to foster models that combine accuracy with resilience. It
will be available at https://spring-benchmark.org.


http://arxiv.org/abs/2505.09743
Guardian Positioning System (GPS) for Location Based Services. (1%)
Wenjie Liu; Panos Papadimitratos
  Location-based service (LBS) applications proliferate and support
transportation, entertainment, and more. Modern mobile platforms, with
smartphones being a prominent example, rely on terrestrial and satellite
infrastructures (e.g., global navigation satellite system (GNSS) and
crowdsourced Wi-Fi, Bluetooth, cellular, and IP databases) for correct
positioning. However, they are vulnerable to attacks that manipulate positions
to control and undermine LBS functionality -- thus enabling the scamming of
users or services. Our work reveals that GNSS spoofing attacks succeed even
though smartphones have multiple sources of positioning information. Moreover,
that Wi-Fi spoofing attacks with GNSS jamming are surprisingly effective. More
concerning is the evidence that sophisticated, coordinated spoofing attacks are
highly effective. Attacks can target GNSS in combination with other positioning
methods, thus defenses that assume that only GNSS is under attack cannot be
effective. More so, resilient GNSS receivers and special-purpose antennas are
not feasible on smartphones. To address this gap, we propose an extended
receiver autonomous integrity monitoring (RAIM) framework that leverages the
readily available, redundant, often so-called opportunistic positioning
information on off-the-shelf platforms. We jointly use onboard sensors,
terrestrial infrastructures, and GNSS. We show that our extended RAIM framework
improves resilience against location spoofing, e.g., achieving a detection
accuracy improvement of up to 24-58% compared to the state-of-the-art
algorithms and location providers; detecting attacks within 5 seconds, with a
low false positive rate.


http://arxiv.org/abs/2505.08999
Towards Adaptive Meta-Gradient Adversarial Examples for Visual Tracking. (99%)
Wei-Long Tian; Peng Gao; Xiao Liu; Long Xu; Hamido Fujita; Hanan Aljuai; Mao-Li Wang
  In recent years, visual tracking methods based on convolutional neural
networks and Transformers have achieved remarkable performance and have been
successfully applied in fields such as autonomous driving. However, the
numerous security issues exposed by deep learning models have gradually
affected the reliable application of visual tracking methods in real-world
scenarios. Therefore, how to reveal the security vulnerabilities of existing
visual trackers through effective adversarial attacks has become a critical
problem that needs to be addressed. To this end, we propose an adaptive
meta-gradient adversarial attack (AMGA) method for visual tracking. This method
integrates multi-model ensembles and meta-learning strategies, combining
momentum mechanisms and Gaussian smoothing, which can significantly enhance the
transferability and attack effectiveness of adversarial examples. AMGA randomly
selects models from a large model repository, constructs diverse tracking
scenarios, and iteratively performs both white- and black-box adversarial
attacks in each scenario, optimizing the gradient directions of each model.
This paradigm minimizes the gap between white- and black-box adversarial
attacks, thus achieving excellent attack performance in black-box scenarios.
Extensive experimental results on large-scale datasets such as OTB2015, LaSOT,
and GOT-10k demonstrate that AMGA significantly improves the attack
performance, transferability, and deception of adversarial examples. Codes and
data are available at https://github.com/pgao-lab/AMGA.


http://arxiv.org/abs/2505.08835
Robustness Analysis against Adversarial Patch Attacks in Fully Unmanned Stores. (98%)
Hyunsik Na; Wonho Lee; Seungdeok Roh; Sohee Park; Daeseon Choi
  The advent of convenient and efficient fully unmanned stores equipped with
artificial intelligence-based automated checkout systems marks a new era in
retail. However, these systems have inherent artificial intelligence security
vulnerabilities, which are exploited via adversarial patch attacks,
particularly in physical environments. This study demonstrated that adversarial
patches can severely disrupt object detection models used in unmanned stores,
leading to issues such as theft, inventory discrepancies, and interference. We
investigated three types of adversarial patch attacks -- Hiding, Creating, and
Altering attacks -- and highlighted their effectiveness. We also introduce the
novel color histogram similarity loss function by leveraging attacker knowledge
of the color information of a target class object. Besides the traditional
confusion-matrix-based attack success rate, we introduce a new
bounding-boxes-based metric to analyze the practical impact of these attacks.
Starting with attacks on object detection models trained on snack and fruit
datasets in a digital environment, we evaluated the effectiveness of
adversarial patches in a physical testbed that mimicked a real unmanned store
with RGB cameras and realistic conditions. Furthermore, we assessed the
robustness of these attacks in black-box scenarios, demonstrating that shadow
attacks can enhance success rates of attacks even without direct access to
model parameters. Our study underscores the necessity for robust defense
strategies to protect unmanned stores from adversarial threats. Highlighting
the limitations of the current defense mechanisms in real-time detection
systems and discussing various proactive measures, we provide insights into
improving the robustness of object detection models and fortifying unmanned
retail environments against these attacks.


http://arxiv.org/abs/2505.11532
Revisiting Adversarial Perception Attacks and Defense Methods on Autonomous Driving Systems. (96%)
Cheng Chen; Yuhong Wang; Nafis S Munir; Xiangwei Zhou; Xugui Zhou
  Autonomous driving systems (ADS) increasingly rely on deep learning-based
perception models, which remain vulnerable to adversarial attacks. In this
paper, we revisit adversarial attacks and defense methods, focusing on road
sign recognition and lead object detection and prediction (e.g., relative
distance). Using a Level-2 production ADS, OpenPilot by Comma$.$ai, and the
widely adopted YOLO model, we systematically examine the impact of adversarial
perturbations and assess defense techniques, including adversarial training,
image processing, contrastive learning, and diffusion models. Our experiments
highlight both the strengths and limitations of these methods in mitigating
complex attacks. Through targeted evaluations of model robustness, we aim to
provide deeper insights into the vulnerabilities of ADS perception systems and
contribute guidance for developing more resilient defense strategies.


http://arxiv.org/abs/2505.08234
Removing Watermarks with Partial Regeneration using Semantic Information. (15%)
Krti Tallam; John Kevin Cava; Caleb Geniesse; N. Benjamin Erichson; Michael W. Mahoney
  As AI-generated imagery becomes ubiquitous, invisible watermarks have emerged
as a primary line of defense for copyright and provenance. The newest
watermarking schemes embed semantic signals - content-aware patterns that are
designed to survive common image manipulations - yet their true robustness
against adaptive adversaries remains under-explored. We expose a previously
unreported vulnerability and introduce SemanticRegen, a three-stage, label-free
attack that erases state-of-the-art semantic and invisible watermarks while
leaving an image's apparent meaning intact. Our pipeline (i) uses a
vision-language model to obtain fine-grained captions, (ii) extracts foreground
masks with zero-shot segmentation, and (iii) inpaints only the background via
an LLM-guided diffusion model, thereby preserving salient objects and style
cues. Evaluated on 1,000 prompts across four watermarking systems - TreeRing,
StegaStamp, StableSig, and DWT/DCT - SemanticRegen is the only method to defeat
the semantic TreeRing watermark (p = 0.10 > 0.05) and reduces bit-accuracy
below 0.75 for the remaining schemes, all while maintaining high perceptual
quality (masked SSIM = 0.94 +/- 0.01). We further introduce masked SSIM (mSSIM)
to quantify fidelity within foreground regions, showing that our attack
achieves up to 12 percent higher mSSIM than prior diffusion-based attackers.
These results highlight an urgent gap between current watermark defenses and
the capabilities of adaptive, semantics-aware adversaries, underscoring the
need for watermarking algorithms that are resilient to content-preserving
regenerative attacks.


http://arxiv.org/abs/2505.08292
On the Account Security Risks Posed by Password Strength Meters. (1%)
Ming Xu; Weili Han; Jitao Yu; Jing Liu; Xinyi Zhang; Yun Lin; Jin Song Dong
  Password strength meters (PSMs) have been widely used by websites to gauge
password strength, encouraging users to create stronger passwords. Popular
data-driven PSMs, e.g., based on Markov, Probabilistic Context-free Grammar
(PCFG) and neural networks, alarm strength based on a model learned from real
passwords. Despite their proven effectiveness, the secure utility that arises
from the leakage of trained passwords remains largely overlooked. To address
this gap, we analyze 11 PSMs and find that 5 data-driven meters are vulnerable
to membership inference attacks that expose their trained passwords, and
seriously, 3 rule-based meters openly disclose their blocked passwords. We
specifically design a PSM privacy leakage evaluation approach, and uncover that
a series of general data-driven meters are vulnerable to leaking between 10^4
to 10^5 trained passwords, with the PCFG-based models being more vulnerable
than other counterparts; furthermore, we aid in deriving insights that the
inherent utility-privacy tradeoff is not as severe as previously thought. To
further exploit the risks, we develop novel meter-aware attacks when a clever
attacker can filter the used passwords during compromising accounts on websites
using the meter, and experimentally show that attackers targeting websites that
deployed the popular Zxcvbn meter can compromise an additional 5.84% user
accounts within 10 attempts, demonstrating the urgent need for
privacy-preserving PSMs that protect the confidentiality of the meter's used
passwords. Finally, we sketch some counter-measures to mitigate these threats.


http://arxiv.org/abs/2505.08459
Strategy-Augmented Planning for Large Language Models via Opponent Exploitation. (1%)
Shuai Xu; Sijia Cui; Yanna Wang; Bo Xu; Qi Wang
  Efficiently modeling and exploiting opponents is a long-standing challenge in
adversarial domains. Large Language Models (LLMs) trained on extensive textual
data have recently demonstrated outstanding performance in general tasks,
introducing new research directions for opponent modeling. Some studies
primarily focus on directly using LLMs to generate decisions based on the
elaborate prompt context that incorporates opponent descriptions, while these
approaches are limited to scenarios where LLMs possess adequate domain
expertise. To address that, we introduce a two-stage Strategy-Augmented
Planning (SAP) framework that significantly enhances the opponent exploitation
capabilities of LLM-based agents by utilizing a critical component, the
Strategy Evaluation Network (SEN). Specifically, in the offline stage, we
construct an explicit strategy space and subsequently collect strategy-outcome
pair data for training the SEN network. During the online phase, SAP
dynamically recognizes the opponent's strategies and greedily exploits them by
searching best response strategy on the well-trained SEN, finally translating
strategy to a course of actions by carefully designed prompts. Experimental
results show that SAP exhibits robust generalization capabilities, allowing it
to perform effectively not only against previously encountered opponent
strategies but also against novel, unseen strategies. In the MicroRTS
environment, SAP achieves a $85.35\%$ performance improvement over baseline
methods and matches the competitiveness of reinforcement learning approaches
against state-of-the-art (SOTA) rule-based AI. Our code is available at
https://github.com/hsushuai/SAP.


http://arxiv.org/abs/2505.08022
Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks. (86%)
Steffen Schotthöfer; H. Lexie Yang; Stefan Schnake
  Deployment of neural networks on resource-constrained devices demands models
that are both compact and robust to adversarial inputs. However, compression
and adversarial robustness often conflict. In this work, we introduce a
dynamical low-rank training scheme enhanced with a novel spectral regularizer
that controls the condition number of the low-rank core in each layer. This
approach mitigates the sensitivity of compressed models to adversarial
perturbations without sacrificing clean accuracy. The method is model- and
data-agnostic, computationally efficient, and supports rank adaptivity to
automatically compress the network at hand. Extensive experiments across
standard architectures, datasets, and adversarial attacks show the regularized
networks can achieve over 94% compression while recovering or improving
adversarial accuracy relative to uncompressed baselines.


http://arxiv.org/abs/2505.07546
GRADA: Graph-based Reranker against Adversarial Documents Attack. (69%)
Jingjie Zheng; Aryo Pradipta Gema; Giwon Hong; Xuanli He; Pasquale Minervini; Youcheng Sun; Qiongkai Xu
  Retrieval Augmented Generation (RAG) frameworks improve the accuracy of large
language models (LLMs) by integrating external knowledge from retrieved
documents, thereby overcoming the limitations of models' static intrinsic
knowledge. However, these systems are susceptible to adversarial attacks that
manipulate the retrieval process by introducing documents that are adversarial
yet semantically similar to the query. Notably, while these adversarial
documents resemble the query, they exhibit weak similarity to benign documents
in the retrieval set. Thus, we propose a simple yet effective Graph-based
Reranking against Adversarial Document Attacks (GRADA) framework aiming at
preserving retrieval quality while significantly reducing the success of
adversaries. Our study evaluates the effectiveness of our approach through
experiments conducted on five LLMs: GPT-3.5-Turbo, GPT-4o, Llama3.1-8b,
Llama3.1-70b, and Qwen2.5-7b. We use three datasets to assess performance, with
results from the Natural Questions dataset demonstrating up to an 80% reduction
in attack success rates while maintaining minimal loss in accuracy.


http://arxiv.org/abs/2505.07584
SecReEvalBench: A Multi-turned Security Resilience Evaluation Benchmark for Large Language Models. (26%)
Huining Cui; Wei Liu
  The increasing deployment of large language models in security-sensitive
domains necessitates rigorous evaluation of their resilience against
adversarial prompt-based attacks. While previous benchmarks have focused on
security evaluations with limited and predefined attack domains, such as
cybersecurity attacks, they often lack a comprehensive assessment of
intent-driven adversarial prompts and the consideration of real-life
scenario-based multi-turn attacks. To address this gap, we present
SecReEvalBench, the Security Resilience Evaluation Benchmark, which defines
four novel metrics: Prompt Attack Resilience Score, Prompt Attack Refusal Logic
Score, Chain-Based Attack Resilience Score and Chain-Based Attack Rejection
Time Score. Moreover, SecReEvalBench employs six questioning sequences for
model assessment: one-off attack, successive attack, successive reverse attack,
alternative attack, sequential ascending attack with escalating threat levels
and sequential descending attack with diminishing threat levels. In addition,
we introduce a dataset customized for the benchmark, which incorporates both
neutral and malicious prompts, categorised across seven security domains and
sixteen attack techniques. In applying this benchmark, we systematically
evaluate five state-of-the-art open-weighted large language models, Llama 3.1,
Gemma 2, Mistral v0.3, DeepSeek-R1 and Qwen 3. Our findings offer critical
insights into the strengths and weaknesses of modern large language models in
defending against evolving adversarial threats. The SecReEvalBench dataset is
publicly available at
https://kaggle.com/datasets/5a7ee22cf9dab6c93b55a73f630f6c9b42e936351b0ae98fbae6ddaca7fe248d,
which provides a groundwork for advancing research in large language model
security.


http://arxiv.org/abs/2505.07775
Must Read: A Systematic Survey of Computational Persuasion. (3%)
Nimet Beyza Bozdag; Shuhaib Mehri; Xiaocheng Yang; Hyeonjeong Ha; Zirui Cheng; Esin Durmus; Jiaxuan You; Heng Ji; Gokhan Tur; Dilek Hakkani-Tür
  Persuasion is a fundamental aspect of communication, influencing
decision-making across diverse contexts, from everyday conversations to
high-stakes scenarios such as politics, marketing, and law. The rise of
conversational AI systems has significantly expanded the scope of persuasion,
introducing both opportunities and risks. AI-driven persuasion can be leveraged
for beneficial applications, but also poses threats through manipulation and
unethical influence. Moreover, AI systems are not only persuaders, but also
susceptible to persuasion, making them vulnerable to adversarial attacks and
bias reinforcement. Despite rapid advancements in AI-generated persuasive
content, our understanding of what makes persuasion effective remains limited
due to its inherently subjective and context-dependent nature. In this survey,
we provide a comprehensive overview of computational persuasion, structured
around three key perspectives: (1) AI as a Persuader, which explores
AI-generated persuasive content and its applications; (2) AI as a Persuadee,
which examines AI's susceptibility to influence and manipulation; and (3) AI as
a Persuasion Judge, which analyzes AI's role in evaluating persuasive
strategies, detecting manipulation, and ensuring ethical persuasion. We
introduce a taxonomy for computational persuasion research and discuss key
challenges, including evaluating persuasiveness, mitigating manipulative
persuasion, and developing responsible AI-driven persuasive systems. Our survey
outlines future research directions to enhance the safety, fairness, and
effectiveness of AI-powered persuasion while addressing the risks posed by
increasingly capable language models.


http://arxiv.org/abs/2505.06860
DP-TRAE: A Dual-Phase Merging Transferable Reversible Adversarial Example for Image Privacy Protection. (99%)
Xia Du; Jiajie Zhu; Jizhe Zhou; Chi-man Pun; Zheng Lin; Cong Wu; Zhe Chen; Jun Luo
  In the field of digital security, Reversible Adversarial Examples (RAE)
combine adversarial attacks with reversible data hiding techniques to
effectively protect sensitive data and prevent unauthorized analysis by
malicious Deep Neural Networks (DNNs). However, existing RAE techniques
primarily focus on white-box attacks, lacking a comprehensive evaluation of
their effectiveness in black-box scenarios. This limitation impedes their
broader deployment in complex, dynamic environments. Further more, traditional
black-box attacks are often characterized by poor transferability and high
query costs, significantly limiting their practical applicability. To address
these challenges, we propose the Dual-Phase Merging Transferable Reversible
Attack method, which generates highly transferable initial adversarial
perturbations in a white-box model and employs a memory augmented black-box
strategy to effectively mislead target mod els. Experimental results
demonstrate the superiority of our approach, achieving a 99.0% attack success
rate and 100% recovery rate in black-box scenarios, highlighting its robustness
in privacy protection. Moreover, we successfully implemented a black-box attack
on a commercial model, further substantiating the potential of this approach
for practical use.


http://arxiv.org/abs/2505.06889
IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method. (47%)
Mihyeon Kim; Juhyoung Park; Youngbin Kim
  Pre-trained Language Models (PLMs) have achieved remarkable performance on
diverse NLP tasks through pre-training and fine-tuning. However, fine-tuning
the model with a large number of parameters on limited downstream datasets
often leads to vulnerability to adversarial attacks, causing overfitting of the
model on standard datasets.
  To address these issues, we propose IM-BERT from the perspective of a dynamic
system by conceptualizing a layer of BERT as a solution of Ordinary
Differential Equations (ODEs). Under the situation of initial value
perturbation, we analyze the numerical stability of two main numerical ODE
solvers: the explicit and implicit Euler approaches.
  Based on these analyses, we introduce a numerically robust IM-connection
incorporating BERT's layers. This strategy enhances the robustness of PLMs
against adversarial attacks, even in low-resource scenarios, without
introducing additional parameters or adversarial training strategies.
  Experimental results on the adversarial GLUE (AdvGLUE) dataset validate the
robustness of IM-BERT under various conditions. Compared to the original BERT,
IM-BERT exhibits a performance improvement of approximately 8.3\%p on the
AdvGLUE dataset. Furthermore, in low-resource scenarios, IM-BERT outperforms
BERT by achieving 5.9\%p higher accuracy.


http://arxiv.org/abs/2505.06958
A Formally Verified Robustness Certifier for Neural Networks (Extended Version). (15%)
James Tobler; Hira Taqdees Syeda; Toby Murray
  Neural networks are often susceptible to minor perturbations in input that
cause them to misclassify. A recent solution to this problem is the use of
globally-robust neural networks, which employ a function to certify that the
classification of an input cannot be altered by such a perturbation. Outputs
that pass this test are called certified robust. However, to the authors'
knowledge, these certification functions have not yet been verified at the
implementation level. We demonstrate how previous unverified implementations
are exploitably unsound in certain circumstances. Moreover, they often rely on
approximation-based algorithms, such as power iteration, that (perhaps
surprisingly) do not guarantee soundness. To provide assurance that a given
output is robust, we implemented and formally verified a certification function
for globally-robust neural networks in Dafny. We describe the program, its
specifications, and the important design decisions taken for its implementation
and verification, as well as our experience applying it in practice.


http://arxiv.org/abs/2505.06843
Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety. (12%)
Zihan Guan; Mengxuan Hu; Ronghang Zhu; Sheng Li; Anil Vullikanti
  Recent studies have uncovered a troubling vulnerability in the fine-tuning
stage of large language models (LLMs): even fine-tuning on entirely benign
datasets can lead to a significant increase in the harmfulness of LLM outputs.
Building on this finding, our red teaming study takes this threat one step
further by developing a more effective attack. Specifically, we analyze and
identify samples within benign datasets that contribute most to safety
degradation, then fine-tune LLMs exclusively on these samples. We approach this
problem from an outlier detection perspective and propose Self-Inf-N, to detect
and extract outliers for fine-tuning. Our findings reveal that fine-tuning LLMs
on 100 outlier samples selected by Self-Inf-N in the benign datasets severely
compromises LLM safety alignment. Extensive experiments across seven mainstream
LLMs demonstrate that our attack exhibits high transferability across different
architectures and remains effective in practical scenarios. Alarmingly, our
results indicate that most existing mitigation strategies fail to defend
against this attack, underscoring the urgent need for more robust alignment
safeguards. Codes are available at
https://github.com/GuanZihan/Benign-Samples-Matter.


http://arxiv.org/abs/2505.07167
One Trigger Token Is Enough: A Defense Strategy for Balancing Safety and Usability in Large Language Models. (11%)
Haoran Gu; Handing Wang; Yi Mei; Mengjie Zhang; Yaochu Jin
  Large Language Models (LLMs) have been extensively used across diverse
domains, including virtual assistants, automated code generation, and
scientific research. However, they remain vulnerable to jailbreak attacks,
which manipulate the models into generating harmful responses despite safety
alignment. Recent studies have shown that current safety-aligned LLMs often
undergo the shallow safety alignment, where the first few tokens largely
determine whether the response will be harmful. Through comprehensive
observations, we find that safety-aligned LLMs and various defense strategies
generate highly similar initial tokens in their refusal responses, which we
define as safety trigger tokens. Building on this insight, we propose
\texttt{D-STT}, a simple yet effective defense algorithm that identifies and
explicitly decodes safety trigger tokens of the given safety-aligned LLM to
trigger the model's learned safety patterns. In this process, the safety
trigger is constrained to a single token, which effectively preserves model
usability by introducing minimum intervention in the decoding process.
Extensive experiments across diverse jailbreak attacks and benign prompts
demonstrate that \ours significantly reduces output harmfulness while
preserving model usability and incurring negligible response time overhead,
outperforming ten baseline methods.


http://arxiv.org/abs/2505.07149
AugMixCloak: A Defense against Membership Inference Attacks via Image Transformation. (8%)
Heqing Ren; Chao Feng; Alberto Huertas; Burkhard Stiller
  Traditional machine learning (ML) raises serious privacy concerns, while
federated learning (FL) mitigates the risk of data leakage by keeping data on
local devices. However, the training process of FL can still leak sensitive
information, which adversaries may exploit to infer private data. One of the
most prominent threats is the membership inference attack (MIA), where the
adversary aims to determine whether a particular data record was part of the
training set.
  This paper addresses this problem through a two-stage defense called
AugMixCloak. The core idea is to apply data augmentation and principal
component analysis (PCA)-based information fusion to query images, which are
detected by perceptual hashing (pHash) as either identical to or highly similar
to images in the training set. Experimental results show that AugMixCloak
successfully defends against both binary classifier-based MIA and metric-based
MIA across five datasets and various decentralized FL (DFL) topologies.
Compared with regularization-based defenses, AugMixCloak demonstrates stronger
protection. Compared with confidence score masking, AugMixCloak exhibits better
generalization.


http://arxiv.org/abs/2505.08804
TokenProber: Jailbreaking Text-to-image Models via Fine-grained Word Impact Analysis. (5%)
Longtian Wang; Xiaofei Xie; Tianlin Li; Yuhan Zhi; Chao Shen
  Text-to-image (T2I) models have significantly advanced in producing
high-quality images. However, such models have the ability to generate images
containing not-safe-for-work (NSFW) content, such as pornography, violence,
political content, and discrimination. To mitigate the risk of generating NSFW
content, refusal mechanisms, i.e., safety checkers, have been developed to
check potential NSFW content. Adversarial prompting techniques have been
developed to evaluate the robustness of the refusal mechanisms. The key
challenge remains to subtly modify the prompt in a way that preserves its
sensitive nature while bypassing the refusal mechanisms. In this paper, we
introduce TokenProber, a method designed for sensitivity-aware differential
testing, aimed at evaluating the robustness of the refusal mechanisms in T2I
models by generating adversarial prompts. Our approach is based on the key
observation that adversarial prompts often succeed by exploiting discrepancies
in how T2I models and safety checkers interpret sensitive content. Thus, we
conduct a fine-grained analysis of the impact of specific words within prompts,
distinguishing between dirty words that are essential for NSFW content
generation and discrepant words that highlight the different sensitivity
assessments between T2I models and safety checkers. Through the
sensitivity-aware mutation, TokenProber generates adversarial prompts, striking
a balance between maintaining NSFW content generation and evading detection.
Our evaluation of TokenProber against 5 safety checkers on 3 popular T2I
models, using 324 NSFW prompts, demonstrates its superior effectiveness in
bypassing safety filters compared to existing methods (e.g., 54%+ increase on
average), highlighting TokenProber's ability to uncover robustness issues in
the existing refusal mechanisms.


http://arxiv.org/abs/2505.06579
POISONCRAFT: Practical Poisoning of Retrieval-Augmented Generation for Large Language Models. (83%)
Yangguang Shao; Xinjie Lin; Haozheng Luo; Chengshang Hou; Gang Xiong; Jiahao Yu; Junzheng Shi
  Large language models (LLMs) have achieved remarkable success in various
domains, primarily due to their strong capabilities in reasoning and generating
human-like text. Despite their impressive performance, LLMs are susceptible to
hallucinations, which can lead to incorrect or misleading outputs. This is
primarily due to the lack of up-to-date knowledge or domain-specific
information. Retrieval-augmented generation (RAG) is a promising approach to
mitigate hallucinations by leveraging external knowledge sources. However, the
security of RAG systems has not been thoroughly studied. In this paper, we
study a poisoning attack on RAG systems named POISONCRAFT, which can mislead
the model to refer to fraudulent websites. Compared to existing poisoning
attacks on RAG systems, our attack is more practical as it does not require
access to the target user query's info or edit the user query. It not only
ensures that injected texts can be retrieved by the model, but also ensures
that the LLM will be misled to refer to the injected texts in its response. We
demonstrate the effectiveness of POISONCRAFTacross different datasets,
retrievers, and language models in RAG pipelines, and show that it remains
effective when transferred across retrievers, including black-box systems.
Moreover, we present a case study revealing how the attack influences both the
retrieval behavior and the step-by-step reasoning trace within the generation
model, and further evaluate the robustness of POISONCRAFTunder multiple defense
mechanisms. These results validate the practicality of our threat model and
highlight a critical security risk for RAG systems deployed in real-world
applications. We release our
code\footnote{https://github.com/AndyShaw01/PoisonCraft} to support future
research on the security and robustness of RAG systems in real-world settings.


http://arxiv.org/abs/2505.06679
T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks. (64%)
Jiayang Liu; Siyuan Liang; Shiqian Zhao; Rongcheng Tu; Wenbo Zhou; Aishan Liu; Dacheng Tao; Siew Kei Lam
  In recent years, fueled by the rapid advancement of diffusion models,
text-to-video (T2V) generation models have achieved remarkable progress, with
notable examples including Pika, Luma, Kling, and Open-Sora. Although these
models exhibit impressive generative capabilities, they also expose significant
security risks due to their vulnerability to jailbreak attacks, where the
models are manipulated to produce unsafe content such as pornography, violence,
or discrimination. Existing works such as T2VSafetyBench provide preliminary
benchmarks for safety evaluation, but lack systematic methods for thoroughly
exploring model vulnerabilities. To address this gap, we are the first to
formalize the T2V jailbreak attack as a discrete optimization problem and
propose a joint objective-based optimization framework, called T2V-OptJail.
This framework consists of two key optimization goals: bypassing the built-in
safety filtering mechanisms to increase the attack success rate, preserving
semantic consistency between the adversarial prompt and the unsafe input
prompt, as well as between the generated video and the unsafe input prompt, to
enhance content controllability. In addition, we introduce an iterative
optimization strategy guided by prompt variants, where multiple semantically
equivalent candidates are generated in each round, and their scores are
aggregated to robustly guide the search toward optimal adversarial prompts. We
conduct large-scale experiments on several T2V models, covering both
open-source models and real commercial closed-source models. The experimental
results show that the proposed method improves 11.4% and 10.0% over the
existing state-of-the-art method in terms of attack success rate assessed by
GPT-4, attack success rate assessed by human accessors, respectively, verifying
the significant advantages of the method in terms of attack effectiveness and
content control.


http://arxiv.org/abs/2505.06643
Practical Reasoning Interruption Attacks on Reasoning Large Language Models. (33%)
Yu Cui; Cong Zuo
  Reasoning large language models (RLLMs) have demonstrated outstanding
performance across a variety of tasks, yet they also expose numerous security
vulnerabilities. Most of these vulnerabilities have centered on the generation
of unsafe content. However, recent work has identified a distinct
"thinking-stopped" vulnerability in DeepSeek-R1: under adversarial prompts, the
model's reasoning process ceases at the system level and produces an empty
final answer. Building upon this vulnerability, researchers developed a novel
prompt injection attack, termed reasoning interruption attack, and also offered
an initial analysis of its root cause. Through extensive experiments, we verify
the previous analyses, correct key errors based on three experimental findings,
and present a more rigorous explanation of the fundamental causes driving the
vulnerability. Moreover, existing attacks typically require over 2,000 tokens,
impose significant overhead, reduce practicality, and are easily detected. To
overcome these limitations, we propose the first practical reasoning
interruption attack. It succeeds with just 109 tokens by exploiting our newly
uncovered "reasoning token overflow" (RTO) effect to overwrite the model's
final answer, forcing it to return an invalid response. Experimental results
demonstrate that our proposed attack is highly effective. Furthermore, we
discover that the method for triggering RTO differs between the official
DeepSeek-R1 release and common unofficial deployments. As a broadened
application of RTO, we also construct a novel jailbreak attack that enables the
transfer of unsafe content within the reasoning tokens into final answer,
thereby exposing it to the user. Our work carries significant implications for
enhancing the security of RLLMs.


http://arxiv.org/abs/2505.06477
Learning from the Good Ones: Risk Profiling-Based Defenses Against Evasion Attacks on DNNs. (99%)
Mohammed Elnawawy; Gargi Mitra; Shahrear Iqbal; Karthik Pattabiraman
  Safety-critical applications such as healthcare and autonomous vehicles use
deep neural networks (DNN) to make predictions and infer decisions. DNNs are
susceptible to evasion attacks, where an adversary crafts a malicious data
instance to trick the DNN into making wrong decisions at inference time.
Existing defenses that protect DNNs against evasion attacks are either static
or dynamic. Static defenses are computationally efficient but do not adapt to
the evolving threat landscape, while dynamic defenses are adaptable but suffer
from an increased computational overhead. To combine the best of both worlds,
in this paper, we propose a novel risk profiling framework that uses a
risk-aware strategy to selectively train static defenses using victim instances
that exhibit the most resilient features and are hence more resilient against
an evasion attack. We hypothesize that training existing defenses on instances
that are less vulnerable to the attack enhances the adversarial detection rate
by reducing false negatives. We evaluate the efficacy of our risk-aware
selective training strategy on a blood glucose management system that
demonstrates how training static anomaly detectors indiscriminately may result
in an increased false negative rate, which could be life-threatening in
safety-critical applications. Our experiments show that selective training on
the less vulnerable patients achieves a recall increase of up to 27.5\% with
minimal impact on precision compared to indiscriminate training.


http://arxiv.org/abs/2505.06134
Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation. (83%)
Julian F. Schumann; Jeroen Hagenus; Frederik Baymler Mathiesen; Arkady Zgonnikov
  Trajectory prediction is a key element of autonomous vehicle systems,
enabling them to anticipate and react to the movements of other road users.
Evaluating the robustness of prediction models against adversarial attacks is
essential to ensure their reliability in real-world traffic. However, current
approaches tend to focus on perturbing the past positions of surrounding
agents, which can generate unrealistic scenarios and overlook critical
vulnerabilities. This limitation may result in overly optimistic assessments of
model performance in real-world conditions.
  In this work, we demonstrate that perturbing not just past but also future
states of adversarial agents can uncover previously undetected weaknesses and
thereby provide a more rigorous evaluation of model robustness. Our novel
approach incorporates dynamic constraints and preserves tactical behaviors,
enabling more effective and realistic adversarial attacks. We introduce new
performance measures to assess the realism and impact of these adversarial
trajectories. Testing our method on a state-of-the-art prediction model
revealed significant increases in prediction errors and collision rates under
adversarial conditions. Qualitative analysis further showed that our attacks
can expose critical weaknesses, such as the inability of the model to detect
potential collisions in what appear to be safe predictions. These results
underscore the need for more comprehensive adversarial testing to better
evaluate and improve the reliability of trajectory prediction models for
autonomous vehicles.


http://arxiv.org/abs/2505.05872
A Taxonomy of Attacks and Defenses in Split Learning. (15%)
Aqsa Shabbir; Halil İbrahim Kanpak; Alptekin Küpçü; Sinem Sav
  Split Learning (SL) has emerged as a promising paradigm for distributed deep
learning, allowing resource-constrained clients to offload portions of their
model computation to servers while maintaining collaborative learning. However,
recent research has demonstrated that SL remains vulnerable to a range of
privacy and security threats, including information leakage, model inversion,
and adversarial attacks. While various defense mechanisms have been proposed, a
systematic understanding of the attack landscape and corresponding
countermeasures is still lacking. In this study, we present a comprehensive
taxonomy of attacks and defenses in SL, categorizing them along three key
dimensions: employed strategies, constraints, and effectiveness. Furthermore,
we identify key open challenges and research gaps in SL based on our
systematization, highlighting potential future directions.


http://arxiv.org/abs/2505.05849
AgentXploit: End-to-End Redteaming of Black-Box AI Agents. (5%)
Zhun Wang; Vincent Siu; Zhe Ye; Tianneng Shi; Yuzhou Nie; Xuandong Zhao; Chenguang Wang; Wenbo Guo; Dawn Song
  The strong planning and reasoning capabilities of Large Language Models
(LLMs) have fostered the development of agent-based systems capable of
leveraging external tools and interacting with increasingly complex
environments. However, these powerful features also introduce a critical
security risk: indirect prompt injection, a sophisticated attack vector that
compromises the core of these agents, the LLM, by manipulating contextual
information rather than direct user prompts. In this work, we propose a generic
black-box fuzzing framework, AgentXploit, designed to automatically discover
and exploit indirect prompt injection vulnerabilities across diverse LLM
agents. Our approach starts by constructing a high-quality initial seed corpus,
then employs a seed selection algorithm based on Monte Carlo Tree Search (MCTS)
to iteratively refine inputs, thereby maximizing the likelihood of uncovering
agent weaknesses. We evaluate AgentXploit on two public benchmarks, AgentDojo
and VWA-adv, where it achieves 71% and 70% success rates against agents based
on o3-mini and GPT-4o, respectively, nearly doubling the performance of
baseline attacks. Moreover, AgentXploit exhibits strong transferability across
unseen tasks and internal LLMs, as well as promising results against defenses.
Beyond benchmark evaluations, we apply our attacks in real-world environments,
successfully misleading agents to navigate to arbitrary URLs, including
malicious sites.


http://arxiv.org/abs/2505.06364
LATENT: LLM-Augmented Trojan Insertion and Evaluation Framework for Analog Netlist Topologies. (2%)
Jayeeta Chaudhuri; Arjun Chaudhuri; Krishnendu Chakrabarty
  Analog and mixed-signal (A/MS) integrated circuits (ICs) are integral to
safety-critical applications. However, the globalization and outsourcing of
A/MS ICs to untrusted third-party foundries expose them to security threats,
particularly analog Trojans. Unlike digital Trojans which have been extensively
studied, analog Trojans remain largely unexplored. There has been only limited
research on their diversity and stealth in analog designs, where a Trojan is
activated only during a narrow input voltage range. Effective defense
techniques require a clear understanding of the attack vectors; however, the
lack of diverse analog Trojan instances limits robust advances in detection
strategies. To address this gap, we present LATENT, the first large language
model (LLM)-driven framework for crafting stealthy, circuit-specific analog
Trojans. LATENT incorporates LLM as an autonomous agent to intelligently insert
and refine Trojan components within analog designs based on iterative feedback
from a detection model. This feedback loop ensures that the inserted Trojans
remain stealthy while successfully evading detection. Experimental results
demonstrate that our generated Trojan designs exhibit an average
Trojan-activation range of 15.74%, ensuring they remain inactive under most
operating voltages, while causing a significant performance degradation of
11.3% upon activation.


http://arxiv.org/abs/2505.06335
Remote Rowhammer Attack using Adversarial Observations on Federated Learning Clients. (2%)
Jinsheng Yuan; Yuhang Hao; Weisi Guo; Yun Wu; Chongyan Gu
  Federated Learning (FL) has the potential for simultaneous global learning
amongst a large number of parallel agents, enabling emerging AI such as LLMs to
be trained across demographically diverse data. Central to this being efficient
is the ability for FL to perform sparse gradient updates and remote direct
memory access at the central server. Most of the research in FL security
focuses on protecting data privacy at the edge client or in the communication
channels between the client and server. Client-facing attacks on the server are
less well investigated as the assumption is that a large collective of clients
offer resilience.
  Here, we show that by attacking certain clients that lead to a high frequency
repetitive memory update in the server, we can remote initiate a rowhammer
attack on the server memory. For the first time, we do not need backdoor access
to the server, and a reinforcement learning (RL) attacker can learn how to
maximize server repetitive memory updates by manipulating the client's sensor
observation. The consequence of the remote rowhammer attack is that we are able
to achieve bit flips, which can corrupt the server memory. We demonstrate the
feasibility of our attack using a large-scale FL automatic speech recognition
(ASR) systems with sparse updates, our adversarial attacking agent can achieve
around 70\% repeated update rate (RUR) in the targeted server model,
effectively inducing bit flips on server DRAM. The security implications are
that can cause disruptions to learning or may inadvertently cause elevated
privilege. This paves the way for further research on practical mitigation
strategies in FL and hardware design.


http://arxiv.org/abs/2505.06454
Sponge Attacks on Sensing AI: Energy-Latency Vulnerabilities and Defense via Model Pruning. (2%)
Syed Mhamudul Hasan; Hussein Zangoti; Iraklis Anagnostopoulos; Abdur R. Shahid
  Recent studies have shown that sponge attacks can significantly increase the
energy consumption and inference latency of deep neural networks (DNNs).
However, prior work has focused primarily on computer vision and natural
language processing tasks, overlooking the growing use of lightweight AI models
in sensing-based applications on resource-constrained devices, such as those in
Internet of Things (IoT) environments. These attacks pose serious threats of
energy depletion and latency degradation in systems where limited battery
capacity and real-time responsiveness are critical for reliable operation. This
paper makes two key contributions. First, we present the first systematic
exploration of energy-latency sponge attacks targeting sensing-based AI models.
Using wearable sensing-based AI as a case study, we demonstrate that sponge
attacks can substantially degrade performance by increasing energy consumption,
leading to faster battery drain, and by prolonging inference latency. Second,
to mitigate such attacks, we investigate model pruning, a widely adopted
compression technique for resource-constrained AI, as a potential defense. Our
experiments show that pruning-induced sparsity significantly improves model
resilience against sponge poisoning. We also quantify the trade-offs between
model efficiency and attack resilience, offering insights into the security
implications of model compression in sensing-based AI systems deployed in IoT
environments.


http://arxiv.org/abs/2505.07856
Unpacking Robustness in Inflectional Languages: Adversarial Evaluation and Mechanistic Insights. (99%)
Paweł Walkowiak; Marek Klonowski; Marcin Oleksy; Arkadiusz Janz
  Various techniques are used in the generation of adversarial examples,
including methods such as TextBugger which introduce minor, hardly visible
perturbations to words leading to changes in model behaviour. Another class of
techniques involves substituting words with their synonyms in a way that
preserves the text's meaning but alters its predicted class, with TextFooler
being a prominent example of such attacks. Most adversarial example generation
methods are developed and evaluated primarily on non-inflectional languages,
typically English. In this work, we evaluate and explain how adversarial
attacks perform in inflectional languages. To explain the impact of inflection
on model behaviour and its robustness under attack, we designed a novel
protocol inspired by mechanistic interpretability, based on Edge Attribution
Patching (EAP) method. The proposed evaluation protocol relies on parallel
task-specific corpora that include both inflected and syncretic variants of
texts in two languages -- Polish and English. To analyse the models and explain
the relationship between inflection and adversarial robustness, we create a new
benchmark based on task-oriented dataset MultiEmo, enabling the identification
of mechanistic inflection-related elements of circuits within the model and
analyse their behaviour under attack.


http://arxiv.org/abs/2505.05528
X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP. (98%)
Hanxun Huang; Sarah Erfani; Yige Li; Xingjun Ma; James Bailey
  As Contrastive Language-Image Pre-training (CLIP) models are increasingly
adopted for diverse downstream tasks and integrated into large vision-language
models (VLMs), their susceptibility to adversarial perturbations has emerged as
a critical concern. In this work, we introduce \textbf{X-Transfer}, a novel
attack method that exposes a universal adversarial vulnerability in CLIP.
X-Transfer generates a Universal Adversarial Perturbation (UAP) capable of
deceiving various CLIP encoders and downstream VLMs across different samples,
tasks, and domains. We refer to this property as \textbf{super
transferability}--a single perturbation achieving cross-data, cross-domain,
cross-model, and cross-task adversarial transferability simultaneously. This is
achieved through \textbf{surrogate scaling}, a key innovation of our approach.
Unlike existing methods that rely on fixed surrogate models, which are
computationally intensive to scale, X-Transfer employs an efficient surrogate
scaling strategy that dynamically selects a small subset of suitable surrogates
from a large search space. Extensive evaluations demonstrate that X-Transfer
significantly outperforms previous state-of-the-art UAP methods, establishing a
new benchmark for adversarial transferability across CLIP models. The code is
publicly available in our
\href{https://github.com/HanxunH/XTransferBench}{GitHub repository}.


http://arxiv.org/abs/2505.05665
Adaptive Stress Testing Black-Box LLM Planners. (12%)
Neeloy Chakraborty; John Pohovey; Melkior Ornik; Katherine Driggs-Campbell
  Large language models (LLMs) have recently demonstrated success in
generalizing across decision-making tasks including planning, control and
prediction, but their tendency to hallucinate unsafe and undesired outputs
poses risks. We argue that detecting such failures is necessary, especially in
safety-critical scenarios. Existing black-box methods often detect
hallucinations by identifying inconsistencies across multiple samples. Many of
these approaches typically introduce prompt perturbations like randomizing
detail order or generating adversarial inputs, with the intuition that a
confident model should produce stable outputs. We first perform a manual case
study showing that other forms of perturbations (e.g., adding noise, removing
sensor details) cause LLMs to hallucinate in a driving environment. We then
propose a novel method for efficiently searching the space of prompt
perturbations using Adaptive Stress Testing (AST) with Monte-Carlo Tree Search
(MCTS). Our AST formulation enables discovery of scenarios and prompts that
cause language models to act with high uncertainty. By generating MCTS prompt
perturbation trees across diverse scenarios, we show that offline analyses can
be used at runtime to automatically generate prompts that influence model
uncertainty, and to inform real-time trust assessments of an LLM.


http://arxiv.org/abs/2505.05091
DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions. (10%)
Shashank Agnihotri; Amaan Ansari; Annika Dackermann; Fabian Rösch; Margret Keuper
  Deep learning (DL) has surpassed human performance on standard benchmarks,
driving its widespread adoption in computer vision tasks. One such task is
disparity estimation, estimating the disparity between matching pixels in
stereo image pairs, which is crucial for safety-critical applications like
medical surgeries and autonomous navigation. However, DL-based disparity
estimation methods are highly susceptible to distribution shifts and
adversarial attacks, raising concerns about their reliability and
generalization. Despite these concerns, a standardized benchmark for evaluating
the robustness of disparity estimation methods remains absent, hindering
progress in the field.
  To address this gap, we introduce DispBench, a comprehensive benchmarking
tool for systematically assessing the reliability of disparity estimation
methods. DispBench evaluates robustness against synthetic image corruptions
such as adversarial attacks and out-of-distribution shifts caused by 2D Common
Corruptions across multiple datasets and diverse corruption scenarios. We
conduct the most extensive performance and robustness analysis of disparity
estimation methods to date, uncovering key correlations between accuracy,
reliability, and generalization. Open-source code for DispBench:
https://github.com/shashankskagnihotri/benchmarking_robustness/tree/disparity_estimation/final/disparity_estimation


http://arxiv.org/abs/2505.05235
Advancing Neural Network Verification through Hierarchical Safety Abstract Interpretation. (9%)
Luca Marzari; Isabella Mastroeni; Alessandro Farinelli
  Traditional methods for formal verification (FV) of deep neural networks
(DNNs) are constrained by a binary encoding of safety properties, where a model
is classified as either safe or unsafe (robust or not robust). This binary
encoding fails to capture the nuanced safety levels within a model, often
resulting in either overly restrictive or too permissive requirements. In this
paper, we introduce a novel problem formulation called Abstract
DNN-Verification, which verifies a hierarchical structure of unsafe outputs,
providing a more granular analysis of the safety aspect for a given DNN.
Crucially, by leveraging abstract interpretation and reasoning about output
reachable sets, our approach enables assessing multiple safety levels during
the FV process, requiring the same (in the worst case) or even potentially less
computational effort than the traditional binary verification approach.
Specifically, we demonstrate how this formulation allows rank adversarial
inputs according to their abstract safety level violation, offering a more
detailed evaluation of the model's safety and robustness. Our contributions
include a theoretical exploration of the relationship between our novel
abstract safety formulation and existing approaches that employ abstract
interpretation for robustness verification, complexity analysis of the novel
problem introduced, and an empirical evaluation considering both a complex deep
reinforcement learning task (based on Habitat 3.0) and standard
DNN-Verification benchmarks.


http://arxiv.org/abs/2505.05619
LiteLMGuard: Seamless and Lightweight On-Device Prompt Filtering for Safeguarding Small Language Models against Quantization-induced Risks and Vulnerabilities. (8%)
Kalyan Nakka; Jimmy Dani; Ausmit Mondal; Nitesh Saxena
  The growing adoption of Large Language Models (LLMs) has influenced the
development of their lighter counterparts-Small Language Models (SLMs)-to
enable on-device deployment across smartphones and edge devices. These SLMs
offer enhanced privacy, reduced latency, server-free functionality, and
improved user experience. However, due to resource constraints of on-device
environment, SLMs undergo size optimization through compression techniques like
quantization, which can inadvertently introduce fairness, ethical and privacy
risks. Critically, quantized SLMs may respond to harmful queries directly,
without requiring adversarial manipulation, raising significant safety and
trust concerns.
  To address this, we propose LiteLMGuard (LLMG), an on-device prompt guard
that provides real-time, prompt-level defense for quantized SLMs. Additionally,
our prompt guard is designed to be model-agnostic such that it can be
seamlessly integrated with any SLM, operating independently of underlying
architectures. Our LLMG formalizes prompt filtering as a deep learning
(DL)-based prompt answerability classification task, leveraging semantic
understanding to determine whether a query should be answered by any SLM. Using
our curated dataset, Answerable-or-Not, we trained and fine-tuned several DL
models and selected ELECTRA as the candidate, with 97.75% answerability
classification accuracy.
  Our safety effectiveness evaluations demonstrate that LLMG defends against
over 87% of harmful prompts, including both direct instruction and jailbreak
attack strategies. We further showcase its ability to mitigate the Open
Knowledge Attacks, where compromised SLMs provide unsafe responses without
adversarial prompting. In terms of prompt filtering effectiveness, LLMG
achieves near state-of-the-art filtering accuracy of 94%, with an average
latency of 135 ms, incurring negligible overhead for users.


http://arxiv.org/abs/2505.06311
Defending against Indirect Prompt Injection by Instruction Detection. (2%)
Tongyu Wen; Chenglong Wang; Xiyuan Yang; Haoyu Tang; Yueqi Xie; Lingjuan Lyu; Zhicheng Dou; Fangzhao Wu
  The integration of Large Language Models (LLMs) with external sources is
becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a
prominent example. However, this integration introduces vulnerabilities of
Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in
external data can manipulate LLMs into executing unintended or harmful actions.
We recognize that the success of IPI attacks fundamentally relies in the
presence of instructions embedded within external content, which can alter the
behavioral state of LLMs. Can effectively detecting such state changes help us
defend against IPI attacks? In this paper, we propose a novel approach that
takes external data as input and leverages the behavioral state of LLMs during
both forward and backward propagation to detect potential IPI attacks.
Specifically, we demonstrate that the hidden states and gradients from
intermediate layers provide highly discriminative features for instruction
detection. By effectively combining these features, our approach achieves a
detection accuracy of 99.60\% in the in-domain setting and 96.90\% in the
out-of-domain setting, while reducing the attack success rate to just 0.12\% on
the BIPIA benchmark.


http://arxiv.org/abs/2505.05183
PaniCar: Securing the Perception of Advanced Driving Assistance Systems Against Emergency Vehicle Lighting. (2%)
Elad Feldman; Jacob Shams; Dudi Biton; Alfred Chen; Shaoyuan Xie; Satoru Koda; Yisroel Mirsky; Asaf Shabtai; Yuval Elovici; Ben Nassi
  The safety of autonomous cars has come under scrutiny in recent years,
especially after 16 documented incidents involving Teslas (with autopilot
engaged) crashing into parked emergency vehicles (police cars, ambulances, and
firetrucks). While previous studies have revealed that strong light sources
often introduce flare artifacts in the captured image, which degrade the image
quality, the impact of flare on object detection performance remains unclear.
In this research, we unveil PaniCar, a digital phenomenon that causes an object
detector's confidence score to fluctuate below detection thresholds when
exposed to activated emergency vehicle lighting. This vulnerability poses a
significant safety risk, and can cause autonomous vehicles to fail to detect
objects near emergency vehicles. In addition, this vulnerability could be
exploited by adversaries to compromise the security of advanced driving
assistance systems (ADASs). We assess seven commercial ADASs (Tesla Model 3,
"manufacturer C", HP, Pelsee, AZDOME, Imagebon, Rexing), four object detectors
(YOLO, SSD, RetinaNet, Faster R-CNN), and 14 patterns of emergency vehicle
lighting to understand the influence of various technical and environmental
factors. We also evaluate four SOTA flare removal methods and show that their
performance and latency are insufficient for real-time driving constraints. To
mitigate this risk, we propose Caracetamol, a robust framework designed to
enhance the resilience of object detectors against the effects of activated
emergency vehicle lighting. Our evaluation shows that on YOLOv3 and Faster
RCNN, Caracetamol improves the models' average confidence of car detection by
0.20, the lower confidence bound by 0.33, and reduces the fluctuation range by
0.33. In addition, Caracetamol is capable of processing frames at a rate of
between 30-50 FPS, enabling real-time ADAS car detection.


http://arxiv.org/abs/2505.05190
Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks. (2%)
Yixin Cheng; Hongcheng Guo; Yangming Li; Leonid Sigal
  Text watermarking aims to subtly embed statistical signals into text by
controlling the Large Language Model (LLM)'s sampling process, enabling
watermark detectors to verify that the output was generated by the specified
model. The robustness of these watermarking algorithms has become a key factor
in evaluating their effectiveness. Current text watermarking algorithms embed
watermarks in high-entropy tokens to ensure text quality. In this paper, we
reveal that this seemingly benign design can be exploited by attackers, posing
a significant risk to the robustness of the watermark. We introduce a generic
efficient paraphrasing attack, the Self-Information Rewrite Attack (SIRA),
which leverages the vulnerability by calculating the self-information of each
token to identify potential pattern tokens and perform targeted attack. Our
work exposes a widely prevalent vulnerability in current watermarking
algorithms. The experimental results show SIRA achieves nearly 100% attack
success rates on seven recent watermarking methods with only 0.88 USD per
million tokens cost. Our approach does not require any access to the watermark
algorithms or the watermarked LLM and can seamlessly transfer to any LLM as the
attack model, even mobile-level models. Our findings highlight the urgent need
for more robust watermarking.


http://arxiv.org/abs/2505.05410
Reasoning Models Don't Always Say What They Think. (1%)
Yanda Chen; Joe Benton; Ansh Radhakrishnan; Jonathan Uesato; Carson Denison; John Schulman; Arushi Somani; Peter Hase; Misha Wagner; Fabien Roger; Vlad Mikulik; Samuel R. Bowman; Jan Leike; Jared Kaplan; Ethan Perez
  Chain-of-thought (CoT) offers a potential boon for AI safety as it allows
monitoring a model's CoT to try to understand its intentions and reasoning
processes. However, the effectiveness of such monitoring hinges on CoTs
faithfully representing models' actual reasoning processes. We evaluate CoT
faithfulness of state-of-the-art reasoning models across 6 reasoning hints
presented in the prompts and find: (1) for most settings and models tested,
CoTs reveal their usage of hints in at least 1% of examples where they use the
hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement
learning initially improves faithfulness but plateaus without saturating, and
(3) when reinforcement learning increases how frequently hints are used (reward
hacking), the propensity to verbalize them does not increase, even without
training against a CoT monitor. These results suggest that CoT monitoring is a
promising way of noticing undesired behaviors during training and evaluations,
but that it is not sufficient to rule them out. They also suggest that in
settings like ours where CoT reasoning is not necessary, test-time monitoring
of CoTs is unlikely to reliably catch rare and catastrophic unexpected
behaviors.


http://arxiv.org/abs/2505.06299
Input-Specific and Universal Adversarial Attack Generation for Spiking Neural Networks in the Spiking Domain. (99%)
Spyridon Raptis; Haralampos-G. Stratigopoulos
  As Spiking Neural Networks (SNNs) gain traction across various applications,
understanding their security vulnerabilities becomes increasingly important. In
this work, we focus on the adversarial attacks, which is perhaps the most
concerning threat. An adversarial attack aims at finding a subtle input
perturbation to fool the network's decision-making. We propose two novel
adversarial attack algorithms for SNNs: an input-specific attack that crafts
adversarial samples from specific dataset inputs and a universal attack that
generates a reusable patch capable of inducing misclassification across most
inputs, thus offering practical feasibility for real-time deployment. The
algorithms are gradient-based operating in the spiking domain proving to be
effective across different evaluation metrics, such as adversarial accuracy,
stealthiness, and generation time. Experimental results on two widely used
neuromorphic vision datasets, NMNIST and IBM DVS Gesture, show that our
proposed attacks surpass in all metrics all existing state-of-the-art methods.
Additionally, we present the first demonstration of adversarial attack
generation in the sound domain using the SHD dataset.


http://arxiv.org/abs/2505.04578
Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization. (31%)
Wenjun Cao
  Reinforcement learning (RL) fine-tuning transforms large language models
while creating a vulnerability we experimentally verify: Our experiment shows
that malicious RL fine-tuning dismantles safety guardrails with remarkable
efficiency, requiring only 50 steps and minimal adversarial prompts, with
harmful escalating from 0-2 to 7-9. This attack vector particularly threatens
open-source models with parameter-level access. Existing defenses targeting
supervised fine-tuning prove ineffective against RL's dynamic feedback
mechanisms. We introduce Reward Neutralization, the first defense framework
specifically designed against RL fine-tuning attacks, establishing concise
rejection patterns that render malicious reward signals ineffective. Our
approach trains models to produce minimal-information rejections that attackers
cannot exploit, systematically neutralizing attempts to optimize toward harmful
outputs. Experiments validate that our approach maintains low harmful scores
(no greater than 2) after 200 attack steps, while standard models rapidly
deteriorate. This work provides the first constructive proof that robust
defense against increasingly accessible RL attacks is achievable, addressing a
critical security gap for open-weight models.


http://arxiv.org/abs/2505.04116
RFNNS: Robust Fixed Neural Network Steganography with Popular Deep Generative Models. (13%)
Yu Cheng; Jiuan Zhou; Jiawei Chen; Zhaoxia Yin; Xinpeng Zhang
  Image steganography is a technique that conceals secret information in a
cover image to achieve covert communication. Recent research has demonstrated
that Fixed Neural Network Steganography (FNNS) exhibits significant practical
advantages, as it enables stable and efficient steganographic embedding and
extraction without requiring neural network training. However, the stego image
generated by existing FNNS methods suffers from considerable distortion and
exhibits poor robustness, severely reducing the security and practicality of
steganography. To address the aforementioned issues, we propose a Robust Fixed
Neural Network Steganography (RFNNS). In RFNNS, we introduce a texture-aware
localization technique to add perturbations carrying secret image information
to complex texture areas that are less perceptible to the human eye, thereby
ensuring the quality of the stego image. To enhance robustness, a robust
steganographic perturbation generation (RSPG) strategy is designed, which
enables slight perturbations to be accurately decoded even after common image
attacks. Subsequently, the generated robust perturbations are combined with the
AI-generated cover image to produce the stego image. The receiver only needs to
share the secret key and employ the same decoding network structure to
accurately extract the secret image from the attacked stego image. Experimental
results demonstrate that RFNNS achieves enhanced performance in terms of
security, including imperceptibility and anti-steganalysis performance.
Furthermore, RFNNS demonstrates superior robustness against common image
attacks, such as JPEG compression, Gaussian noise, and contrast adjustment,
across diverse embedding capacities, outperforming existing SOTA FNNS methods.


http://arxiv.org/abs/2505.06284
DMRL: Data- and Model-aware Reward Learning for Data Extraction. (1%)
Zhiqiang Wang; Ruoxi Cheng
  Large language models (LLMs) are inherently vulnerable to unintended privacy
breaches. Consequently, systematic red-teaming research is essential for
developing robust defense mechanisms. However, current data extraction methods
suffer from several limitations: (1) rely on dataset duplicates (addressable
via deduplication), (2) depend on prompt engineering (now countered by
detection and defense), and (3) rely on random-search adversarial generation.
To address these challenges, we propose DMRL, a Data- and Model-aware Reward
Learning approach for data extraction. This technique leverages inverse
reinforcement learning to extract sensitive data from LLMs. Our method consists
of two main components: (1) constructing an introspective reasoning dataset
that captures leakage mindsets to guide model behavior, and (2) training reward
models with Group Relative Policy Optimization (GRPO), dynamically tuning
optimization based on task difficulty at both the data and model levels.
Comprehensive experiments across various LLMs demonstrate that DMRL outperforms
all baseline methods in data extraction performance.


http://arxiv.org/abs/2505.04896
Memory Under Siege: A Comprehensive Survey of Side-Channel Attacks on Memory. (1%)
MD Mahady Hassan; Shanto Roy; Reza Rahaeimehr
  Side-channel attacks on memory (SCAM) exploit unintended data leaks from
memory subsystems to infer sensitive information, posing significant threats to
system security. These attacks exploit vulnerabilities in memory access
patterns, cache behaviors, and other microarchitectural features to bypass
traditional security measures. The purpose of this research is to examine SCAM,
classify various attack techniques, and evaluate existing defense mechanisms.
It guides researchers and industry professionals in improving memory security
and mitigating emerging threats. We begin by identifying the major
vulnerabilities in the memory system that are frequently exploited in SCAM,
such as cache timing, speculative execution, \textit{Rowhammer}, and other
sophisticated approaches. Next, we outline a comprehensive taxonomy that
systematically classifies these attacks based on their types, target systems,
attack vectors, and adversarial capabilities required to execute them. In
addition, we review the current landscape of mitigation strategies, emphasizing
their strengths and limitations. This work aims to provide a comprehensive
overview of memory-based side-channel attacks with the goal of providing
significant insights for researchers and practitioners to better understand,
detect, and mitigate SCAM risks.


http://arxiv.org/abs/2505.03435
Robustness in AI-Generated Detection: Enhancing Resistance to Adversarial Attacks. (99%)
Sun Haoxuan; Hong Yan; Zhan Jiahui; Chen Haoxing; Lan Jun; Zhu Huijia; Wang Weiqiang; Zhang Liqing; Zhang Jianfu
  The rapid advancement of generative image technology has introduced
significant security concerns, particularly in the domain of face generation
detection. This paper investigates the vulnerabilities of current AI-generated
face detection systems. Our study reveals that while existing detection methods
often achieve high accuracy under standard conditions, they exhibit limited
robustness against adversarial attacks. To address these challenges, we propose
an approach that integrates adversarial training to mitigate the impact of
adversarial examples. Furthermore, we utilize diffusion inversion and
reconstruction to further enhance detection robustness. Experimental results
demonstrate that minor adversarial perturbations can easily bypass existing
detection systems, but our method significantly improves the robustness of
these systems. Additionally, we provide an in-depth analysis of adversarial and
benign examples, offering insights into the intrinsic characteristics of
AI-generated content. All associated code will be made publicly available in a
dedicated repository to facilitate further research and verification.


http://arxiv.org/abs/2505.03383
Attention-aggregated Attack for Boosting the Transferability of Facial Adversarial Examples. (99%)
Jian-Wei Li; Wen-Ze Shao
  Adversarial examples have revealed the vulnerability of deep learning models
and raised serious concerns about information security. The transfer-based
attack is a hot topic in black-box attacks that are practical to real-world
scenarios where the training datasets, parameters, and structure of the target
model are unknown to the attacker. However, few methods consider the
particularity of class-specific deep models for fine-grained vision tasks, such
as face recognition (FR), giving rise to unsatisfactory attacking performance.
In this work, we first investigate what in a face exactly contributes to the
embedding learning of FR models and find that both decisive and auxiliary
facial features are specific to each FR model, which is quite different from
the biological mechanism of human visual system. Accordingly we then propose a
novel attack method named Attention-aggregated Attack (AAA) to enhance the
transferability of adversarial examples against FR, which is inspired by the
attention divergence and aims to destroy the facial features that are critical
for the decision-making of other FR models by imitating their attentions on the
clean face images. Extensive experiments conducted on various FR models
validate the superiority and robust effectiveness of the proposed method over
existing methods.


http://arxiv.org/abs/2505.04662
Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders. (95%)
Yuqiu Liu; Huanqian Yan; Xiaopei Zhu; Xiaolin Hu; Liang Tang; Hang Su; Chen Lv
  Recently we have witnessed progress in hiding road vehicles against object
detectors through adversarial camouflage in the digital world. The extension of
this technique to the physical world is crucial for testing the robustness of
autonomous driving systems. However, existing methods do not show good
performances when applied to the physical world. This is partly due to
insufficient photorealism in training examples, and lack of proper physical
realization methods for camouflage. To generate a robust adversarial camouflage
suitable for real vehicles, we propose a novel method called PAV-Camou. We
propose to adjust the mapping from the coordinates in the 2D map to those of
corresponding 3D model. This process is critical for mitigating texture
distortion and ensuring the camouflage's effectiveness when applied in the real
world. Then we combine two renderers with different characteristics to obtain
adversarial examples that are photorealistic that closely mimic real-world
lighting and texture properties. The method ensures that the generated textures
remain effective under diverse environmental conditions. Our adversarial
camouflage can be optimized and printed in the form of 2D patterns, allowing
for direct application on real vehicles. Extensive experiments demonstrated
that our proposed method achieved good performance in both the digital world
and the physical world.


http://arxiv.org/abs/2505.03519
Uncovering the Limitations of Model Inversion Evaluation -- Benchmarks and Connection to Type-I Adversarial Attacks. (92%)
Sy-Tuyen Ho; Koh Jun Hao; Ngoc-Bao Nguyen; Alexander Binder; Ngai-Man Cheung
  Model Inversion (MI) attacks aim to reconstruct information of private
training data by exploiting access to machine learning models. The most common
evaluation framework for MI attacks/defenses relies on an evaluation model that
has been utilized to assess progress across almost all MI attacks and defenses
proposed in recent years. In this paper, for the first time, we present an
in-depth study of MI evaluation. Firstly, we construct the first comprehensive
human-annotated dataset of MI attack samples, based on 28 setups of different
MI attacks, defenses, private and public datasets. Secondly, using our dataset,
we examine the accuracy of the MI evaluation framework and reveal that it
suffers from a significant number of false positives. These findings raise
questions about the previously reported success rates of SOTA MI attacks.
Thirdly, we analyze the causes of these false positives, design controlled
experiments, and discover the surprising effect of Type I adversarial features
on MI evaluation, as well as adversarial transferability, highlighting a
relationship between two previously distinct research areas. Our findings
suggest that the performance of SOTA MI attacks has been overestimated, with
the actual privacy leakage being significantly less than previously reported.
In conclusion, we highlight critical limitations in the widely used MI
evaluation framework and present our methods to mitigate false positive rates.
We remark that prior research has shown that Type I adversarial attacks are
very challenging, with no existing solution. Therefore, we urge to consider
human evaluation as a primary MI evaluation framework rather than merely a
supplement as in previous MI research. We also encourage further work on
developing more robust and reliable automatic evaluation frameworks.


http://arxiv.org/abs/2505.03863
Data-Driven Falsification of Cyber-Physical Systems. (88%)
Atanu Kundu; Sauvik Gon; Rajarshi Ray
  Cyber-Physical Systems (CPS) are abundant in safety-critical domains such as
healthcare, avionics, and autonomous vehicles. Formal verification of their
operational safety is, therefore, of utmost importance. In this paper, we
address the falsification problem, where the focus is on searching for an
unsafe execution in the system instead of proving their absence. The
contribution of this paper is a framework that (a) connects the falsification
of CPS with the falsification of deep neural networks (DNNs) and (b) leverages
the inherent interpretability of Decision Trees for faster falsification of
CPS. This is achieved by: (1) building a surrogate model of the CPS under test,
either as a DNN model or a Decision Tree, (2) application of various DNN
falsification tools to falsify CPS, and (3) a novel falsification algorithm
guided by the explanations of safety violations of the CPS model extracted from
its Decision Tree surrogate. The proposed framework has the potential to
exploit a repertoire of \emph{adversarial attack} algorithms designed to
falsify robustness properties of DNNs, as well as state-of-the-art
falsification algorithms for DNNs. Although the presented methodology is
applicable to systems that can be executed/simulated in general, we demonstrate
its effectiveness, particularly in CPS. We show that our framework, implemented
as a tool \textsc{FlexiFal}, can detect hard-to-find counterexamples in CPS
that have linear and non-linear dynamics. Decision tree-guided falsification
shows promising results in efficiently finding multiple counterexamples in the
ARCH-COMP 2024 falsification benchmarks~\cite{khandait2024arch}.


http://arxiv.org/abs/2505.03424
Framework GNN-AID: Graph Neural Network Analysis Interpretation and Defense. (81%)
Kirill Lukyanov; Mikhail Drobyshevskiy; Georgii Sazonov; Mikhail Soloviov; Ilya Makarov
  The growing need for Trusted AI (TAI) highlights the importance of
interpretability and robustness in machine learning models. However, many
existing tools overlook graph data and rarely combine these two aspects into a
single solution. Graph Neural Networks (GNNs) have become a popular approach,
achieving top results across various tasks. We introduce GNN-AID (Graph Neural
Network Analysis, Interpretation, and Defense), an open-source framework
designed for graph data to address this gap. Built as a Python library, GNN-AID
supports advanced trust methods and architectural layers, allowing users to
analyze graph datasets and GNN behavior using attacks, defenses, and
interpretability methods.
  GNN-AID is built on PyTorch-Geometric, offering preloaded datasets, models,
and support for any GNNs through customizable interfaces. It also includes a
web interface with tools for graph visualization and no-code features like an
interactive model builder, simplifying the exploration and analysis of GNNs.
The framework also supports MLOps techniques, ensuring reproducibility and
result versioning to track and revisit analyses efficiently.
  GNN-AID is a flexible tool for developers and researchers. It helps
developers create, analyze, and customize graph models, while also providing
access to prebuilt datasets and models for quick experimentation. Researchers
can use the framework to explore advanced topics on the relationship between
interpretability and robustness, test defense strategies, and combine methods
to protect against different types of attacks.
  We also show how defenses against evasion and poisoning attacks can conflict
when applied to graph data, highlighting the complex connections between
defense strategies.
  GNN-AID is available at
\href{https://github.com/ispras/GNN-AID}{github.com/ispras/GNN-AID}


http://arxiv.org/abs/2505.04046
Reliable Disentanglement Multi-view Learning Against View Adversarial Attacks. (67%)
Xuyang Wang; Siyuan Duan; Qizhi Li; Guiduo Duan; Yuan Sun; Dezhong Peng
  Recently, trustworthy multi-view learning has attracted extensive attention
because evidence learning can provide reliable uncertainty estimation to
enhance the credibility of multi-view predictions. Existing trusted multi-view
learning methods implicitly assume that multi-view data is secure. In practice,
however, in safety-sensitive applications such as autonomous driving and
security monitoring, multi-view data often faces threats from adversarial
perturbations, thereby deceiving or disrupting multi-view learning models. This
inevitably leads to the adversarial unreliability problem (AUP) in trusted
multi-view learning. To overcome this tricky problem, we propose a novel
multi-view learning framework, namely Reliable Disentanglement Multi-view
Learning (RDML). Specifically, we first propose evidential disentanglement
learning to decompose each view into clean and adversarial parts under the
guidance of corresponding evidences, which is extracted by a pretrained
evidence extractor. Then, we employ the feature recalibration module to
mitigate the negative impact of adversarial perturbations and extract potential
informative features from them. Finally, to further ignore the irreparable
adversarial interferences, a view-level evidential attention mechanism is
designed. Extensive experiments on multi-view classification tasks with
adversarial attacks show that our RDML outperforms the state-of-the-art
multi-view learning methods by a relatively large margin.


http://arxiv.org/abs/2505.03501
BadLingual: A Novel Lingual-Backdoor Attack against Large Language Models. (38%)
Zihan Wang; Hongwei Li; Rui Zhang; Wenbo Jiang; Kangjie Chen; Tianwei Zhang; Qingchuan Zhao; Guowen Xu
  In this paper, we present a new form of backdoor attack against Large
Language Models (LLMs): lingual-backdoor attacks. The key novelty of
lingual-backdoor attacks is that the language itself serves as the trigger to
hijack the infected LLMs to generate inflammatory speech. They enable the
precise targeting of a specific language-speaking group, exacerbating racial
discrimination by malicious entities. We first implement a baseline
lingual-backdoor attack, which is carried out by poisoning a set of training
data for specific downstream tasks through translation into the trigger
language. However, this baseline attack suffers from poor task generalization
and is impractical in real-world settings. To address this challenge, we design
BadLingual, a novel task-agnostic lingual-backdoor, capable of triggering any
downstream tasks within the chat LLMs, regardless of the specific questions of
these tasks. We design a new approach using PPL-constrained Greedy Coordinate
Gradient-based Search (PGCG) based adversarial training to expand the decision
boundary of lingual-backdoor, thereby enhancing the generalization ability of
lingual-backdoor across various tasks. We perform extensive experiments to
validate the effectiveness of our proposed attacks. Specifically, the baseline
attack achieves an ASR of over 90% on the specified tasks. However, its ASR
reaches only 37.61% across six tasks in the task-agnostic scenario. In
contrast, BadLingual brings up to 37.35% improvement over the baseline. Our
study sheds light on a new perspective of vulnerabilities in LLMs with
multilingual capabilities and is expected to promote future research on the
potential defenses to enhance the LLMs' robustness


http://arxiv.org/abs/2505.03208
A Chaos Driven Metric for Backdoor Attack Detection. (38%)
Hema Karnam School of Conflict and Security Studies, National Institute of Advanced Studies, Indian Institute of Science Campus, Bengaluru Surendrababu; Nithin Complex Systems Programme, National Institute of Advanced Studies, Indian Institute of Science Campus, Bengaluru Nagaraj
  The advancement and adoption of Artificial Intelligence (AI) models across
diverse domains have transformed the way we interact with technology. However,
it is essential to recognize that while AI models have introduced remarkable
advancements, they also present inherent challenges such as their vulnerability
to adversarial attacks. The current work proposes a novel defense mechanism
against one of the most significant attack vectors of AI models - the backdoor
attack via data poisoning of training datasets. In this defense technique, an
integrated approach that combines chaos theory with manifold learning is
proposed. A novel metric - Precision Matrix Dependency Score (PDS) that is
based on the conditional variance of Neurochaos features is formulated. The PDS
metric has been successfully evaluated to distinguish poisoned samples from
non-poisoned samples across diverse datasets.


http://arxiv.org/abs/2505.03455
Mitigating Backdoor Triggered and Targeted Data Poisoning Attacks in Voice Authentication Systems. (16%)
Alireza Mohammadi; Keshav Sood; Dhananjay Thiruvady; Asef Nazari
  Voice authentication systems remain susceptible to two major threats:
backdoor triggered attacks and targeted data poisoning attacks. This dual
vulnerability is critical because conventional solutions typically address each
threat type separately, leaving systems exposed to adversaries who can exploit
both attacks simultaneously. We propose a unified defense framework that
effectively addresses both BTA and TDPA. Our framework integrates a frequency
focused detection mechanism that flags covert pitch boosting and sound masking
backdoor attacks in near real time, followed by a convolutional neural network
that addresses TDPA. This dual layered defense approach utilizes
multidimensional acoustic features to isolate anomalous signals without
requiring costly model retraining. In particular, our PBSM detection mechanism
can seamlessly integrate into existing voice authentication pipelines and scale
effectively for large scale deployments. Experimental results on benchmark
datasets and their compression with the state of the art algorithm demonstrate
that our PBSM detection mechanism outperforms the state of the art. Our
framework reduces attack success rates to as low as five to fifteen percent
while maintaining a recall rate of up to ninety five percent in recognizing
TDPA.


http://arxiv.org/abs/2505.04087
SEVA: Leveraging Single-Step Ensemble of Vicinal Augmentations for Test-Time Adaptation. (1%)
Zixuan Hu; Yichun Hu; Ling-Yu Duan
  Test-Time adaptation (TTA) aims to enhance model robustness against
distribution shifts through rapid model adaptation during inference. While
existing TTA methods often rely on entropy-based unsupervised training and
achieve promising results, the common practice of a single round of entropy
training is typically unable to adequately utilize reliable samples, hindering
adaptation efficiency. In this paper, we discover augmentation strategies can
effectively unleash the potential of reliable samples, but the rapidly growing
computational cost impedes their real-time application. To address this
limitation, we propose a novel TTA approach named Single-step Ensemble of
Vicinal Augmentations (SEVA), which can take advantage of data augmentations
without increasing the computational burden. Specifically, instead of
explicitly utilizing the augmentation strategy to generate new data, SEVA
develops a theoretical framework to explore the impacts of multiple
augmentations on model adaptation and proposes to optimize an upper bound of
the entropy loss to integrate the effects of multiple rounds of augmentation
training into a single step. Furthermore, we discover and verify that using the
upper bound as the loss is more conducive to the selection mechanism, as it can
effectively filter out harmful samples that confuse the model. Combining these
two key advantages, the proposed efficient loss and a complementary selection
strategy can simultaneously boost the potential of reliable samples and meet
the stringent time requirements of TTA. The comprehensive experiments on
various network architectures across challenging testing scenarios demonstrate
impressive performances and the broad adaptability of SEVA. The code will be
publicly available.


http://arxiv.org/abs/2505.03361
Interpretable Zero-shot Learning with Infinite Class Concepts. (1%)
Zihan Ye; Shreyank N Gowda; Shiming Chen; Yaochu Jin; Kaizhu Huang; Xiaobo Jin
  Zero-shot learning (ZSL) aims to recognize unseen classes by aligning images
with intermediate class semantics, like human-annotated concepts or class
definitions. An emerging alternative leverages Large-scale Language Models
(LLMs) to automatically generate class documents. However, these methods often
face challenges with transparency in the classification process and may suffer
from the notorious hallucination problem in LLMs, resulting in non-visual class
semantics. This paper redefines class semantics in ZSL with a focus on
transferability and discriminability, introducing a novel framework called
Zero-shot Learning with Infinite Class Concepts (InfZSL). Our approach
leverages the powerful capabilities of LLMs to dynamically generate an
unlimited array of phrase-level class concepts. To address the hallucination
challenge, we introduce an entropy-based scoring process that incorporates a
``goodness" concept selection mechanism, ensuring that only the most
transferable and discriminative concepts are selected. Our InfZSL framework not
only demonstrates significant improvements on three popular benchmark datasets
but also generates highly interpretable, image-grounded concepts. Code will be
released upon acceptance.


http://arxiv.org/abs/2505.03559
Event-Triggered GAT-LSTM Framework for Attack Detection in Heating, Ventilation, and Air Conditioning Systems. (1%)
Zhenan Feng; Ehsan Nekouei
  Heating, Ventilation, and Air Conditioning (HVAC) systems are essential for
maintaining indoor environmental quality, but their interconnected nature and
reliance on sensor networks make them vulnerable to cyber-physical attacks.
Such attacks can interrupt system operations and risk leaking sensitive
personal information through measurement data. In this paper, we propose a
novel attack detection framework for HVAC systems, integrating an
Event-Triggering Unit (ETU) for local monitoring and a cloud-based
classification system using the Graph Attention Network (GAT) and the Long
Short-Term Memory (LSTM) network. The ETU performs a binary classification to
identify potential anomalies and selectively triggers encrypted data
transmission to the cloud, significantly reducing communication cost. The
cloud-side GAT module models the spatial relationships among HVAC components,
while the LSTM module captures temporal dependencies across encrypted state
sequences to classify the attack type. Our approach is evaluated on datasets
that simulate diverse attack scenarios. Compared to GAT-only (94.2% accuracy)
and LSTM-only (91.5%) ablations, our full GAT-LSTM model achieves 98.8% overall
detection accuracy and reduces data transmission to 15%. These results
demonstrate that the proposed framework achieves high detection accuracy while
preserving data privacy by using the spatial-temporal characteristics of HVAC
systems and minimizing transmission costs through event-triggered
communication.


http://arxiv.org/abs/2505.03120
Adversarial Sample Generation for Anomaly Detection in Industrial Control Systems. (99%)
Abdul Mustafa; Muhammad Talha Khan; Muhammad Azmi Umer; Zaki Masood; Chuadhry Mujeeb Ahmed
  Machine learning (ML)-based intrusion detection systems (IDS) are vulnerable
to adversarial attacks. It is crucial for an IDS to learn to recognize
adversarial examples before malicious entities exploit them. In this paper, we
generated adversarial samples using the Jacobian Saliency Map Attack (JSMA). We
validate the generalization and scalability of the adversarial samples to
tackle a broad range of real attacks on Industrial Control Systems (ICS). We
evaluated the impact by assessing multiple attacks generated using the proposed
method. The model trained with adversarial samples detected attacks with 95%
accuracy on real-world attack data not used during training. The study was
conducted using an operational secure water treatment (SWaT) testbed.


http://arxiv.org/abs/2505.02971
Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation. (99%)
Anjila Budathoki; Manish Dhakal
  Adversarial attacks have been fairly explored for computer vision and
vision-language models. However, the avenue of adversarial attack for the
vision language segmentation models (VLSMs) is still under-explored, especially
for medical image analysis.
  Thus, we have investigated the robustness of VLSMs against adversarial
attacks for 2D medical images with different modalities with radiology,
photography, and endoscopy. The main idea of this project was to assess the
robustness of the fine-tuned VLSMs specially in the medical domain setting to
address the high risk scenario.
  First, we have fine-tuned pre-trained VLSMs for medical image segmentation
with adapters.
  Then, we have employed adversarial attacks -- projected gradient descent
(PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to
determine its robustness against adversaries.
  We have reported models' performance decline to analyze the adversaries'
impact.
  The results exhibit significant drops in the DSC and IoU scores after the
introduction of these adversaries. Furthermore, we also explored universal
perturbation but were not able to find for the medical images.
  \footnote{https://github.com/anjilab/secure-private-ai}


http://arxiv.org/abs/2505.02360
Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless $l^p$ Norm Solution for Fast Adversarial Training. (87%)
Fares B. Mehouachi; Saif Eddin Jabari
  Adversarial training is a cornerstone of robust deep learning, but fast
methods like the Fast Gradient Sign Method (FGSM) often suffer from
Catastrophic Overfitting (CO), where models become robust to single-step
attacks but fail against multi-step variants. While existing solutions rely on
noise injection, regularization, or gradient clipping, we propose a novel
solution that purely controls the $l^p$ training norm to mitigate CO.
  Our study is motivated by the empirical observation that CO is more prevalent
under the $l^{\infty}$ norm than the $l^2$ norm. Leveraging this insight, we
develop a framework for generalized $l^p$ attack as a fixed point problem and
craft $l^p$-FGSM attacks to understand the transition mechanics from $l^2$ to
$l^{\infty}$. This leads to our core insight: CO emerges when highly
concentrated gradients where information localizes in few dimensions interact
with aggressive norm constraints. By quantifying gradient concentration through
Participation Ratio and entropy measures, we develop an adaptive $l^p$-FGSM
that automatically tunes the training norm based on gradient information.
Extensive experiments demonstrate that this approach achieves strong robustness
without requiring additional regularization or noise injection, providing a
novel and theoretically-principled pathway to mitigate the CO problem.


http://arxiv.org/abs/2505.02566
Robustness questions the interpretability of graph neural networks: what to do? (83%)
Kirill ISP RAS Research Center for Trusted Artificial Intelligence Ivannikov Institute for System Programming of the Russian Academy of Sciences Moscow Institute of Physics and Technology Lukyanov; Georgii Ivannikov Institute for System Programming of the Russian Academy of Sciences Lomonosov Moscow State University Sazonov; Serafim Yandex School of Data Analysis Boyarsky; Ilya 1 v 5 Makarov
  Graph Neural Networks (GNNs) have become a cornerstone in graph-based data
analysis, with applications in diverse domains such as bioinformatics, social
networks, and recommendation systems. However, the interplay between model
interpretability and robustness remains poorly understood, especially under
adversarial scenarios like poisoning and evasion attacks. This paper presents a
comprehensive benchmark to systematically analyze the impact of various factors
on the interpretability of GNNs, including the influence of
robustness-enhancing defense mechanisms.
  We evaluate six GNN architectures based on GCN, SAGE, GIN, and GAT across
five datasets from two distinct domains, employing four interpretability
metrics: Fidelity, Stability, Consistency, and Sparsity. Our study examines how
defenses against poisoning and evasion attacks, applied before and during model
training, affect interpretability and highlights critical trade-offs between
robustness and interpretability. The framework will be published as open
source.
  The results reveal significant variations in interpretability depending on
the chosen defense methods and model architecture characteristics. By
establishing a standardized benchmark, this work provides a foundation for
developing GNNs that are both robust to adversarial threats and interpretable,
facilitating trust in their deployment in sensitive applications.


http://arxiv.org/abs/2505.03084
Adversarial Attacks in Multimodal Systems: A Practitioner's Survey. (76%)
Shashank Kapoor; Sanjay Surendranath Girija; Lakshit Arora; Dipen Pradhan; Ankit Shetgaonkar; Aman Raj
  The introduction of multimodal models is a huge step forward in Artificial
Intelligence. A single model is trained to understand multiple modalities:
text, image, video, and audio. Open-source multimodal models have made these
breakthroughs more accessible. However, considering the vast landscape of
adversarial attacks across these modalities, these models also inherit
vulnerabilities of all the modalities, and ultimately, the adversarial threat
amplifies. While broad research is available on possible attacks within or
across these modalities, a practitioner-focused view that outlines attack types
remains absent in the multimodal world. As more Machine Learning Practitioners
adopt, fine-tune, and deploy open-source models in real-world applications,
it's crucial that they can view the threat landscape and take the preventive
actions necessary. This paper addresses the gap by surveying adversarial
attacks targeting all four modalities: text, image, video, and audio. This
survey provides a view of the adversarial attack landscape and presents how
multimodal adversarial threats have evolved. To the best of our knowledge, this
survey is the first comprehensive summarization of the threat landscape in the
multimodal world.


http://arxiv.org/abs/2505.02824
Towards Dataset Copyright Evasion Attack against Personalized Text-to-Image Diffusion Models. (56%)
Kuofeng Gao; Yufei Zhu; Yiming Li; Jiawang Bai; Yong Yang; Zhifeng Li; Shu-Tao Xia
  Text-to-image (T2I) diffusion models have rapidly advanced, enabling
high-quality image generation conditioned on textual prompts. However, the
growing trend of fine-tuning pre-trained models for personalization raises
serious concerns about unauthorized dataset usage. To combat this, dataset
ownership verification (DOV) has emerged as a solution, embedding watermarks
into the fine-tuning datasets using backdoor techniques. These watermarks
remain inactive under benign samples but produce owner-specified outputs when
triggered. Despite the promise of DOV for T2I diffusion models, its robustness
against copyright evasion attacks (CEA) remains unexplored. In this paper, we
explore how attackers can bypass these mechanisms through CEA, allowing models
to circumvent watermarks even when trained on watermarked datasets. We propose
the first copyright evasion attack (i.e., CEAT2I) specifically designed to
undermine DOV in T2I diffusion models. Concretely, our CEAT2I comprises three
stages: watermarked sample detection, trigger identification, and efficient
watermark mitigation. A key insight driving our approach is that T2I models
exhibit faster convergence on watermarked samples during the fine-tuning,
evident through intermediate feature deviation. Leveraging this, CEAT2I can
reliably detect the watermarked samples. Then, we iteratively ablate tokens
from the prompts of detected watermarked samples and monitor shifts in
intermediate features to pinpoint the exact trigger tokens. Finally, we adopt a
closed-form concept erasure method to remove the injected watermark. Extensive
experiments show that our CEAT2I effectively evades DOV mechanisms while
preserving model performance.


http://arxiv.org/abs/2505.02490
Bayesian Robust Aggregation for Federated Learning. (3%)
Aleksandr Uppsala University Karakulev; Usama Uppsala University Zafar; Salman Uppsala University Scaleout Systems Toor; Prashant Uppsala University Science for Life Laboratory, Sweden Singh
  Federated Learning enables collaborative training of machine learning models
on decentralized data. This scheme, however, is vulnerable to adversarial
attacks, when some of the clients submit corrupted model updates. In real-world
scenarios, the total number of compromised clients is typically unknown, with
the extent of attacks potentially varying over time. To address these
challenges, we propose an adaptive approach for robust aggregation of model
updates based on Bayesian inference. The mean update is defined by the maximum
of the likelihood marginalized over probabilities of each client to be
`honest'. As a result, the method shares the simplicity of the classical
average estimators (e.g., sample mean or geometric median), being independent
of the number of compromised clients. At the same time, it is as effective
against attacks as methods specifically tailored to Federated Learning, such as
Krum. We compare our approach with other aggregation schemes in federated
setting on three benchmark image classification data sets. The proposed method
consistently achieves state-of-the-art performance across various attack types
with static and varying number of malicious clients.


http://arxiv.org/abs/2505.02713
SoK: Stealing Cars Since Remote Keyless Entry Introduction and How to Defend From It. (3%)
Tommaso Bianchi; Alessandro Brighente; Mauro Conti; Edoardo Pavan
  Remote Keyless Entry (RKE) systems have been the target of thieves since
their introduction in automotive industry. Robberies targeting vehicles and
their remote entry systems are booming again without a significant advancement
from the industrial sector being able to protect against them. Researchers and
attackers continuously play cat and mouse to implement new methodologies to
exploit weaknesses and defense strategies for RKEs. In this fragment, different
attacks and defenses have been discussed in research and industry without
proper bridging. In this paper, we provide a Systematization Of Knowledge (SOK)
on RKE and Passive Keyless Entry and Start (PKES), focusing on their history
and current situation, ranging from legacy systems to modern web-based ones. We
provide insight into vehicle manufacturers' technologies and attacks and
defense mechanisms involving them. To the best of our knowledge, this is the
first comprehensive SOK on RKE systems, and we address specific research
questions to understand the evolution and security status of such systems. By
identifying the weaknesses RKE still faces, we provide future directions for
security researchers and companies to find viable solutions to address old
attacks, such as Relay and RollJam, as well as new ones, like API
vulnerabilities.


http://arxiv.org/abs/2505.02073
Lightweight Defense Against Adversarial Attacks in Time Series Classification. (92%)
Yi Independent Researcher, Australia Han
  As time series classification (TSC) gains prominence, ensuring robust TSC
models against adversarial attacks is crucial. While adversarial defense is
well-studied in Computer Vision (CV), the TSC field has primarily relied on
adversarial training (AT), which is computationally expensive. In this paper,
five data augmentation-based defense methods tailored for time series are
developed, with the most computationally intensive method among them increasing
the computational resources by only 14.07% compared to the original TSC model.
Moreover, the deployment process for these methods is straightforward. By
leveraging these advantages of our methods, we create two combined methods. One
of these methods is an ensemble of all the proposed techniques, which not only
provides better defense performance than PGD-based AT but also enhances the
generalization ability of TSC models. Moreover, the computational resources
required for our ensemble are less than one-third of those required for
PGD-based AT. These methods advance robust TSC in data mining. Furthermore, as
foundation models are increasingly explored for time series feature learning,
our work provides insights into integrating data augmentation-based adversarial
defense with large-scale pre-trained models in future research.


http://arxiv.org/abs/2505.01900
CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation. (93%)
Mazal Bethany; Nishant Vishwamitra; Cho-Yu Jason Chiang; Peyman Najafirad
  Automated evidence-based misinformation detection systems, which evaluate the
veracity of short claims against evidence, lack comprehensive analysis of their
adversarial vulnerabilities. Existing black-box text-based adversarial attacks
are ill-suited for evidence-based misinformation detection systems, as these
attacks primarily focus on token-level substitutions involving gradient or
logit-based optimization strategies, which are incapable of fooling the
multi-component nature of these detection systems. These systems incorporate
both retrieval and claim-evidence comparison modules, which requires attacks to
break the retrieval of evidence and/or the comparison module so that it draws
incorrect inferences. We present CAMOUFLAGE, an iterative, LLM-driven approach
that employs a two-agent system, a Prompt Optimization Agent and an Attacker
Agent, to create adversarial claim rewritings that manipulate evidence
retrieval and mislead claim-evidence comparison, effectively bypassing the
system without altering the meaning of the claim. The Attacker Agent produces
semantically equivalent rewrites that attempt to mislead detectors, while the
Prompt Optimization Agent analyzes failed attack attempts and refines the
prompt of the Attacker to guide subsequent rewrites. This enables larger
structural and stylistic transformations of the text rather than token-level
substitutions, adapting the magnitude of changes based on previous outcomes.
Unlike existing approaches, CAMOUFLAGE optimizes its attack solely based on
binary model decisions to guide its rewriting process, eliminating the need for
classifier logits or extensive querying. We evaluate CAMOUFLAGE on four
systems, including two recent academic systems and two real-world APIs, with an
average attack success rate of 46.92\% while preserving textual coherence and
semantic equivalence to the original claims.


http://arxiv.org/abs/2505.01816
Rogue Cell: Adversarial Attack and Defense in Untrusted O-RAN Setup Exploiting the Traffic Steering xApp. (93%)
Eran Aizikovich; Dudu Mimran; Edita Grolman; Yuval Elovici; Asaf Shabtai
  The Open Radio Access Network (O-RAN) architecture is revolutionizing
cellular networks with its open, multi-vendor design and AI-driven management,
aiming to enhance flexibility and reduce costs. Although it has many
advantages, O-RAN is not threat-free. While previous studies have mainly
examined vulnerabilities arising from O-RAN's intelligent components, this
paper is the first to focus on the security challenges and vulnerabilities
introduced by transitioning from single-operator to multi-operator RAN
architectures. This shift increases the risk of untrusted third-party operators
managing different parts of the network. To explore these vulnerabilities and
their potential mitigation, we developed an open-access testbed environment
that integrates a wireless network simulator with the official O-RAN Software
Community (OSC) RAN intelligent component (RIC) cluster. This environment
enables realistic, live data collection and serves as a platform for
demonstrating APATE (adversarial perturbation against traffic efficiency), an
evasion attack in which a malicious cell manipulates its reported key
performance indicators (KPIs) and deceives the O-RAN traffic steering to gain
unfair allocations of user equipment (UE). To ensure that O-RAN's legitimate
activity continues, we introduce MARRS (monitoring adversarial RAN reports), a
detection framework based on a long-short term memory (LSTM) autoencoder (AE)
that learns contextual features across the network to monitor malicious
telemetry (also demonstrated in our testbed). Our evaluation showed that by
executing APATE, an attacker can obtain a 248.5% greater UE allocation than it
was supposed to in a benign scenario. In addition, the MARRS detection method
was also shown to successfully classify malicious cell activity, achieving
accuracy of 99.2% and an F1 score of 0.978.


http://arxiv.org/abs/2505.01884
Adversarial Robustness of Deep Learning Models for Inland Water Body Segmentation from SAR Images. (93%)
Siddharth Kothari; Srinivasan Murali; Sankalp Kothari; Ujjwal Verma; Jaya Sreevalsan-Nair
  Inland water body segmentation from Synthetic Aperture Radar (SAR) images is
an important task needed for several applications, such as flood mapping. While
SAR sensors capture data in all-weather conditions as high-resolution images,
differentiating water and water-like surfaces from SAR images is not
straightforward. Inland water bodies, such as large river basins, have complex
geometry, which adds to the challenge of segmentation. U-Net is a widely used
deep learning model for land-water segmentation of SAR images. In practice,
manual annotation is often used to generate the corresponding water masks as
ground truth. Manual annotation of the images is prone to label noise owing to
data poisoning attacks, especially due to complex geometry. In this work, we
simulate manual errors in the form of adversarial attacks on the U-Net model
and study the robustness of the model to human errors in annotation. Our
results indicate that U-Net can tolerate a certain level of corruption before
its performance drops significantly. This finding highlights the crucial role
that the quality of manual annotations plays in determining the effectiveness
of the segmentation model. The code and the new dataset, along with adversarial
examples for robust training, are publicly available. (GitHub link -
https://github.com/GVCL/IWSeg-SAR-Poison.git)


http://arxiv.org/abs/2505.01050
Transferable Adversarial Attacks on Black-Box Vision-Language Models. (99%)
Kai Hu; Weichen Yu; Li Zhang; Alexander Robey; Andy Zou; Chengming Xu; Haoqi Hu; Matt Fredrikson
  Vision Large Language Models (VLLMs) are increasingly deployed to offer
advanced capabilities on inputs comprising both text and images. While prior
research has shown that adversarial attacks can transfer from open-source to
proprietary black-box models in text-only and vision-only contexts, the extent
and effectiveness of such vulnerabilities remain underexplored for VLLMs. We
present a comprehensive analysis demonstrating that targeted adversarial
examples are highly transferable to widely-used proprietary VLLMs such as
GPT-4o, Claude, and Gemini. We show that attackers can craft perturbations to
induce specific attacker-chosen interpretations of visual information, such as
misinterpreting hazardous content as safe, overlooking sensitive or restricted
material, or generating detailed incorrect responses aligned with the
attacker's intent. Furthermore, we discover that universal perturbations --
modifications applicable to a wide set of images -- can consistently induce
these misinterpretations across multiple proprietary VLLMs. Our experimental
results on object recognition, visual question answering, and image captioning
show that this vulnerability is common across current state-of-the-art models,
and underscore an urgent need for robust mitigations to ensure the safe and
secure deployment of VLLMs.


http://arxiv.org/abs/2505.01168
Harmonizing Intra-coherence and Inter-divergence in Ensemble Attacks for Adversarial Transferability. (99%)
Zhaoyang Ma; Zhihao Wu; Wang Lu; Xin Gao; Jinghang Yue; Taolin Zhang; Lipo Wang; Youfang Lin; Jing Wang
  The development of model ensemble attacks has significantly improved the
transferability of adversarial examples, but this progress also poses severe
threats to the security of deep neural networks. Existing methods, however,
face two critical challenges: insufficient capture of shared gradient
directions across models and a lack of adaptive weight allocation mechanisms.
To address these issues, we propose a novel method Harmonized Ensemble for
Adversarial Transferability (HEAT), which introduces domain generalization into
adversarial example generation for the first time. HEAT consists of two key
modules: Consensus Gradient Direction Synthesizer, which uses Singular Value
Decomposition to synthesize shared gradient directions; and Dual-Harmony Weight
Orchestrator which dynamically balances intra-domain coherence, stabilizing
gradients within individual models, and inter-domain diversity, enhancing
transferability across models. Experimental results demonstrate that HEAT
significantly outperforms existing methods across various datasets and
settings, offering a new perspective and direction for adversarial attack
research.


http://arxiv.org/abs/2505.01328
Constrained Network Adversarial Attacks: Validity, Robustness, and Transferability. (99%)
Anass Grini; Oumaima Taheri; Btissam El Khamlichi; Amal El Fallah-Seghrouchni
  While machine learning has significantly advanced Network Intrusion Detection
Systems (NIDS), particularly within IoT environments where devices generate
large volumes of data and are increasingly susceptible to cyber threats, these
models remain vulnerable to adversarial attacks. Our research reveals a
critical flaw in existing adversarial attack methodologies: the frequent
violation of domain-specific constraints, such as numerical and categorical
limits, inherent to IoT and network traffic. This leads to up to 80.3% of
adversarial examples being invalid, significantly overstating real-world
vulnerabilities. These invalid examples, though effective in fooling models, do
not represent feasible attacks within practical IoT deployments. Consequently,
relying on these results can mislead resource allocation for defense, inflating
the perceived susceptibility of IoT-enabled NIDS models to adversarial
manipulation. Furthermore, we demonstrate that simpler surrogate models like
Multi-Layer Perceptron (MLP) generate more valid adversarial examples compared
to complex architectures such as CNNs and LSTMs. Using the MLP as a surrogate,
we analyze the transferability of adversarial severity to other ML/DL models
commonly used in IoT contexts. This work underscores the importance of
considering both domain constraints and model architecture when evaluating and
designing robust ML/DL models for security-critical IoT and network
applications.


http://arxiv.org/abs/2505.01315
Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System. (83%)
Sheikh Samit Muhaimin; Spyridon Mastorakis
  The recent growth in the use of Large Language Models has made them
vulnerable to sophisticated adversarial assaults, manipulative prompts, and
encoded malicious inputs. Existing countermeasures frequently necessitate
retraining models, which is computationally costly and impracticable for
deployment. Without the need for retraining or fine-tuning, this study presents
a unique defense paradigm that allows LLMs to recognize, filter, and defend
against adversarial or malicious inputs on their own. There are two main parts
to the suggested framework: (1) A prompt filtering module that uses
sophisticated Natural Language Processing (NLP) techniques, including zero-shot
classification, keyword analysis, and encoded content detection (e.g. base64,
hexadecimal, URL encoding), to detect, decode, and classify harmful inputs; and
(2) A summarization module that processes and summarizes adversarial research
literature to give the LLM context-aware defense knowledge. This approach
strengthens LLMs' resistance to adversarial exploitation by fusing text
extraction, summarization, and harmful prompt analysis. According to
experimental results, this integrated technique has a 98.71% success rate in
identifying harmful patterns, manipulative language structures, and encoded
prompts. By employing a modest amount of adversarial research literature as
context, the methodology also allows the model to react correctly to harmful
inputs with a larger percentage of jailbreak resistance and refusal rate. While
maintaining the quality of LLM responses, the framework dramatically increases
LLM's resistance to hostile misuse, demonstrating its efficacy as a quick and
easy substitute for time-consuming, retraining-based defenses.


http://arxiv.org/abs/2505.01267
Diffusion-based Adversarial Purification from the Perspective of the Frequency Domain. (76%)
Gaozheng Pei; Ke Ma; Yingfei Sun; Qianqian Xu; Qingming Huang
  The diffusion-based adversarial purification methods attempt to drown
adversarial perturbations into a part of isotropic noise through the forward
process, and then recover the clean images through the reverse process. Due to
the lack of distribution information about adversarial perturbations in the
pixel domain, it is often unavoidable to damage normal semantics. We turn to
the frequency domain perspective, decomposing the image into amplitude spectrum
and phase spectrum. We find that for both spectra, the damage caused by
adversarial perturbations tends to increase monotonically with frequency. This
means that we can extract the content and structural information of the
original clean sample from the frequency components that are less damaged.
Meanwhile, theoretical analysis indicates that existing purification methods
indiscriminately damage all frequency components, leading to excessive damage
to the image. Therefore, we propose a purification method that can eliminate
adversarial perturbations while maximizing the preservation of the content and
structure of the original image. Specifically, at each time step during the
reverse process, for the amplitude spectrum, we replace the low-frequency
components of the estimated image's amplitude spectrum with the corresponding
parts of the adversarial image. For the phase spectrum, we project the phase of
the estimated image into a designated range of the adversarial image's phase
spectrum, focusing on the low frequencies. Empirical evidence from extensive
experiments demonstrates that our method significantly outperforms most current
defense methods.


http://arxiv.org/abs/2505.01139
Active Sybil Attack and Efficient Defense Strategy in IPFS DHT. (50%)
V. H. M. Netto; T. Cholez; C. L. Ignat
  The InterPlanetary File System (IPFS) is a decentralized peer-to-peer (P2P)
storage that relies on Kademlia, a Distributed Hash Table (DHT) structure
commonly used in P2P systems for its proved scalability. However, DHTs are
known to be vulnerable to Sybil attacks, in which a single entity controls
multiple malicious nodes. Recent studies have shown that IPFS is affected by a
passive content eclipse attack, leveraging Sybils, in which adversarial nodes
hide received indexed information from other peers, making the content appear
unavailable. Fortunately, the latest mitigation strategy coupling an attack
detection based on statistical tests and a wider publication strategy upon
detection was able to circumvent it.
  In this work, we present a new active attack, with malicious nodes responding
with semantically correct but intentionally false data, exploiting both an
optimized placement of Sybils to stay below the detection threshold and an
early trigger of the content discovery termination in Kubo, the main IPFS
implementation. Our attack achieves to completely eclipse content on the latest
Kubo release. When evaluated against the most recent known mitigation, it
successfully denies access to the target content in approximately 80\% of
lookup attempts.
  To address this vulnerability, we propose a new mitigation called
SR-DHT-Store, which enables efficient, Sybil-resistant content publication
without relying on attack detection but instead on a systematic and precise use
of region-based queries, defined by a dynamically computed XOR distance to the
target ID. SR-DHT-Store can be combined with other defense mechanisms resulting
in a defense strategy that completely mitigates both passive and active Sybil
attacks at a lower overhead, while allowing an incremental deployment.


http://arxiv.org/abs/2505.01012
Quantum Support Vector Regression for Robust Anomaly Detection. (33%)
Kilian Tscharke; Maximilian Wendlinger; Sebastian Issel; Pascal Debus
  Anomaly Detection (AD) is critical in data analysis, particularly within the
domain of IT security. In recent years, Machine Learning (ML) algorithms have
emerged as a powerful tool for AD in large-scale data. In this study, we
explore the potential of quantum ML approaches, specifically quantum kernel
methods, for the application to robust AD. We build upon previous work on
Quantum Support Vector Regression (QSVR) for semisupervised AD by conducting a
comprehensive benchmark on IBM quantum hardware using eleven datasets. Our
results demonstrate that QSVR achieves strong classification performance and
even outperforms the noiseless simulation on two of these datasets. Moreover,
we investigate the influence of - in the NISQ-era inevitable - quantum noise on
the performance of the QSVR. Our findings reveal that the model exhibits
robustness to depolarizing, phase damping, phase flip, and bit flip noise,
while amplitude damping and miscalibration noise prove to be more disruptive.
Finally, we explore the domain of Quantum Adversarial Machine Learning and
demonstrate that QSVR is highly vulnerable to adversarial attacks and that
noise does not improve the adversarial robustness of the model.


http://arxiv.org/abs/2505.01186
Secure Cluster-Based Hierarchical Federated Learning in Vehicular Networks. (15%)
M. Saeid HaghighiFard; Sinem Coleri
  Hierarchical Federated Learning (HFL) has recently emerged as a promising
solution for intelligent decision-making in vehicular networks, helping to
address challenges such as limited communication resources, high vehicle
mobility, and data heterogeneity. However, HFL remains vulnerable to
adversarial and unreliable vehicles, whose misleading updates can significantly
compromise the integrity and convergence of the global model. To address these
challenges, we propose a novel defense framework that integrates dynamic
vehicle selection with robust anomaly detection within a cluster-based HFL
architecture, specifically designed to counter Gaussian noise and gradient
ascent attacks. The framework performs a comprehensive reliability assessment
for each vehicle by evaluating historical accuracy, contribution frequency, and
anomaly records. Anomaly detection combines Z-score and cosine similarity
analyses on model updates to identify both statistical outliers and directional
deviations in model updates. To further refine detection, an adaptive
thresholding mechanism is incorporated into the cosine similarity metric,
dynamically adjusting the threshold based on the historical accuracy of each
vehicle to enforce stricter standards for consistently high-performing
vehicles. In addition, a weighted gradient averaging mechanism is implemented,
which assigns higher weights to gradient updates from more trustworthy
vehicles. To defend against coordinated attacks, a cross-cluster consistency
check is applied to identify collaborative attacks in which multiple
compromised clusters coordinate misleading updates. Together, these mechanisms
form a multi-level defense strategy to filter out malicious contributions
effectively. Simulation results show that the proposed algorithm significantly
reduces convergence time compared to benchmark methods across both 1-hop and
3-hop topologies.


http://arxiv.org/abs/2505.01518
Rubber Mallet: A Study of High Frequency Localized Bit Flips and Their Impact on Security. (9%)
Andrew Adiletta; Zane Weissman; Fatemeh Khojasteh Dana; Berk Sunar; Shahin Tajik
  The increasing density of modern DRAM has heightened its vulnerability to
Rowhammer attacks, which induce bit flips by repeatedly accessing specific
memory rows. This paper presents an analysis of bit flip patterns generated by
advanced Rowhammer techniques that bypass existing hardware defenses. First, we
investigate the phenomenon of adjacent bit flips--where two or more physically
neighboring bits are corrupted simultaneously--and demonstrate they occur with
significantly higher frequency than previously documented. We also show that if
multiple bits flip within a byte, they are more likely to be adjacent than
randomly distributed: for example, if 4 bits flip within a byte, there is an
87% chance that they are all adjacent. We also demonstrate that bit flips
within a row will naturally cluster together likely due to the underlying
physics of the attack. We then investigate two fault injection attacks enabled
by multiple adjacent or nearby bit flips. First, we show how these correlated
flips enable efficient cryptographic signature correction attacks, successfully
recovering ECDSA private keys from OpenSSL implementations where single-bit
approaches would be unfeasible. Second, we introduce a targeted attack against
large language models by exploiting Rowhammer-induced corruptions in tokenizer
dictionaries of GGUF model files. This attack effectively rewrites safety
instructions in system prompts by swapping safety-critical tokens with benign
alternatives, circumventing model guardrails while maintaining normal
functionality in other contexts. Our experimental results across multiple DRAM
configurations reveal that current memory protection schemes are inadequate
against these sophisticated attack vectors, which can achieve their objectives
with precise, minimal modifications rather than random corruption.


http://arxiv.org/abs/2505.01292
Fine-grained Manipulation Attacks to Local Differential Privacy Protocols for Data Streams. (8%)
Xinyu Li; Xuebin Ren; Shusen Yang; Liang Shi; Chia-Mu Yu
  Local Differential Privacy (LDP) enables massive data collection and analysis
while protecting end users' privacy against untrusted aggregators. It has been
applied to various data types (e.g., categorical, numerical, and graph data)
and application settings (e.g., static and streaming). Recent findings indicate
that LDP protocols can be easily disrupted by poisoning or manipulation
attacks, which leverage injected/corrupted fake users to send crafted data
conforming to the LDP reports. However, current attacks primarily target static
protocols, neglecting the security of LDP protocols in the streaming settings.
Our research fills the gap by developing novel fine-grained manipulation
attacks to LDP protocols for data streams. By reviewing the attack surfaces in
existing algorithms, We introduce a unified attack framework with composable
modules, which can manipulate the LDP estimated stream toward a target stream.
Our attack framework can adapt to state-of-the-art streaming LDP algorithms
with different analytic tasks (e.g., frequency and mean) and LDP models
(event-level, user-level, w-event level). We validate our attacks theoretically
and through extensive experiments on real-world datasets, and finally explore a
possible defense mechanism for mitigating these attacks.


http://arxiv.org/abs/2505.01177
LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasures. (4%)
Francisco Aguilera-Martínez; Fernando Berzal
  As large language models (LLMs) continue to evolve, it is critical to assess
the security threats and vulnerabilities that may arise both during their
training phase and after models have been deployed. This survey seeks to define
and categorize the various attacks targeting LLMs, distinguishing between those
that occur during the training phase and those that affect already trained
models. A thorough analysis of these attacks is presented, alongside an
exploration of defense mechanisms designed to mitigate such threats. Defenses
are classified into two primary categories: prevention-based and
detection-based defenses. Furthermore, our survey summarizes possible attacks
and their corresponding defense strategies. It also provides an evaluation of
the effectiveness of the known defense mechanisms for the different security
threats. Our survey aims to offer a structured framework for securing LLMs,
while also identifying areas that require further research to improve and
strengthen defenses against emerging security challenges.


http://arxiv.org/abs/2505.01181
Explainable AI Based Diagnosis of Poisoning Attacks in Evolutionary Swarms. (1%)
Mehrdad Asadi; Roxana Rădulescu; Ann Nowé
  Swarming systems, such as for example multi-drone networks, excel at
cooperative tasks like monitoring, surveillance, or disaster assistance in
critical environments, where autonomous agents make decentralized decisions in
order to fulfill team-level objectives in a robust and efficient manner.
Unfortunately, team-level coordinated strategies in the wild are vulnerable to
data poisoning attacks, resulting in either inaccurate coordination or
adversarial behavior among the agents. To address this challenge, we contribute
a framework that investigates the effects of such data poisoning attacks, using
explainable AI methods. We model the interaction among agents using
evolutionary intelligence, where an optimal coalition strategically emerges to
perform coordinated tasks. Then, through a rigorous evaluation, the swarm model
is systematically poisoned using data manipulation attacks. We showcase the
applicability of explainable AI methods to quantify the effects of poisoning on
the team strategy and extract footprint characterizations that enable
diagnosing. Our findings indicate that when the model is poisoned above 10%,
non-optimal strategies resulting in inefficient cooperation can be identified.


http://arxiv.org/abs/2505.00843
OET: Optimization-based prompt injection Evaluation Toolkit. (99%)
Jinsheng Pan; Xiaogeng Liu; Chaowei Xiao
  Large Language Models (LLMs) have demonstrated remarkable capabilities in
natural language understanding and generation, enabling their widespread
adoption across various domains. However, their susceptibility to prompt
injection attacks poses significant security risks, as adversarial inputs can
manipulate model behavior and override intended instructions. Despite numerous
defense strategies, a standardized framework to rigorously evaluate their
effectiveness, especially under adaptive adversarial scenarios, is lacking. To
address this gap, we introduce OET, an optimization-based evaluation toolkit
that systematically benchmarks prompt injection attacks and defenses across
diverse datasets using an adaptive testing framework. Our toolkit features a
modular workflow that facilitates adversarial string generation, dynamic attack
execution, and comprehensive result analysis, offering a unified platform for
assessing adversarial robustness. Crucially, the adaptive testing framework
leverages optimization methods with both white-box and black-box access to
generate worst-case adversarial examples, thereby enabling strict red-teaming
evaluations. Extensive experiments underscore the limitations of current
defense mechanisms, with some models remaining susceptible even after
implementing security enhancements.


http://arxiv.org/abs/2505.00598
Fast and Low-Cost Genomic Foundation Models via Outlier Removal. (98%)
Haozheng Luo; Chenghao Qiu; Maojiang Su; Zhihan Zhou; Zoe Mehta; Guo Ye; Jerry Yao-Chieh Hu; Han Liu
  We propose the first unified adversarial attack benchmark for Genomic
Foundation Models (GFMs), named GERM. Unlike existing GFM benchmarks, GERM
offers the first comprehensive evaluation framework to systematically assess
the vulnerability of GFMs to adversarial attacks. Methodologically, we evaluate
the adversarial robustness of five state-of-the-art GFMs using four widely
adopted attack algorithms and three defense strategies. Importantly, our
benchmark provides an accessible and comprehensive framework to analyze GFM
vulnerabilities with respect to model architecture, quantization schemes, and
training datasets. Empirically, transformer-based models exhibit greater
robustness to adversarial perturbations compared to HyenaDNA, highlighting the
impact of architectural design on vulnerability. Moreover, adversarial attacks
frequently target biologically significant genomic regions, suggesting that
these models effectively capture meaningful sequence features.


http://arxiv.org/abs/2505.00487
Analysis of the vulnerability of machine learning regression models to adversarial attacks using data from 5G wireless networks. (96%)
Leonid Legashev; Artur Zhigalov; Denis Parfenov
  This article describes the process of creating a script and conducting an
analytical study of a dataset using the DeepMIMO emulator. An advertorial
attack was carried out using the FGSM method to maximize the gradient. A
comparison is made of the effectiveness of binary classifiers in the task of
detecting distorted data. The dynamics of changes in the quality indicators of
the regression model were analyzed in conditions without adversarial attacks,
during an adversarial attack and when the distorted data was isolated. It is
shown that an adversarial FGSM attack with gradient maximization leads to an
increase in the value of the MSE metric by 33% and a decrease in the R2
indicator by 10% on average. The LightGBM binary classifier effectively
identifies data with adversarial anomalies with 98% accuracy. Regression
machine learning models are susceptible to adversarial attacks, but rapid
analysis of network traffic and data transmitted over the network makes it
possible to identify malicious activity


http://arxiv.org/abs/2505.00395
GAN-based Generator of Adversarial Attack on Intelligent End-to-End Autoencoder-based Communication System. (91%)
Jianyuan Chen; Lin Zhang; Zuwei Chen; Yawen Chen; Hongcheng Zhuang
  Deep neural networks have been applied in wireless communications system to
intelligently adapt to dynamically changing channel conditions, while the users
are still under the threat of the malicious attacks due to the broadcasting
property of wireless channels. However, most attack models require the
knowledge of the target details, which is difficult to be implemented in real
systems. Our objective is to develop an attack model with no requirement for
the target information, while enhancing the block error rate. In our design, we
propose a novel Generative Adversarial Networks(GANs) based attack
architecture, which exploits the property of deep learning models being
vulnerable to perturbations induced by dynamically changing channel conditions.
In the proposed generator, the attack network is composed of convolution layer,
convolution transpose layer and linear layer. Then we present the training
strategy and the details of the training algorithm. Subsequently, we propose
the validation strategy to evaluate the performance of the generator.
Simulations are conducted and the results show that our proposed adversarial
attack generator achieve better block error rate attack performance than that
of benchmark schemes over Additive White Gaussian Noise (AWGN) channel,
Rayleigh channel and High-Speed Railway channel.


http://arxiv.org/abs/2505.00881
Protocol-agnostic and Data-free Backdoor Attacks on Pre-trained Models in RF Fingerprinting. (16%)
Tianya Zhao; Ningning Wang; Junqing Zhang; Xuyu Wang
  While supervised deep neural networks (DNNs) have proven effective for device
authentication via radio frequency (RF) fingerprinting, they are hindered by
domain shift issues and the scarcity of labeled data. The success of large
language models has led to increased interest in unsupervised pre-trained
models (PTMs), which offer better generalization and do not require labeled
datasets, potentially addressing the issues mentioned above. However, the
inherent vulnerabilities of PTMs in RF fingerprinting remain insufficiently
explored. In this paper, we thoroughly investigate data-free backdoor attacks
on such PTMs in RF fingerprinting, focusing on a practical scenario where
attackers lack access to downstream data, label information, and training
processes. To realize the backdoor attack, we carefully design a set of
triggers and predefined output representations (PORs) for the PTMs. By mapping
triggers and PORs through backdoor training, we can implant backdoor behaviors
into the PTMs, thereby introducing vulnerabilities across different downstream
RF fingerprinting tasks without requiring prior knowledge. Extensive
experiments demonstrate the wide applicability of our proposed attack to
various input domains, protocols, and PTMs. Furthermore, we explore potential
detection and defense methods, demonstrating the difficulty of fully
safeguarding against our proposed backdoor attack.


http://arxiv.org/abs/2505.00976
Attack and defense techniques in large language models: A survey and new perspectives. (12%)
Zhiyu Liao; Kang Chen; Yuanguo Lin; Kangkang Li; Yunxuan Liu; Hefeng Chen; Xingwang Huang; Yuanhui Yu
  Large Language Models (LLMs) have become central to numerous natural language
processing tasks, but their vulnerabilities present significant security and
ethical challenges. This systematic survey explores the evolving landscape of
attack and defense techniques in LLMs. We classify attacks into adversarial
prompt attack, optimized attacks, model theft, as well as attacks on
application of LLMs, detailing their mechanisms and implications. Consequently,
we analyze defense strategies, including prevention-based and detection-based
defense methods. Although advances have been made, challenges remain to adapt
to the dynamic threat landscape, balance usability with robustness, and address
resource constraints in defense implementation. We highlight open problems,
including the need for adaptive scalable defenses, explainable security
techniques, and standardized evaluation frameworks. This survey provides
actionable insights and directions for developing secure and resilient LLMs,
emphasizing the importance of interdisciplinary collaboration and ethical
considerations to mitigate risks in real-world applications.


http://arxiv.org/abs/2505.00817
Spill The Beans: Exploiting CPU Cache Side-Channels to Leak Tokens from Large Language Models. (1%)
Andrew Adiletta; Berk Sunar
  Side-channel attacks on shared hardware resources increasingly threaten
confidentiality, especially with the rise of Large Language Models (LLMs). In
this work, we introduce Spill The Beans, a novel application of cache
side-channels to leak tokens generated by an LLM. By co-locating an attack
process on the same hardware as the victim model, we flush and reload embedding
vectors from the embedding layer, where each token corresponds to a unique
embedding vector. When accessed during token generation, it results in a cache
hit detectable by our attack on shared lower-level caches.
  A significant challenge is the massive size of LLMs, which, by nature of
their compute intensive operation, quickly evicts embedding vectors from the
cache. We address this by balancing the number of tokens monitored against the
amount of information leaked. Monitoring more tokens increases potential
vocabulary leakage but raises the chance of missing cache hits due to eviction;
monitoring fewer tokens improves detection reliability but limits vocabulary
coverage.
  Through extensive experimentation, we demonstrate the feasibility of leaking
tokens from LLMs via cache side-channels. Our findings reveal a new
vulnerability in LLM deployments, highlighting that even sophisticated models
are susceptible to traditional side-channel attacks. We discuss the
implications for privacy and security in LLM-serving infrastructures and
suggest considerations for mitigating such threats. For proof of concept we
consider two concrete attack scenarios: Our experiments show that an attacker
can recover as much as 80%-90% of a high entropy API key with single shot
monitoring. As for English text we can reach a 40% recovery rate with a single
shot. We should note that the rate highly depends on the monitored token set
and these rates can be improved by targeting more specialized output domains.


http://arxiv.org/abs/2504.21730
Cert-SSB: Toward Certified Sample-Specific Backdoor Defense. (98%)
Ting Qiao; Yingjia Wang; Xing Liu; Sixing Wu; Jianbing Li; Yiming Li
  Deep neural networks (DNNs) are vulnerable to backdoor attacks, where an
attacker manipulates a small portion of the training data to implant hidden
backdoors into the model. The compromised model behaves normally on clean
samples but misclassifies backdoored samples into the attacker-specified target
class, posing a significant threat to real-world DNN applications. Currently,
several empirical defense methods have been proposed to mitigate backdoor
attacks, but they are often bypassed by more advanced backdoor techniques. In
contrast, certified defenses based on randomized smoothing have shown promise
by adding random noise to training and testing samples to counteract backdoor
attacks. In this paper, we reveal that existing randomized smoothing defenses
implicitly assume that all samples are equidistant from the decision boundary.
However, it may not hold in practice, leading to suboptimal certification
performance. To address this issue, we propose a sample-specific certified
backdoor defense method, termed Cert-SSB. Cert-SSB first employs stochastic
gradient ascent to optimize the noise magnitude for each sample, ensuring a
sample-specific noise level that is then applied to multiple poisoned training
sets to retrain several smoothed models. After that, Cert-SSB aggregates the
predictions of multiple smoothed models to generate the final robust
prediction. In particular, in this case, existing certification methods become
inapplicable since the optimized noise varies across different samples. To
conquer this challenge, we introduce a storage-update-based certification
method, which dynamically adjusts each sample's certification region to improve
certification performance. We conduct extensive experiments on multiple
benchmark datasets, demonstrating the effectiveness of our proposed method. Our
code is available at https://github.com/NcepuQiaoTing/Cert-SSB.


http://arxiv.org/abs/2504.21307
The Dual Power of Interpretable Token Embeddings: Jailbreaking Attacks and Defenses for Diffusion Model Unlearning. (86%)
Siyi Chen; Yimeng Zhang; Sijia Liu; Qing Qu
  Despite the remarkable generation capabilities of diffusion models, recent
studies have shown that they can memorize and create harmful content when given
specific text prompts. Although fine-tuning approaches have been developed to
mitigate this issue by unlearning harmful concepts, these methods can be easily
circumvented through jailbreaking attacks. This implies that the harmful
concept has not been fully erased from the model. However, existing
jailbreaking attack methods, while effective, lack interpretability regarding
why unlearned models still retain the concept, thereby hindering the
development of defense strategies. In this work, we address these limitations
by proposing an attack method that learns an orthogonal set of interpretable
attack token embeddings. The attack token embeddings can be decomposed into
human-interpretable textual elements, revealing that unlearned models still
retain the target concept through implicit textual components. Furthermore,
these attack token embeddings are powerful and transferable across text
prompts, initial noises, and unlearned models, emphasizing that unlearned
models are more vulnerable than expected. Finally, building on the insights
from our interpretable attack, we develop a defense method to protect unlearned
models against both our proposed and existing jailbreaking attacks. Extensive
experimental results demonstrate the effectiveness of our attack and defense
strategies.


http://arxiv.org/abs/2504.21323
How to Backdoor the Knowledge Distillation. (81%)
Chen Wu; Qian Ma; Prasenjit Mitra; Sencun Zhu
  Knowledge distillation has become a cornerstone in modern machine learning
systems, celebrated for its ability to transfer knowledge from a large, complex
teacher model to a more efficient student model. Traditionally, this process is
regarded as secure, assuming the teacher model is clean. This belief stems from
conventional backdoor attacks relying on poisoned training data with backdoor
triggers and attacker-chosen labels, which are not involved in the distillation
process. Instead, knowledge distillation uses the outputs of a clean teacher
model to guide the student model, inherently preventing recognition or response
to backdoor triggers as intended by an attacker. In this paper, we challenge
this assumption by introducing a novel attack methodology that strategically
poisons the distillation dataset with adversarial examples embedded with
backdoor triggers. This technique allows for the stealthy compromise of the
student model while maintaining the integrity of the teacher model. Our
innovative approach represents the first successful exploitation of
vulnerabilities within the knowledge distillation process using clean teacher
models. Through extensive experiments conducted across various datasets and
attack settings, we demonstrate the robustness, stealthiness, and effectiveness
of our method. Our findings reveal previously unrecognized vulnerabilities and
pave the way for future research aimed at securing knowledge distillation
processes against backdoor attacks.


http://arxiv.org/abs/2504.21646
Diffusion-based Adversarial Identity Manipulation for Facial Privacy Protection. (31%)
Liqin Wang; Qianyue Hu; Wei Lu; Xiangyang Luo
  The success of face recognition (FR) systems has led to serious privacy
concerns due to potential unauthorized surveillance and user tracking on social
networks. Existing methods for enhancing privacy fail to generate natural face
images that can protect facial privacy. In this paper, we propose
diffusion-based adversarial identity manipulation (DiffAIM) to generate natural
and highly transferable adversarial faces against malicious FR systems. To be
specific, we manipulate facial identity within the low-dimensional latent space
of a diffusion model. This involves iteratively injecting gradient-based
adversarial identity guidance during the reverse diffusion process,
progressively steering the generation toward the desired adversarial faces. The
guidance is optimized for identity convergence towards a target while promoting
semantic divergence from the source, facilitating effective impersonation while
maintaining visual naturalness. We further incorporate structure-preserving
regularization to preserve facial structure consistency during manipulation.
Extensive experiments on both face verification and identification tasks
demonstrate that compared with the state-of-the-art, DiffAIM achieves stronger
black-box attack transferability while maintaining superior visual quality. We
also demonstrate the effectiveness of the proposed approach for commercial FR
APIs, including Face++ and Aliyun.


http://arxiv.org/abs/2504.21420
A Test Suite for Efficient Robustness Evaluation of Face Recognition Systems. (26%)
Ruihan Zhang; Jun Sun
  Face recognition is a widely used authentication technology in practice,
where robustness is required. It is thus essential to have an efficient and
easy-to-use method for evaluating the robustness of (possibly third-party)
trained face recognition systems. Existing approaches to evaluating the
robustness of face recognition systems are either based on empirical evaluation
(e.g., measuring attacking success rate using state-of-the-art attacking
methods) or formal analysis (e.g., measuring the Lipschitz constant). While the
former demands significant user efforts and expertise, the latter is extremely
time-consuming. In pursuit of a comprehensive, efficient, easy-to-use and
scalable estimation of the robustness of face recognition systems, we take an
old-school alternative approach and introduce RobFace, i.e., evaluation using
an optimised test suite. It contains transferable adversarial face images that
are designed to comprehensively evaluate a face recognition system's robustness
along a variety of dimensions. RobFace is system-agnostic and still consistent
with system-specific empirical evaluation or formal analysis. We support this
claim through extensive experimental results with various perturbations on
multiple face recognition systems. To our knowledge, RobFace is the first
system-agnostic robustness estimation test suite.


http://arxiv.org/abs/2504.21680
Hoist with His Own Petard: Inducing Guardrails to Facilitate Denial-of-Service Attacks on Retrieval-Augmented Generation of LLMs. (16%)
Pan Suo; Yu-Ming Shang; San-Chuan Guo; Xi Zhang
  Retrieval-Augmented Generation (RAG) integrates Large Language Models (LLMs)
with external knowledge bases, improving output quality while introducing new
security risks. Existing studies on RAG vulnerabilities typically focus on
exploiting the retrieval mechanism to inject erroneous knowledge or malicious
texts, inducing incorrect outputs. However, these approaches overlook critical
weaknesses within LLMs, leaving important attack vectors unexplored and
limiting the scope and efficiency of attacks. In this paper, we uncover a novel
vulnerability: the safety guardrails of LLMs, while designed for protection,
can also be exploited as an attack vector by adversaries. Building on this
vulnerability, we propose MutedRAG, a novel denial-of-service attack that
reversely leverages the guardrails of LLMs to undermine the availability of RAG
systems. By injecting minimalistic jailbreak texts, such as "\textit{How to
build a bomb}", into the knowledge base, MutedRAG intentionally triggers the
LLM's safety guardrails, causing the system to reject legitimate queries.
Besides, due to the high sensitivity of guardrails, a single jailbreak sample
can affect multiple queries, effectively amplifying the efficiency of attacks
while reducing their costs. Experimental results on three datasets demonstrate
that MutedRAG achieves an attack success rate exceeding 60% in many scenarios,
requiring only less than one malicious text to each target query on average. In
addition, we evaluate potential defense strategies against MutedRAG, finding
that some of current mechanisms are insufficient to mitigate this threat,
underscoring the urgent need for more robust solutions.


http://arxiv.org/abs/2505.00061
Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems. (11%)
Sahar Yarmohammadtoosky; Yiyun Zhou; Victoria Yaneva; Peter Baldwin; Saed Rezayi; Brian Clauser; Polina Harikeo
  This study examines vulnerabilities in transformer-based automated
short-answer grading systems used in medical education, with a focus on how
these systems can be manipulated through adversarial gaming strategies. Our
research identifies three main types of gaming strategies that exploit the
system's weaknesses, potentially leading to false positives. To counteract
these vulnerabilities, we implement several adversarial training methods
designed to enhance the systems' robustness. Our results indicate that these
methods significantly reduce the susceptibility of grading systems to such
manipulations, especially when combined with ensemble techniques like majority
voting and ridge regression, which further improve the system's defense against
sophisticated adversarial inputs. Additionally, employing large language models
such as GPT-4 with varied prompting techniques has shown promise in recognizing
and scoring gaming strategies effectively. The findings underscore the
importance of continuous improvements in AI-driven educational tools to ensure
their reliability and fairness in high-stakes settings.


http://arxiv.org/abs/2505.01456
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation. (9%)
Vaidehi Patil; Yi-Lin Sung; Peter Hase; Jie Peng; Tianlong Chen; Mohit Bansal
  LLMs trained on massive datasets may inadvertently acquire sensitive
information such as personal details and potentially harmful content. This risk
is further heightened in multimodal LLMs as they integrate information from
multiple modalities (image and text). Adversaries can exploit this knowledge
through multimodal prompts to extract sensitive details. Evaluating how
effectively MLLMs can forget such information (targeted unlearning)
necessitates the creation of high-quality, well-annotated image-text pairs.
While prior work on unlearning has focused on text, multimodal unlearning
remains underexplored. To address this gap, we first introduce a multimodal
unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as
an attack-and-defense framework to evaluate methods for deleting specific
multimodal knowledge from MLLMs. We extend a visual question-answering dataset
using an automated pipeline that generates varying-proximity samples for
testing generalization and specificity, followed by manual filtering for
maintaining high quality. We then evaluate six defense objectives against seven
attacks (four whitebox, three blackbox), including a novel whitebox method
leveraging interpretability of hidden states. Our results show multimodal
attacks outperform text- or image-only ones, and that the most effective
defense removes answer information from internal model states. Additionally,
larger models exhibit greater post-editing robustness, suggesting that scale
enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing
unlearning in MLLMs.


http://arxiv.org/abs/2504.21668
Traceback of Poisoning Attacks to Retrieval-Augmented Generation. (8%)
Baolei Zhang; Haoran Xin; Minghong Fang; Zhuqing Liu; Biao Yi; Tong Li; Zheli Liu
  Large language models (LLMs) integrated with retrieval-augmented generation
(RAG) systems improve accuracy by leveraging external knowledge sources.
However, recent research has revealed RAG's susceptibility to poisoning
attacks, where the attacker injects poisoned texts into the knowledge database,
leading to attacker-desired responses. Existing defenses, which predominantly
focus on inference-time mitigation, have proven insufficient against
sophisticated attacks. In this paper, we introduce RAGForensics, the first
traceback system for RAG, designed to identify poisoned texts within the
knowledge database that are responsible for the attacks. RAGForensics operates
iteratively, first retrieving a subset of texts from the database and then
utilizing a specially crafted prompt to guide an LLM in detecting potential
poisoning texts. Empirical evaluations across multiple datasets demonstrate the
effectiveness of RAGForensics against state-of-the-art poisoning attacks. This
work pioneers the traceback of poisoned texts in RAG systems, providing a
practical and promising defense mechanism to enhance their security.


http://arxiv.org/abs/2505.00162
Stochastic Subspace Descent Accelerated via Bi-fidelity Line Search. (1%)
Nuojin Cheng; Alireza Doostan; Stephen Becker
  Efficient optimization remains a fundamental challenge across numerous
scientific and engineering domains, especially when objective function and
gradient evaluations are computationally expensive. While zeroth-order
optimization methods offer effective approaches when gradients are
inaccessible, their practical performance can be limited by the high cost
associated with function queries. This work introduces the bi-fidelity
stochastic subspace descent (BF-SSD) algorithm, a novel zeroth-order
optimization method designed to reduce this computational burden. BF-SSD
leverages a bi-fidelity framework, constructing a surrogate model from a
combination of computationally inexpensive low-fidelity (LF) and accurate
high-fidelity (HF) function evaluations. This surrogate model facilitates an
efficient backtracking line search for step size selection, for which we
provide theoretical convergence guarantees under standard assumptions. We
perform a comprehensive empirical evaluation of BF-SSD across four distinct
problems: a synthetic optimization benchmark, dual-form kernel ridge
regression, black-box adversarial attacks on machine learning models, and
transformer-based black-box language model fine-tuning. Numerical results
demonstrate that BF-SSD consistently achieves superior optimization performance
while requiring significantly fewer HF function evaluations compared to
relevant baseline methods. This study highlights the efficacy of integrating
bi-fidelity strategies within zeroth-order optimization, positioning BF-SSD as
a promising and computationally efficient approach for tackling large-scale,
high-dimensional problems encountered in various real-world applications.


http://arxiv.org/abs/2504.20848
Mitigating the Structural Bias in Graph Adversarial Defenses. (92%)
Junyuan Fang; Huimin Liu; Han Yang; Jiajing Wu; Zibin Zheng; Chi K. Tse
  In recent years, graph neural networks (GNNs) have shown great potential in
addressing various graph structure-related downstream tasks. However, recent
studies have found that current GNNs are susceptible to malicious adversarial
attacks. Given the inevitable presence of adversarial attacks in the real
world, a variety of defense methods have been proposed to counter these attacks
and enhance the robustness of GNNs. Despite the commendable performance of
these defense methods, we have observed that they tend to exhibit a structural
bias in terms of their defense capability on nodes with low degree (i.e., tail
nodes), which is similar to the structural bias of traditional GNNs on nodes
with low degree in the clean graph. Therefore, in this work, we propose a
defense strategy by including hetero-homo augmented graph construction, $k$NN
augmented graph construction, and multi-view node-wise attention modules to
mitigate the structural bias of GNNs against adversarial attacks. Notably, the
hetero-homo augmented graph consists of removing heterophilic links (i.e.,
links connecting nodes with dissimilar features) globally and adding homophilic
links (i.e., links connecting nodes with similar features) for nodes with low
degree. To further enhance the defense capability, an attention mechanism is
adopted to adaptively combine the representations from the above two kinds of
graph views. We conduct extensive experiments to demonstrate the defense and
debiasing effect of the proposed strategy on benchmark datasets.


http://arxiv.org/abs/2504.21052
SFIBA: Spatial-based Full-target Invisible Backdoor Attacks. (84%)
Yangxu Yin; Honglong Chen; Yudong Gao; Peng Sun; Zhishuai Li; Weifeng Liu
  Multi-target backdoor attacks pose significant security threats to deep
neural networks, as they can preset multiple target classes through a single
backdoor injection. This allows attackers to control the model to misclassify
poisoned samples with triggers into any desired target class during inference,
exhibiting superior attack performance compared with conventional backdoor
attacks. However, existing multi-target backdoor attacks fail to guarantee
trigger specificity and stealthiness in black-box settings, resulting in two
main issues. First, they are unable to simultaneously target all classes when
only training data can be manipulated, limiting their effectiveness in
realistic attack scenarios. Second, the triggers often lack visual
imperceptibility, making poisoned samples easy to detect. To address these
problems, we propose a Spatial-based Full-target Invisible Backdoor Attack,
called SFIBA. It restricts triggers for different classes to specific local
spatial regions and morphologies in the pixel space to ensure specificity,
while employing a frequency-domain-based trigger injection method to guarantee
stealthiness. Specifically, for injection of each trigger, we first apply fast
fourier transform to obtain the amplitude spectrum of clean samples in local
spatial regions. Then, we employ discrete wavelet transform to extract the
features from the amplitude spectrum and use singular value decomposition to
integrate the trigger. Subsequently, we selectively filter parts of the trigger
in pixel space to implement trigger morphology constraints and adjust injection
coefficients based on visual effects. We conduct experiments on multiple
datasets and models. The results demonstrate that SFIBA can achieve excellent
attack performance and stealthiness, while preserving the model's performance
on benign samples, and can also bypass existing backdoor defenses.


http://arxiv.org/abs/2504.20472
Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction. (81%)
Yulin Chen; Haoran Li; Yuan Sui; Yue Liu; Yufei He; Yangqiu Song; Bryan Hooi
  Large language models (LLMs) have demonstrated impressive performance and
have come to dominate the field of natural language processing (NLP) across
various tasks. However, due to their strong instruction-following capabilities
and inability to distinguish between instructions and data content, LLMs are
vulnerable to prompt injection attacks. These attacks manipulate LLMs into
deviating from the original input instructions and executing maliciously
injected instructions within data content, such as web documents retrieved from
search engines. Existing defense methods, including prompt-engineering and
fine-tuning approaches, typically instruct models to follow the original input
instructions while suppressing their tendencies to execute injected
instructions. However, our experiments reveal that suppressing
instruction-following tendencies is challenging. Through analyzing failure
cases, we observe that although LLMs tend to respond to any recognized
instructions, they are aware of which specific instructions they are executing
and can correctly reference them within the original prompt. Motivated by these
findings, we propose a novel defense method that leverages, rather than
suppresses, the instruction-following abilities of LLMs. Our approach prompts
LLMs to generate responses that include both answers and their corresponding
instruction references. Based on these references, we filter out answers not
associated with the original input instructions. Comprehensive experiments
demonstrate that our method outperforms prompt-engineering baselines and
achieves performance comparable to fine-tuning methods, reducing the attack
success rate (ASR) to 0 percent in some scenarios. Moreover, our approach has
minimal impact on overall utility.


http://arxiv.org/abs/2504.20869
Quantifying the Noise of Structural Perturbations on Graph Adversarial Attacks. (80%)
Junyuan Fang; Han Yang; Haixian Wen; Jiajing Wu; Zibin Zheng; Chi K. Tse
  Graph neural networks have been widely utilized to solve graph-related tasks
because of their strong learning power in utilizing the local information of
neighbors. However, recent studies on graph adversarial attacks have proven
that current graph neural networks are not robust against malicious attacks.
Yet much of the existing work has focused on the optimization objective based
on attack performance to obtain (near) optimal perturbations, but paid less
attention to the strength quantification of each perturbation such as the
injection of a particular node/link, which makes the choice of perturbations a
black-box model that lacks interpretability. In this work, we propose the
concept of noise to quantify the attack strength of each adversarial link.
Furthermore, we propose three attack strategies based on the defined noise and
classification margins in terms of single and multiple steps optimization.
Extensive experiments conducted on benchmark datasets against three
representative graph neural networks demonstrate the effectiveness of the
proposed attack strategies. Particularly, we also investigate the preferred
patterns of effective adversarial perturbations by analyzing the corresponding
properties of the selected perturbation nodes.


http://arxiv.org/abs/2504.20965
AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security. (15%)
Zikui Cai; Shayan Shabihi; Bang An; Zora Che; Brian R. Bartoldson; Bhavya Kailkhura; Tom Goldstein; Furong Huang
  We introduce AegisLLM, a cooperative multi-agent defense against adversarial
attacks and information leakage. In AegisLLM, a structured workflow of
autonomous agents - orchestrator, deflector, responder, and evaluator -
collaborate to ensure safe and compliant LLM outputs, while self-improving over
time through prompt optimization. We show that scaling agentic reasoning system
at test-time - both by incorporating additional agent roles and by leveraging
automated prompt optimization (such as DSPy)- substantially enhances robustness
without compromising model utility. This test-time defense enables real-time
adaptability to evolving attacks, without requiring model retraining.
Comprehensive evaluations across key threat scenarios, including unlearning and
jailbreaking, demonstrate the effectiveness of AegisLLM. On the WMDP unlearning
benchmark, AegisLLM achieves near-perfect unlearning with only 20 training
examples and fewer than 300 LM calls. For jailbreaking benchmarks, we achieve
51% improvement compared to the base model on StrongReject, with false refusal
rates of only 7.9% on PHTest compared to 18-55% for comparable methods. Our
results highlight the advantages of adaptive, agentic reasoning over static
defenses, establishing AegisLLM as a strong runtime alternative to traditional
approaches based on model modifications. Code is available at
https://github.com/zikuicai/aegisllm


http://arxiv.org/abs/2504.21278
Robust Multi-agent Communication Based on Decentralization-Oriented Adversarial Training. (12%)
Xuyan Ma; Yawen Wang; Junjie Wang; Xiaofei Xie; Boyu Wu; Shoubin Li; Fanjiang Xu; Qing Wang
  In typical multi-agent reinforcement learning (MARL) problems, communication
is important for agents to share information and make the right decisions.
However, due to the complexity of training multi-agent communication, existing
methods often fall into the dilemma of local optimization, which leads to the
concentration of communication in a limited number of channels and presents an
unbalanced structure. Such unbalanced communication policy are vulnerable to
abnormal conditions, where the damage of critical communication channels can
trigger the crash of the entire system. Inspired by decentralization theory in
sociology, we propose DMAC, which enhances the robustness of multi-agent
communication policies by retraining them into decentralized patterns.
Specifically, we train an adversary DMAC\_Adv which can dynamically identify
and mask the critical communication channels, and then apply the adversarial
samples generated by DMAC\_Adv to the adversarial learning of the communication
policy to force the policy in exploring other potential communication schemes
and transition to a decentralized structure. As a training method to improve
robustness, DMAC can be fused with any learnable communication policy
algorithm. The experimental results in two communication policies and four
multi-agent tasks demonstrate that DMAC achieves higher improvement on
robustness and performance of communication policy compared with two
state-of-the-art and commonly-used baselines. Also, the results demonstrate
that DMAC can achieve decentralized communication structure with acceptable
communication cost.


http://arxiv.org/abs/2504.21228
CachePrune: Neural-Based Attribution Defense Against Indirect Prompt Injection Attacks. (12%)
Rui Wang; Junda Wu; Yu Xia; Tong Yu; Ruiyi Zhang; Ryan Rossi; Lina Yao; Julian McAuley
  Large Language Models (LLMs) are identified as being susceptible to indirect
prompt injection attack, where the model undesirably deviates from
user-provided instructions by executing tasks injected in the prompt context.
This vulnerability stems from LLMs' inability to distinguish between data and
instructions within a prompt. In this paper, we propose CachePrune that defends
against this attack by identifying and pruning task-triggering neurons from the
KV cache of the input prompt context. By pruning such neurons, we encourage the
LLM to treat the text spans of input prompt context as only pure data, instead
of any indicator of instruction following. These neurons are identified via
feature attribution with a loss function induced from an upperbound of the
Direct Preference Optimization (DPO) objective. We show that such a loss
function enables effective feature attribution with only a few samples. We
further improve on the quality of feature attribution, by exploiting an
observed triggering effect in instruction following. Our approach does not
impose any formatting on the original prompt or introduce extra test-time LLM
calls. Experiments show that CachePrune significantly reduces attack success
rates without compromising the response quality. Note: This paper aims to
defend against indirect prompt injection attacks, with the goal of developing
more secure and robust AI systems.


http://arxiv.org/abs/2504.20829
GaussTrap: Stealthy Poisoning Attacks on 3D Gaussian Splatting for Targeted Scene Confusion. (8%)
Jiaxin Hong; Sixu Chen; Shuoyang Sun; Hongyao Yu; Hao Fang; Yuqi Tan; Bin Chen; Shuhan Qi; Jiawei Li
  As 3D Gaussian Splatting (3DGS) emerges as a breakthrough in scene
representation and novel view synthesis, its rapid adoption in safety-critical
domains (e.g., autonomous systems, AR/VR) urgently demands scrutiny of
potential security vulnerabilities. This paper presents the first systematic
study of backdoor threats in 3DGS pipelines. We identify that adversaries may
implant backdoor views to induce malicious scene confusion during inference,
potentially leading to environmental misperception in autonomous navigation or
spatial distortion in immersive environments. To uncover this risk, we propose
GuassTrap, a novel poisoning attack method targeting 3DGS models. GuassTrap
injects malicious views at specific attack viewpoints while preserving
high-quality rendering in non-target views, ensuring minimal detectability and
maximizing potential harm. Specifically, the proposed method consists of a
three-stage pipeline (attack, stabilization, and normal training) to implant
stealthy, viewpoint-consistent poisoned renderings in 3DGS, jointly optimizing
attack efficacy and perceptual realism to expose security risks in 3D
rendering. Extensive experiments on both synthetic and real-world datasets
demonstrate that GuassTrap can effectively embed imperceptible yet harmful
backdoor views while maintaining high-quality rendering in normal views,
validating its robustness, adaptability, and practical applicability.


http://arxiv.org/abs/2504.20414
Enhancing Leakage Attacks on Searchable Symmetric Encryption Using LLM-Based Synthetic Data Generation. (8%)
Joshua Chiu; Partha Protim Paul; Zahin Wahab
  Searchable Symmetric Encryption (SSE) enables efficient search capabilities
over encrypted data, allowing users to maintain privacy while utilizing cloud
storage. However, SSE schemes are vulnerable to leakage attacks that exploit
access patterns, search frequency, and volume information. Existing studies
frequently assume that adversaries possess a substantial fraction of the
encrypted dataset to mount effective inference attacks, implying there is a
database leakage of such documents, thus, an assumption that may not hold in
real-world scenarios. In this work, we investigate the feasibility of enhancing
leakage attacks under a more realistic threat model in which adversaries have
access to minimal leaked data. We propose a novel approach that leverages large
language models (LLMs), specifically GPT-4 variants, to generate synthetic
documents that statistically and semantically resemble the real-world dataset
of Enron emails. Using the email corpus as a case study, we evaluate the
effectiveness of synthetic data generated via random sampling and hierarchical
clustering methods on the performance of the SAP (Search Access Pattern)
keyword inference attack restricted to token volumes only. Our results
demonstrate that, while the choice of LLM has limited effect, increasing
dataset size and employing clustering-based generation significantly improve
attack accuracy, achieving comparable performance to attacks using larger
amounts of real data. We highlight the growing relevance of LLMs in adversarial
contexts.


http://arxiv.org/abs/2504.21072
Erased but Not Forgotten: How Backdoors Compromise Concept Erasure. (5%)
Jonas Henry Grebe; Tobias Braun; Marcus Rohrbach; Anna Rohrbach
  The expansion of large-scale text-to-image diffusion models has raised
growing concerns about their potential to generate undesirable or harmful
content, ranging from fabricated depictions of public figures to sexually
explicit images. To mitigate these risks, prior work has devised machine
unlearning techniques that attempt to erase unwanted concepts through
fine-tuning. However, in this paper, we introduce a new threat model, Toxic
Erasure (ToxE), and demonstrate how recent unlearning algorithms, including
those explicitly designed for robustness, can be circumvented through targeted
backdoor attacks. The threat is realized by establishing a link between a
trigger and the undesired content. Subsequent unlearning attempts fail to erase
this link, allowing adversaries to produce harmful content. We instantiate ToxE
via two established backdoor attacks: one targeting the text encoder and
another manipulating the cross-attention layers. Further, we introduce Deep
Intervention Score-based Attack (DISA), a novel, deeper backdoor attack that
optimizes the entire U-Net using a score-based objective, improving the
attack's persistence across different erasure methods. We evaluate five recent
concept erasure methods against our threat model. For celebrity identity
erasure, our deep attack circumvents erasure with up to 82% success, averaging
57% across all erasure methods. For explicit content erasure, ToxE attacks can
elicit up to 9 times more exposed body parts, with DISA yielding an average
increase by a factor of 2.9. These results highlight a critical security gap in
current unlearning strategies.


http://arxiv.org/abs/2504.21053
NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Models. (1%)
Yi Zhou; Wenpeng Xing; Dezhang Kong; Changting Lin; Meng Han
  Safety alignment in large language models (LLMs) is achieved through
fine-tuning mechanisms that regulate neuron activations to suppress harmful
content. In this work, we propose a novel approach to induce disalignment by
identifying and modifying the neurons responsible for safety constraints. Our
method consists of three key steps: Neuron Activation Analysis, where we
examine activation patterns in response to harmful and harmless prompts to
detect neurons that are critical for distinguishing between harmful and
harmless inputs; Similarity-Based Neuron Identification, which systematically
locates the neurons responsible for safe alignment; and Neuron Relearning for
Safety Removal, where we fine-tune these selected neurons to restore the
model's ability to generate previously restricted responses. Experimental
results demonstrate that our method effectively removes safety constraints with
minimal fine-tuning, highlighting a critical vulnerability in current alignment
techniques. Our findings underscore the need for robust defenses against
adversarial fine-tuning attacks on LLMs.


http://arxiv.org/abs/2504.20493
Token-Efficient Prompt Injection Attack: Provoking Cessation in LLM Reasoning via Adaptive Token Compression. (1%)
Yu Cui; Yujun Cai; Yiwei Wang
  While reasoning large language models (LLMs) demonstrate remarkable
performance across various tasks, they also contain notable security
vulnerabilities. Recent research has uncovered a "thinking-stopped"
vulnerability in DeepSeek-R1, where model-generated reasoning tokens can
forcibly interrupt the inference process, resulting in empty responses that
compromise LLM-integrated applications. However, existing methods triggering
this vulnerability require complex mathematical word problems with long
prompts--even exceeding 5,000 tokens. To reduce the token cost and formally
define this vulnerability, we propose a novel prompt injection attack named
"Reasoning Interruption Attack", based on adaptive token compression. We
demonstrate that simple standalone arithmetic tasks can effectively trigger
this vulnerability, and the prompts based on such tasks exhibit simpler logical
structures than mathematical word problems. We develop a systematic approach to
efficiently collect attack prompts and an adaptive token compression framework
that utilizes LLMs to automatically compress these prompts. Experiments show
our compression framework significantly reduces prompt length while maintaining
effective attack capabilities. We further investigate the attack's performance
via output prefix and analyze the underlying causes of the vulnerability,
providing valuable insights for improving security in reasoning LLMs.


http://arxiv.org/abs/2504.19730
Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge. (98%)
Wenhan Mu; Ling Xu; Shuren Pei; Le Mi; Huichi Zhou
  The widespread adoption of code language models in software engineering tasks
has exposed vulnerabilities to adversarial attacks, especially the identifier
substitution attacks. Although existing identifier substitution attackers
demonstrate high success rates, they often produce adversarial examples with
unnatural code patterns. In this paper, we systematically assess the quality of
adversarial examples using LLM-as-a-Judge. Our analysis reveals that over 80%
of adversarial examples generated by state-of-the-art identifier substitution
attackers (e.g., ALERT) are actually detectable. Based on this insight, we
propose EP-Shield, a unified framework for evaluating and purifying identifier
substitution attacks via naturalness-aware reasoning. Specifically, we first
evaluate the naturalness of code and identify the perturbed adversarial code,
then purify it so that the victim model can restore correct prediction.
Extensive experiments demonstrate the superiority of EP-Shield over adversarial
fine-tuning (up to 83.36% improvement) and its lightweight design 7B
parameters) with GPT-4-level performance.


http://arxiv.org/abs/2504.20295
The Dark Side of Digital Twins: Adversarial Attacks on AI-Driven Water Forecasting. (98%)
Mohammadhossein Homaei; Victor Gonzalez Morales; Oscar Mogollon-Gutierrez; Andres Caro
  Digital twins (DTs) are improving water distribution systems by using
real-time data, analytics, and prediction models to optimize operations. This
paper presents a DT platform designed for a Spanish water supply network,
utilizing Long Short-Term Memory (LSTM) networks to predict water consumption.
However, machine learning models are vulnerable to adversarial attacks, such as
the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD).
These attacks manipulate critical model parameters, injecting subtle
distortions that degrade forecasting accuracy. To further exploit these
vulnerabilities, we introduce a Learning Automata (LA) and Random LA-based
approach that dynamically adjusts perturbations, making adversarial attacks
more difficult to detect. Experimental results show that this approach
significantly impacts prediction reliability, causing the Mean Absolute
Percentage Error (MAPE) to rise from 26% to over 35%. Moreover, adaptive attack
strategies amplify this effect, highlighting cybersecurity risks in AI-driven
DTs. These findings emphasize the urgent need for robust defenses, including
adversarial training, anomaly detection, and secure data pipelines.


http://arxiv.org/abs/2504.21044
AGATE: Stealthy Black-box Watermarking for Multimodal Model Copyright Protection. (89%)
Jianbo Gao; Keke Gai; Jing Yu; Liehuang Zhu; Qi Wu
  Recent advancement in large-scale Artificial Intelligence (AI) models
offering multimodal services have become foundational in AI systems, making
them prime targets for model theft. Existing methods select Out-of-Distribution
(OoD) data as backdoor watermarks and retrain the original model for copyright
protection. However, existing methods are susceptible to malicious detection
and forgery by adversaries, resulting in watermark evasion. In this work, we
propose Model-\underline{ag}nostic Black-box Backdoor W\underline{ate}rmarking
Framework (AGATE) to address stealthiness and robustness challenges in
multimodal model copyright protection. Specifically, we propose an adversarial
trigger generation method to generate stealthy adversarial triggers from
ordinary dataset, providing visual fidelity while inducing semantic shifts. To
alleviate the issue of anomaly detection among model outputs, we propose a
post-transform module to correct the model output by narrowing the distance
between adversarial trigger image embedding and text embedding. Subsequently, a
two-phase watermark verification is proposed to judge whether the current model
infringes by comparing the two results with and without the transform module.
Consequently, we consistently outperform state-of-the-art methods across five
datasets in the downstream tasks of multimodal image-text retrieval and image
classification. Additionally, we validated the robustness of AGATE under two
adversarial attack scenarios.


http://arxiv.org/abs/2504.20310
A Cryptographic Perspective on Mitigation vs. Detection in Machine Learning. (83%)
Greg Gluch; Shafi Goldwasser
  In this paper, we initiate a cryptographically inspired theoretical study of
detection versus mitigation of adversarial inputs produced by attackers of
Machine Learning algorithms during inference time.
  We formally define defense by detection (DbD) and defense by mitigation
(DbM). Our definitions come in the form of a 3-round protocol between two
resource-bounded parties: a trainer/defender and an attacker. The attacker aims
to produce inference-time inputs that fool the training algorithm. We define
correctness, completeness, and soundness properties to capture successful
defense at inference time while not degrading (too much) the performance of the
algorithm on inputs from the training distribution.
  We first show that achieving DbD and achieving DbM are equivalent for ML
classification tasks. Surprisingly, this is not the case for ML generative
learning tasks, where there are many possible correct outputs that can be
generated for each input. We show a separation between DbD and DbM by
exhibiting a generative learning task for which is possible to defend by
mitigation but is provably impossible to defend by detection under the
assumption that the Identity-Based Fully Homomorphic Encryption (IB-FHE),
publicly-verifiable zero-knowledge Succinct Non-Interactive Arguments of
Knowledge (zk-SNARK) and Strongly Unforgeable Signatures exist. The mitigation
phase uses significantly fewer samples than the initial training algorithm.


http://arxiv.org/abs/2504.19793
Prompt Injection Attack to Tool Selection in LLM Agents. (64%)
Jiawen Shi; Zenghui Yuan; Guiyao Tie; Pan Zhou; Neil Zhenqiang Gong; Lichao Sun
  Tool selection is a key component of LLM agents. The process operates through
a two-step mechanism - \emph{retrieval} and \emph{selection} - to pick the most
appropriate tool from a tool library for a given task. In this work, we
introduce \textit{ToolHijacker}, a novel prompt injection attack targeting tool
selection in no-box scenarios. ToolHijacker injects a malicious tool document
into the tool library to manipulate the LLM agent's tool selection process,
compelling it to consistently choose the attacker's malicious tool for an
attacker-chosen target task. Specifically, we formulate the crafting of such
tool documents as an optimization problem and propose a two-phase optimization
strategy to solve it. Our extensive experimental evaluation shows that
ToolHijacker is highly effective, significantly outperforming existing
manual-based and automated prompt injection attacks when applied to tool
selection. Moreover, we explore various defenses, including prevention-based
defenses (StruQ and SecAlign) and detection-based defenses (known-answer
detection, perplexity detection, and perplexity windowed detection). Our
experimental results indicate that these defenses are insufficient,
highlighting the urgent need for developing new defense strategies.


http://arxiv.org/abs/2504.21038
Prefill-Based Jailbreak: A Novel Approach of Bypassing LLM Safety Boundary. (38%)
Yakai Li; Jiekang Hu; Weiduan Sang; Luping Ma; Jing Xie; Weijuan Zhang; Aimin Yu; Shijie Zhao; Qingjia Huang; Qihang Zhou
  Large Language Models (LLMs) are designed to generate helpful and safe
content. However, adversarial attacks, commonly referred to as jailbreak, can
bypass their safety protocols, prompting LLMs to generate harmful content or
reveal sensitive data. Consequently, investigating jailbreak methodologies is
crucial for exposing systemic vulnerabilities within LLMs, ultimately guiding
the continuous implementation of security enhancements by developers. In this
paper, we introduce a novel jailbreak attack method that leverages the
prefilling feature of LLMs, a feature designed to enhance model output
constraints. Unlike traditional jailbreak methods, the proposed attack
circumvents LLMs' safety mechanisms by directly manipulating the probability
distribution of subsequent tokens, thereby exerting control over the model's
output. We propose two attack variants: Static Prefilling (SP), which employs a
universal prefill text, and Optimized Prefilling (OP), which iteratively
optimizes the prefill text to maximize the attack success rate. Experiments on
six state-of-the-art LLMs using the AdvBench benchmark validate the
effectiveness of our method and demonstrate its capability to substantially
enhance attack success rates when combined with existing jailbreak approaches.
The OP method achieved attack success rates of up to 99.82% on certain models,
significantly outperforming baseline methods. This work introduces a new
jailbreak attack method in LLMs, emphasizing the need for robust content
validation mechanisms to mitigate the adversarial exploitation of prefilling
features. All code and data used in this paper are publicly available.


http://arxiv.org/abs/2504.20245
A Case Study on the Use of Representativeness Bias as a Defense Against Adversarial Cyber Threats. (16%)
Briland Hitaj; Grit Denker; Laura Tinnel; Michael McAnally; Bruce DeBruhl; Nathan Bunting; Alex Fafard; Daniel Aaron; Richard D. Roberts; Joshua Lawson; Greg McCain; Dylan Starink
  Cyberspace is an ever-evolving battleground involving adversaries seeking to
circumvent existing safeguards and defenders aiming to stay one step ahead by
predicting and mitigating the next threat. Existing mitigation strategies have
focused primarily on solutions that consider software or hardware aspects,
often ignoring the human factor. This paper takes a first step towards
psychology-informed, active defense strategies, where we target biases that
human beings are susceptible to under conditions of uncertainty.
  Using capture-the-flag events, we create realistic challenges that tap into a
particular cognitive bias: representativeness. This study finds that this bias
can be triggered to thwart hacking attempts and divert hackers into
non-vulnerable attack paths. Participants were exposed to two different
challenges designed to exploit representativeness biases. One of the
representativeness challenges significantly thwarted attackers away from
vulnerable attack vectors and onto non-vulnerable paths, signifying an
effective bias-based defense mechanism. This work paves the way towards cyber
defense strategies that leverage additional human biases to thwart future,
sophisticated adversarial attacks.


http://arxiv.org/abs/2504.20376
Inception: Jailbreak the Memory Mechanism of Text-to-Image Generation Systems. (13%)
Shiqian Zhao; Jiayang Liu; Yiming Li; Runyi Hu; Xiaojun Jia; Wenshu Fan; Xinfeng Li; Jie Zhang; Wei Dong; Tianwei Zhang; Luu Anh Tuan
  Currently, the memory mechanism has been widely and successfully exploited in
online text-to-image (T2I) generation systems ($e.g.$, DALL$\cdot$E 3) for
alleviating the growing tokenization burden and capturing key information in
multi-turn interactions. Despite its practicality, its security analyses have
fallen far behind. In this paper, we reveal that this mechanism exacerbates the
risk of jailbreak attacks. Different from previous attacks that fuse the unsafe
target prompt into one ultimate adversarial prompt, which can be easily
detected or may generate non-unsafe images due to under- or over-optimization,
we propose Inception, the first multi-turn jailbreak attack against the memory
mechanism in real-world text-to-image generation systems. Inception embeds the
malice at the inception of the chat session turn by turn, leveraging the
mechanism that T2I generation systems retrieve key information in their memory.
Specifically, Inception mainly consists of two modules. It first segments the
unsafe prompt into chunks, which are subsequently fed to the system in multiple
turns, serving as pseudo-gradients for directive optimization. Specifically, we
develop a series of segmentation policies that ensure the images generated are
semantically consistent with the target prompt. Secondly, after segmentation,
to overcome the challenge of the inseparability of minimum unsafe words, we
propose recursion, a strategy that makes minimum unsafe words subdivisible.
Collectively, segmentation and recursion ensure that all the request prompts
are benign but can lead to malicious outcomes. We conduct experiments on the
real-world text-to-image generation system ($i.e.$, DALL$\cdot$E 3) to validate
the effectiveness of Inception. The results indicate that Inception surpasses
the state-of-the-art by a 14\% margin in attack success rate.


http://arxiv.org/abs/2504.20369
Perception-aware Sampling for Scatterplot Visualizations. (5%)
Zafeiria Moumoulidou; Hamza Elhamdadi; Ke Yang; Subrata Mitra; Cindy Xiong Bearfield; Alexandra Meliou
  Visualizing data is often a crucial first step in data analytics workflows,
but growing data sizes pose challenges due to computational and visual
perception limitations. As a result, data analysts commonly down-sample their
data and work with subsets. Deriving representative samples, however, remains a
challenge. This paper focuses on scatterplots, a widely-used visualization
type, and introduces a novel sampling objective -- perception-awareness --
aiming to improve sample efficacy by targeting humans' perception of a
visualization.
  We make the following contributions: (1) We propose perception-augmented
databases and design PAwS: a novel perception-aware sampling method for
scatterplots that leverages saliency maps -- a computer vision tool for
predicting areas of attention focus in visualizations -- and models
perception-awareness via saliency, density, and coverage objectives. (2) We
design ApproPAwS: a fast, perception-aware method for approximate
visualizations, which exploits the fact that small visual perturbations are
often imperceptible to humans. (3) We introduce the concept of perceptual
similarity as a metric for sample quality, and present a novel method that
compares saliency maps to measure it. (4) Our extensive experimental evaluation
shows that our methods consistently outperform prior art in producing samples
with high perceptual similarity, while ApproPAwS achieves up to 100x speed-ups
with minimal loss in visual fidelity. Our user study shows that PAwS is often
preferred by humans, validating our quantitative findings.


http://arxiv.org/abs/2504.21042
What's Pulling the Strings? Evaluating Integrity and Attribution in AI Training and Inference through Concept Shift. (1%)
Jiamin Chang; Haoyang Li; Hammond Pearce; Ruoxi Sun; Bo Li; Minhui Xue
  The growing adoption of artificial intelligence (AI) has amplified concerns
about trustworthiness, including integrity, privacy, robustness, and bias. To
assess and attribute these threats, we propose ConceptLens, a generic framework
that leverages pre-trained multimodal models to identify the root causes of
integrity threats by analyzing Concept Shift in probing samples. ConceptLens
demonstrates strong detection performance for vanilla data poisoning attacks
and uncovers vulnerabilities to bias injection, such as the generation of
covert advertisements through malicious concept shifts. It identifies privacy
risks in unaltered but high-risk samples, filters them before training, and
provides insights into model weaknesses arising from incomplete or imbalanced
training data. Additionally, at the model level, it attributes concepts that
the target model is overly dependent on, identifies misleading concepts, and
explains how disrupting key concepts negatively impacts the model. Furthermore,
it uncovers sociological biases in generative content, revealing disparities
across sociological contexts. Strikingly, ConceptLens reveals how safe training
and inference data can be unintentionally and easily exploited, potentially
undermining safety alignment. Our study informs actionable insights to breed
trust in AI systems, thereby speeding adoption and driving greater innovation.


http://arxiv.org/abs/2504.19529
Adversarial Shallow Watermarking. (1%)
Guobiao Li; Lei Tan; Yuliang Xue; Gaozhi Liu; Zhenxing Qian; Sheng Li; Xinpeng Zhang
  Recent advances in digital watermarking make use of deep neural networks for
message embedding and extraction. They typically follow the ``encoder-noise
layer-decoder''-based architecture. By deliberately establishing a
differentiable noise layer to simulate the distortion of the watermarked
signal, they jointly train the deep encoder and decoder to fit the noise layer
to guarantee robustness. As a result, they are usually weak against unknown
distortions that are not used in their training pipeline. In this paper, we
propose a novel watermarking framework to resist unknown distortions, namely
Adversarial Shallow Watermarking (ASW). ASW utilizes only a shallow decoder
that is randomly parameterized and designed to be insensitive to distortions
for watermarking extraction. During the watermark embedding, ASW freezes the
shallow decoder and adversarially optimizes a host image until its updated
version (i.e., the watermarked image) stably triggers the shallow decoder to
output the watermark message. During the watermark extraction, it accurately
recovers the message from the watermarked image by leveraging the insensitive
nature of the shallow decoder against arbitrary distortions. Our ASW is
training-free, encoder-free, and noise layer-free. Experiments indicate that
the watermarked images created by ASW have strong robustness against various
unknown distortions. Compared to the existing ``encoder-noise layer-decoder''
approaches, ASW achieves comparable results on known distortions and better
robustness on unknown distortions.


http://arxiv.org/abs/2504.19876
DeeCLIP: A Robust and Generalizable Transformer-Based Framework for Detecting AI-Generated Images. (1%)
Mamadou Keita; Wassim Hamidouche; Hessen Bougueffa Eutamene; Abdelmalik Taleb-Ahmed; Abdenour Hadid
  This paper introduces DeeCLIP, a novel framework for detecting AI-generated
images using CLIP-ViT and fusion learning. Despite significant advancements in
generative models capable of creating highly photorealistic images, existing
detection methods often struggle to generalize across different models and are
highly sensitive to minor perturbations. To address these challenges, DeeCLIP
incorporates DeeFuser, a fusion module that combines high-level and low-level
features, improving robustness against degradations such as compression and
blurring. Additionally, we apply triplet loss to refine the embedding space,
enhancing the model's ability to distinguish between real and synthetic
content. To further enable lightweight adaptation while preserving pre-trained
knowledge, we adopt parameter-efficient fine-tuning using low-rank adaptation
(LoRA) within the CLIP-ViT backbone. This approach supports effective zero-shot
learning without sacrificing generalization. Trained exclusively on 4-class
ProGAN data, DeeCLIP achieves an average accuracy of 89.00% on 19 test subsets
composed of generative adversarial network (GAN) and diffusion models. Despite
having fewer trainable parameters, DeeCLIP outperforms existing methods,
demonstrating superior robustness against various generative models and
real-world distortions. The code is publicly available at
https://github.com/Mamadou-Keita/DeeCLIP for research purposes.


http://arxiv.org/abs/2504.20111
Forging and Removing Latent-Noise Diffusion Watermarks Using a Single Image. (87%)
Anubhav Jain; Yuya Kobayashi; Naoki Murata; Yuhta Takida; Takashi Shibuya; Yuki Mitsufuji; Niv Cohen; Nasir Memon; Julian Togelius
  Watermarking techniques are vital for protecting intellectual property and
preventing fraudulent use of media. Most previous watermarking schemes designed
for diffusion models embed a secret key in the initial noise. The resulting
pattern is often considered hard to remove and forge into unrelated images. In
this paper, we propose a black-box adversarial attack without presuming access
to the diffusion model weights. Our attack uses only a single watermarked
example and is based on a simple observation: there is a many-to-one mapping
between images and initial noises. There are regions in the clean image latent
space pertaining to each watermark that get mapped to the same initial noise
when inverted. Based on this intuition, we propose an adversarial attack to
forge the watermark by introducing perturbations to the images such that we can
enter the region of watermarked images. We show that we can also apply a
similar approach for watermark removal by learning perturbations to exit this
region. We report results on multiple watermarking schemes (Tree-Ring, RingID,
WIND, and Gaussian Shading) across two diffusion models (SDv1.4 and SDv2.0).
Our results demonstrate the effectiveness of the attack and expose
vulnerabilities in the watermarking methods, motivating future research on
improving them.


http://arxiv.org/abs/2504.19456
FCGHunter: Towards Evaluating Robustness of Graph-Based Android Malware Detection. (78%)
Shiwen Song; Xiaofei Xie; Ruitao Feng; Qi Guo; Sen Chen
  Graph-based detection methods leveraging Function Call Graphs (FCGs) have
shown promise for Android malware detection (AMD) due to their semantic
insights. However, the deployment of malware detectors in dynamic and hostile
environments raises significant concerns about their robustness. While recent
approaches evaluate the robustness of FCG-based detectors using adversarial
attacks, their effectiveness is constrained by the vast perturbation space,
particularly across diverse models and features.
  To address these challenges, we introduce FCGHunter, a novel robustness
testing framework for FCG-based AMD systems. Specifically, FCGHunter employs
innovative techniques to enhance exploration and exploitation within this huge
search space. Initially, it identifies critical areas within the FCG related to
malware behaviors to narrow down the perturbation space. We then develop a
dependency-aware crossover and mutation method to enhance the validity and
diversity of perturbations, generating diverse FCGs. Furthermore, FCGHunter
leverages multi-objective feedback to select perturbed FCGs, significantly
improving the search process with interpretation-based feature change feedback.
  Extensive evaluations across 40 scenarios demonstrate that FCGHunter achieves
an average attack success rate of 87.9%, significantly outperforming baselines
by at least 44.7%. Notably, FCGHunter achieves a 100% success rate on robust
models (e.g., AdaBoost with MalScan), where baselines achieve only 11% or are
inapplicable.


http://arxiv.org/abs/2504.19212
CapsFake: A Multimodal Capsule Network for Detecting Instruction-Guided Deepfakes. (74%)
Tuan Nguyen; Naseem Khan; Issa Khalil
  The rapid evolution of deepfake technology, particularly in
instruction-guided image editing, threatens the integrity of digital images by
enabling subtle, context-aware manipulations. Generated conditionally from real
images and textual prompts, these edits are often imperceptible to both humans
and existing detection systems, revealing significant limitations in current
defenses. We propose a novel multimodal capsule network, CapsFake, designed to
detect such deepfake image edits by integrating low-level capsules from visual,
textual, and frequency-domain modalities. High-level capsules, predicted
through a competitive routing mechanism, dynamically aggregate local features
to identify manipulated regions with precision. Evaluated on diverse datasets,
including MagicBrush, Unsplash Edits, Open Images Edits, and Multi-turn Edits,
CapsFake outperforms state-of-the-art methods by up to 20% in detection
accuracy. Ablation studies validate its robustness, achieving detection rates
above 94% under natural perturbations and 96% against adversarial attacks, with
excellent generalization to unseen editing scenarios. This approach establishes
a powerful framework for countering sophisticated image manipulations.


http://arxiv.org/abs/2504.19440
JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift. (67%)
Julien Piet; Xiao Huang; Dennis Jacob; Annabella Chow; Maha Alrashed; Geng Zhao; Zhanhao Hu; Chawin Sitawarin; Basel Alomair; David Wagner
  Safety and security remain critical concerns in AI deployment. Despite safety
training through reinforcement learning with human feedback (RLHF) [ 32],
language models remain vulnerable to jailbreak attacks that bypass safety
guardrails. Universal jailbreaks - prefixes that can circumvent alignment for
any payload - are particularly concerning. We show empirically that jailbreak
detection systems face distribution shift, with detectors trained at one point
in time performing poorly against newer exploits. To study this problem, we
release JailbreaksOverTime, a comprehensive dataset of timestamped real user
interactions containing both benign requests and jailbreak attempts collected
over 10 months. We propose a two-pronged method for defenders to detect new
jailbreaks and continuously update their detectors. First, we show how to use
continuous learning to detect jailbreaks and adapt rapidly to new emerging
jailbreaks. While detectors trained at a single point in time eventually fail
due to drift, we find that universal jailbreaks evolve slowly enough for
self-training to be effective. Retraining our detection model weekly using its
own labels - with no new human labels - reduces the false negative rate from 4%
to 0.3% at a false positive rate of 0.1%. Second, we introduce an unsupervised
active monitoring approach to identify novel jailbreaks. Rather than
classifying inputs directly, we recognize jailbreaks by their behavior,
specifically, their ability to trigger models to respond to known-harmful
prompts. This approach has a higher false negative rate (4.1%) than supervised
methods, but it successfully identified some out-of-distribution attacks that
were missed by the continuous learning approach.


http://arxiv.org/abs/2504.19373
Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Models. (22%)
Weidi Luo; Tianyu Lu; Qiming Zhang; Xiaogeng Liu; Bin Hu; Yue Zhao; Jieyu Zhao; Song Gao; Patrick McDaniel; Zhen Xiang; Chaowei Xiao
  Recent advances in multi-modal large reasoning models (MLRMs) have shown
significant ability to interpret complex visual content. While these models
enable impressive reasoning capabilities, they also introduce novel and
underexplored privacy risks. In this paper, we identify a novel category of
privacy leakage in MLRMs: Adversaries can infer sensitive geolocation
information, such as a user's home address or neighborhood, from user-generated
images, including selfies captured in private settings. To formalize and
evaluate these risks, we propose a three-level visual privacy risk framework
that categorizes image content based on contextual sensitivity and potential
for location inference. We further introduce DoxBench, a curated dataset of 500
real-world images reflecting diverse privacy scenarios. Our evaluation across
11 advanced MLRMs and MLLMs demonstrates that these models consistently
outperform non-expert humans in geolocation inference and can effectively leak
location-related private information. This significantly lowers the barrier for
adversaries to obtain users' sensitive geolocation information. We further
analyze and identify two primary factors contributing to this vulnerability:
(1) MLRMs exhibit strong reasoning capabilities by leveraging visual clues in
combination with their internal world knowledge; and (2) MLRMs frequently rely
on privacy-related visual clues for inference without any built-in mechanisms
to suppress or avoid such usage. To better understand and demonstrate
real-world attack feasibility, we propose GeoMiner, a collaborative attack
framework that decomposes the prediction process into two stages: clue
extraction and reasoning to improve geolocation performance while introducing a
novel attack perspective. Our findings highlight the urgent need to reassess
inference-time privacy risks in MLRMs to better protect users' sensitive
information.


http://arxiv.org/abs/2504.19000
Unveiling and Mitigating Adversarial Vulnerabilities in Iterative Optimizers. (98%)
Elad Sofer; Tomer Shaked; Caroline Chaux; Nir Shlezinger
  Machine learning (ML) models are often sensitive to carefully crafted yet
seemingly unnoticeable perturbations. Such adversarial examples are considered
to be a property of ML models, often associated with their black-box operation
and sensitivity to features learned from data. This work examines the
adversarial sensitivity of non-learned decision rules, and particularly of
iterative optimizers. Our analysis is inspired by the recent developments in
deep unfolding, which cast such optimizers as ML models. We show that
non-learned iterative optimizers share the sensitivity to adversarial examples
of ML models, and that attacking iterative optimizers effectively alters the
optimization objective surface in a manner that modifies the minima sought. We
then leverage the ability to cast iteration-limited optimizers as ML models to
enhance robustness via adversarial training. For a class of proximal gradient
optimizers, we rigorously prove how their learning affects adversarial
sensitivity. We numerically back our findings, showing the vulnerability of
various optimizers, as well as the robustness induced by unfolding and
adversarial training.


http://arxiv.org/abs/2504.19019
Graph of Attacks: Improved Black-Box and Interpretable Jailbreaks for LLMs. (92%)
Mohammad Akbar-Tajari; Mohammad Taher Pilehvar; Mohammad Mahmoody
  The challenge of ensuring Large Language Models (LLMs) align with societal
standards is of increasing interest, as these models are still prone to
adversarial jailbreaks that bypass their safety mechanisms. Identifying these
vulnerabilities is crucial for enhancing the robustness of LLMs against such
exploits. We propose Graph of ATtacks (GoAT), a method for generating
adversarial prompts to test the robustness of LLM alignment using the Graph of
Thoughts framework [Besta et al., 2024]. GoAT excels at generating highly
effective jailbreak prompts with fewer queries to the victim model than
state-of-the-art attacks, achieving up to five times better jailbreak success
rate against robust models like Llama. Notably, GoAT creates high-quality,
human-readable prompts without requiring access to the targeted model's
parameters, making it a black-box attack. Unlike approaches constrained by
tree-based reasoning, GoAT's reasoning is based on a more intricate graph
structure. By making simultaneous attack paths aware of each other's progress,
this dynamic framework allows a deeper integration and refinement of reasoning
paths, significantly enhancing the collaborative exploration of adversarial
vulnerabilities in LLMs. At a technical level, GoAT starts with a graph
structure and iteratively refines it by combining and improving thoughts,
enabling synergy between different thought paths. The code for our
implementation can be found at: https://github.com/GoAT-pydev/Graph_of_Attacks.


http://arxiv.org/abs/2504.18827
Test It Before You Trust It: Applying Software Testing for Trustworthy In-context Learning. (86%)
Teeradaj Racharak; Chaiyong Ragkhitwetsagul; Chommakorn Sontesadisai; Thanwadee Sunetnanta
  In-context learning (ICL) has emerged as a powerful capability of large
language models (LLMs), enabling them to perform new tasks based on a few
provided examples without explicit fine-tuning. Despite their impressive
adaptability, these models remain vulnerable to subtle adversarial
perturbations and exhibit unpredictable behavior when faced with linguistic
variations. Inspired by software testing principles, we introduce a software
testing-inspired framework, called MMT4NL, for evaluating the trustworthiness
of in-context learning by utilizing adversarial perturbations and software
testing techniques. It includes diverse evaluation aspects of linguistic
capabilities for testing the ICL capabilities of LLMs. MMT4NL is built around
the idea of crafting metamorphic adversarial examples from a test set in order
to quantify and pinpoint bugs in the designed prompts of ICL. Our philosophy is
to treat any LLM as software and validate its functionalities just like testing
the software. Finally, we demonstrate applications of MMT4NL on the sentiment
analysis and question-answering tasks. Our experiments could reveal various
linguistic bugs in state-of-the-art LLMs.


http://arxiv.org/abs/2504.18872
Latent Adversarial Training Improves the Representation of Refusal. (10%)
Alexandra Abbas; Nora Petrova; Helios Ael Lyons; Natalia Perez-Campanero
  Recent work has shown that language models' refusal behavior is primarily
encoded in a single direction in their latent space, making it vulnerable to
targeted attacks. Although Latent Adversarial Training (LAT) attempts to
improve robustness by introducing noise during training, a key question
remains: How does this noise-based training affect the underlying
representation of refusal behavior? Understanding this encoding is crucial for
evaluating LAT's effectiveness and limitations, just as the discovery of linear
refusal directions revealed vulnerabilities in traditional supervised safety
fine-tuning (SSFT).
  Through the analysis of Llama 2 7B, we examine how LAT reorganizes the
refusal behavior in the model's latent space compared to SSFT and embedding
space adversarial training (AT). By computing activation differences between
harmful and harmless instruction pairs and applying Singular Value
Decomposition (SVD), we find that LAT significantly alters the refusal
representation, concentrating it in the first two SVD components which explain
approximately 75 percent of the activation differences variance - significantly
higher than in reference models. This concentrated representation leads to more
effective and transferable refusal vectors for ablation attacks: LAT models
show improved robustness when attacked with vectors from reference models but
become more vulnerable to self-generated vectors compared to SSFT and AT. Our
findings suggest that LAT's training perturbations enable a more comprehensive
representation of refusal behavior, highlighting both its potential strengths
and vulnerabilities for improving model safety.


http://arxiv.org/abs/2504.19027
DiCE-Extended: A Robust Approach to Counterfactual Explanations in Machine Learning. (3%)
Volkan Bakir; Polat Goktas; Sureyya Akyuz
  Explainable artificial intelligence (XAI) has become increasingly important
in decision-critical domains such as healthcare, finance, and law.
Counterfactual (CF) explanations, a key approach in XAI, provide users with
actionable insights by suggesting minimal modifications to input features that
lead to different model outcomes. Despite significant advancements, existing CF
generation methods often struggle to balance proximity, diversity, and
robustness, limiting their real-world applicability. A widely adopted
framework, Diverse Counterfactual Explanations (DiCE), emphasizes diversity but
lacks robustness, making CF explanations sensitive to perturbations and domain
constraints. To address these challenges, we introduce DiCE-Extended, an
enhanced CF explanation framework that integrates multi-objective optimization
techniques to improve robustness while maintaining interpretability. Our
approach introduces a novel robustness metric based on the Dice-Sorensen
coefficient, ensuring stability under small input variations. Additionally, we
refine CF generation using weighted loss components (lambda_p, lambda_d,
lambda_r) to balance proximity, diversity, and robustness. We empirically
validate DiCE-Extended on benchmark datasets (COMPAS, Lending Club, German
Credit, Adult Income) across multiple ML backends (Scikit-learn, PyTorch,
TensorFlow). Results demonstrate improved CF validity, stability, and alignment
with decision boundaries compared to standard DiCE-generated explanations. Our
findings highlight the potential of DiCE-Extended in generating more reliable
and interpretable CFs for high-stakes applications. Future work will explore
adaptive optimization techniques and domain-specific constraints to further
enhance CF generation in real-world scenarios.


http://arxiv.org/abs/2504.18990
Safety Interventions against Adversarial Patches in an Open-Source Driver Assistance System. (1%)
Cheng Chen; Grant Xiao; Daehyun Lee; Lishan Yang; Evgenia Smirni; Homa Alemzadeh; Xugui Zhou
  Drivers are becoming increasingly reliant on advanced driver assistance
systems (ADAS) as autonomous driving technology becomes more popular and
developed with advanced safety features to enhance road safety. However, the
increasing complexity of the ADAS makes autonomous vehicles (AVs) more exposed
to attacks and accidental faults. In this paper, we evaluate the resilience of
a widely used ADAS against safety-critical attacks that target perception
inputs. Various safety mechanisms are simulated to assess their impact on
mitigating attacks and enhancing ADAS resilience. Experimental results
highlight the importance of timely intervention by human drivers and automated
safety mechanisms in preventing accidents in both driving and lateral
directions and the need to resolve conflicts among safety interventions to
enhance system resilience and reliability.


http://arxiv.org/abs/2504.20077
Edge-Based Learning for Improved Classification Under Adversarial Noise. (99%)
Manish Kansana; Keyan Alexander Rahimi; Elias Hossain; Iman Dehzangi; Noorbakhsh Amiri Golilarz
  Adversarial noise introduces small perturbations in images, misleading deep
learning models into misclassification and significantly impacting recognition
accuracy. In this study, we analyzed the effects of Fast Gradient Sign Method
(FGSM) adversarial noise on image classification and investigated whether
training on specific image features can improve robustness. We hypothesize that
while adversarial noise perturbs various regions of an image, edges may remain
relatively stable and provide essential structural information for
classification. To test this, we conducted a series of experiments using brain
tumor and COVID datasets. Initially, we trained the models on clean images and
then introduced subtle adversarial perturbations, which caused deep learning
models to significantly misclassify the images. Retraining on a combination of
clean and noisy images led to improved performance. To evaluate the robustness
of the edge features, we extracted edges from the original/clean images and
trained the models exclusively on edge-based representations. When noise was
introduced to the images, the edge-based models demonstrated greater resilience
to adversarial attacks compared to those trained on the original or clean
images. These results suggest that while adversarial noise is able to exploit
complex non-edge regions significantly more than edges, the improvement in the
accuracy after retraining is marginally more in the original data as compared
to the edges. Thus, leveraging edge-based learning can improve the resilience
of deep learning models against adversarial perturbations.


http://arxiv.org/abs/2504.18519
Intelligent Attacks and Defense Methods in Federated Learning-enabled Energy-Efficient Wireless Networks. (74%)
Han Zhang; Hao Zhou; Medhat Elsayed; Majid Bavand; Raimundas Gaigalas; Yigit Ozcan; Melike Erol-Kantarci
  Federated learning (FL) is a promising technique for learning-based functions
in wireless networks, thanks to its distributed implementation capability. On
the other hand, distributed learning may increase the risk of exposure to
malicious attacks where attacks on a local model may spread to other models by
parameter exchange. Meanwhile, such attacks can be hard to detect due to the
dynamic wireless environment, especially considering local models can be
heterogeneous with non-independent and identically distributed (non-IID) data.
Therefore, it is critical to evaluate the effect of malicious attacks and
develop advanced defense techniques for FL-enabled wireless networks. In this
work, we introduce a federated deep reinforcement learning-based cell sleep
control scenario that enhances the energy efficiency of the network. We propose
multiple intelligent attacks targeting the learning-based approach and we
propose defense methods to mitigate such attacks. In particular, we have
designed two attack models, generative adversarial network (GAN)-enhanced model
poisoning attack and regularization-based model poisoning attack. As a
counteraction, we have proposed two defense schemes, autoencoder-based defense,
and knowledge distillation (KD)-enabled defense. The autoencoder-based defense
method leverages an autoencoder to identify the malicious participants and only
aggregate the parameters of benign local models during the global aggregation,
while KD-based defense protects the model from attacks by controlling the
knowledge transferred between the global model and local models.


http://arxiv.org/abs/2504.18497
DeSIA: Attribute Inference Attacks Against Limited Fixed Aggregate Statistics. (1%)
Yifeng Mao; Bozhidar Stevanoski; Montjoye Yves-Alexandre de
  Empirical inference attacks are a popular approach for evaluating the privacy
risk of data release mechanisms in practice. While an active attack literature
exists to evaluate machine learning models or synthetic data release, we
currently lack comparable methods for fixed aggregate statistics, in particular
when only a limited number of statistics are released. We here propose an
inference attack framework against fixed aggregate statistics and an attribute
inference attack called DeSIA. We instantiate DeSIA against the U.S. Census
PPMF dataset and show it to strongly outperform reconstruction-based attacks.
In particular, we show DeSIA to be highly effective at identifying vulnerable
users, achieving a true positive rate of 0.14 at a false positive rate of
$10^{-3}$. We then show DeSIA to perform well against users whose attributes
cannot be verified and when varying the number of aggregate statistics and
level of noise addition. We also perform an extensive ablation study of DeSIA
and show how DeSIA can be successfully adapted to the membership inference
task. Overall, our results show that aggregation alone is not sufficient to
protect privacy, even when a relatively small number of aggregates are being
released, and emphasize the need for formal privacy mechanisms and testing
before aggregate statistics are released.


http://arxiv.org/abs/2504.17884
Unsupervised Corpus Poisoning Attacks in Continuous Space for Dense Retrieval. (99%)
Yongkang Li; Panagiotis Eustratiadis; Simon Lupart; Evangelos Kanoulas
  This paper concerns corpus poisoning attacks in dense information retrieval,
where an adversary attempts to compromise the ranking performance of a search
algorithm by injecting a small number of maliciously generated documents into
the corpus. Our work addresses two limitations in the current literature.
First, attacks that perform adversarial gradient-based word substitution search
do so in the discrete lexical space, while retrieval itself happens in the
continuous embedding space. We thus propose an optimization method that
operates in the embedding space directly. Specifically, we train a perturbation
model with the objective of maintaining the geometric distance between the
original and adversarial document embeddings, while also maximizing the
token-level dissimilarity between the original and adversarial documents.
Second, it is common for related work to have a strong assumption that the
adversary has prior knowledge about the queries. In this paper, we focus on a
more challenging variant of the problem where the adversary assumes no prior
knowledge about the query distribution (hence, unsupervised). Our core
contribution is an adversarial corpus attack that is fast and effective. We
present comprehensive experimental results on both in- and out-of-domain
datasets, focusing on two related tasks: a top-1 attack and a corpus poisoning
attack. We consider attacks under both a white-box and a black-box setting.
Notably, our method can generate successful adversarial examples in under two
minutes per target document; four times faster compared to the fastest
gradient-based word substitution methods in the literature with the same
hardware. Furthermore, our adversarial generation method generates text that is
more likely to occur under the distribution of natural text (low perplexity),
and is therefore more difficult to detect.


http://arxiv.org/abs/2504.18594
A Simple DropConnect Approach to Transfer-based Targeted Attack. (99%)
Tongrui Su; Qingbin Li; Shengyu Zhu; Wei Chen; Xueqi Cheng
  We study the problem of transfer-based black-box attack, where adversarial
samples generated using a single surrogate model are directly applied to target
models. Compared with untargeted attacks, existing methods still have lower
Attack Success Rates (ASRs) in the targeted setting, i.e., the obtained
adversarial examples often overfit the surrogate model but fail to mislead
other models. In this paper, we hypothesize that the pixels or features in
these adversarial examples collaborate in a highly dependent manner to maximize
the success of an adversarial attack on the surrogate model, which we refer to
as perturbation co-adaptation. Then, we propose to Mitigate perturbation
Co-adaptation by DropConnect (MCD) to enhance transferability, by creating
diverse variants of surrogate model at each optimization iteration. We conduct
extensive experiments across various CNN- and Transformer-based models to
demonstrate the effectiveness of MCD. In the challenging scenario of
transferring from a CNN-based model to Transformer-based models, MCD achieves
13% higher average ASRs compared with state-of-the-art baselines. MCD boosts
the performance of self-ensemble methods by bringing in more diversification
across the variants while reserving sufficient semantic information for each
variant. In addition, MCD attains the highest performance gain when scaling the
compute of crafting adversarial examples.


http://arxiv.org/abs/2504.17457
Unveiling Hidden Vulnerabilities in Digital Human Generation via Adversarial Attacks. (98%)
Zhiying Li; Yeying Jin; Fan Shen; Zhi Liu; Weibin Chen; Pengju Zhang; Xiaomei Zhang; Boyu Chen; Michael Shen; Kejian Wu; Zhaoxin Fan; Jin Dong
  Expressive human pose and shape estimation (EHPS) is crucial for digital
human generation, especially in applications like live streaming. While
existing research primarily focuses on reducing estimation errors, it largely
neglects robustness and security aspects, leaving these systems vulnerable to
adversarial attacks. To address this significant challenge, we propose the
\textbf{Tangible Attack (TBA)}, a novel framework designed to generate
adversarial examples capable of effectively compromising any digital human
generation model. Our approach introduces a \textbf{Dual Heterogeneous Noise
Generator (DHNG)}, which leverages Variational Autoencoders (VAE) and
ControlNet to produce diverse, targeted noise tailored to the original image
features. Additionally, we design a custom \textbf{adversarial loss function}
to optimize the noise, ensuring both high controllability and potent
disruption. By iteratively refining the adversarial sample through
multi-gradient signals from both the noise and the state-of-the-art EHPS model,
TBA substantially improves the effectiveness of adversarial attacks. Extensive
experiments demonstrate TBA's superiority, achieving a remarkable 41.0\%
increase in estimation error, with an average improvement of approximately
17.0\%. These findings expose significant security vulnerabilities in current
EHPS models and highlight the need for stronger defenses in digital human
generation systems.


http://arxiv.org/abs/2504.17684
Evaluating the Vulnerability of ML-Based Ethereum Phishing Detectors to Single-Feature Adversarial Perturbations. (97%)
Ahod Alghuried; Ali Alkinoon; Abdulaziz Alghamdi; Soohyeon Choi; Manar Mohaisen; David Mohaisen
  This paper explores the vulnerability of machine learning models to simple
single-feature adversarial attacks in the context of Ethereum fraudulent
transaction detection. Through comprehensive experimentation, we investigate
the impact of various adversarial attack strategies on model performance
metrics. Our findings, highlighting how prone those techniques are to simple
attacks, are alarming, and the inconsistency in the attacks' effect on
different algorithms promises ways for attack mitigation. We examine the
effectiveness of different mitigation strategies, including adversarial
training and enhanced feature selection, in enhancing model robustness and show
their effectiveness.


http://arxiv.org/abs/2504.17829
Fine-Tuning Adversarially-Robust Transformers for Single-Image Dehazing. (92%)
Vlad Vasilescu; Ana Neacsu; Daniela Faur
  Single-image dehazing is an important topic in remote sensing applications,
enhancing the quality of acquired images and increasing object detection
precision. However, the reliability of such structures has not been
sufficiently analyzed, which poses them to the risk of imperceptible
perturbations that can significantly hinder their performance. In this work, we
show that state-of-the-art image-to-image dehazing transformers are susceptible
to adversarial noise, with even 1 pixel change being able to decrease the PSNR
by as much as 2.8 dB. Next, we propose two lightweight fine-tuning strategies
aimed at increasing the robustness of pre-trained transformers. Our methods
results in comparable clean performance, while significantly increasing the
protection against adversarial data. We further present their applicability in
two remote sensing scenarios, showcasing their robust behavior for
out-of-distribution data. The source code for adversarial fine-tuning and
attack algorithms can be found at github.com/Vladimirescu/RobustDehazing.


http://arxiv.org/abs/2504.17723
Towards Robust LLMs: an Adversarial Robustness Measurement Framework. (87%)
Natan Levy; Adiel Ashrov; Guy Katz
  The rise of Large Language Models (LLMs) has revolutionized artificial
intelligence, yet these models remain vulnerable to adversarial perturbations,
undermining their reliability in high-stakes applications. While adversarial
robustness in vision-based neural networks has been extensively studied, LLM
robustness remains under-explored. We adapt the Robustness Measurement and
Assessment (RoMA) framework to quantify LLM resilience against adversarial
inputs without requiring access to model parameters. By comparing RoMA's
estimates to those of formal verification methods, we demonstrate its accuracy
with minimal error margins while maintaining computational efficiency. Our
empirical evaluation reveals that robustness varies significantly not only
between different models but also across categories within the same task and
between various types of perturbations. This non-uniformity underscores the
need for task-specific robustness evaluations, enabling practitioners to
compare and select models based on application-specific robustness
requirements. Our work provides a systematic methodology to assess LLM
robustness, advancing the development of more reliable language models for
real-world deployment.


http://arxiv.org/abs/2504.17690
On the Generalization of Adversarially Trained Quantum Classifiers. (82%)
Petros Georgiou; Aaron Mark Thomas; Sharu Theresa Jose; Osvaldo Simeone
  Quantum classifiers are vulnerable to adversarial attacks that manipulate
their input classical or quantum data. A promising countermeasure is
adversarial training, where quantum classifiers are trained by using an
attack-aware, adversarial loss function. This work establishes novel bounds on
the generalization error of adversarially trained quantum classifiers when
tested in the presence of perturbation-constrained adversaries. The bounds
quantify the excess generalization error incurred to ensure robustness to
adversarial attacks as scaling with the training sample size $m$ as
$1/\sqrt{m}$, while yielding insights into the impact of the quantum embedding.
For quantum binary classifiers employing \textit{rotation embedding}, we find
that, in the presence of adversarial attacks on classical inputs $\mathbf{x}$,
the increase in sample complexity due to adversarial training over conventional
training vanishes in the limit of high dimensional inputs $\mathbf{x}$. In
contrast, when the adversary can directly attack the quantum state
$\rho(\mathbf{x})$ encoding the input $\mathbf{x}$, the excess generalization
error depends on the choice of embedding only through its Hilbert space
dimension. The results are also extended to multi-class classifiers. We
validate our theoretical findings with numerical experiments.


http://arxiv.org/abs/2504.17894
DCT-Shield: A Robust Frequency Domain Defense against Malicious Image Editing. (75%)
Aniruddha Bala; Rohit Chowdhury; Rohan Jaiswal; Siddharth Roheda
  Advancements in diffusion models have enabled effortless image editing via
text prompts, raising concerns about image security. Attackers with access to
user images can exploit these tools for malicious edits. Recent defenses
attempt to protect images by adding a limited noise in the pixel space to
disrupt the functioning of diffusion-based editing models. However, the
adversarial noise added by previous methods is easily noticeable to the human
eye. Moreover, most of these methods are not robust to purification techniques
like JPEG compression under a feasible pixel budget. We propose a novel
optimization approach that introduces adversarial perturbations directly in the
frequency domain by modifying the Discrete Cosine Transform (DCT) coefficients
of the input image. By leveraging the JPEG pipeline, our method generates
adversarial images that effectively prevent malicious image editing. Extensive
experiments across a variety of tasks and datasets demonstrate that our
approach introduces fewer visual artifacts while maintaining similar levels of
edit protection and robustness to noise purification techniques.


http://arxiv.org/abs/2504.17300
The Ultimate Cookbook for Invisible Poison: Crafting Subtle Clean-Label Text Backdoors with Style Attributes. (69%)
Wencong You; Daniel Lowd
  Backdoor attacks on text classifiers can cause them to predict a predefined
label when a particular "trigger" is present. Prior attacks often rely on
triggers that are ungrammatical or otherwise unusual, leading to conspicuous
attacks. As a result, human annotators, who play a critical role in curating
training data in practice, can easily detect and filter out these unnatural
texts during manual inspection, reducing the risk of such attacks. We argue
that a key criterion for a successful attack is for text with and without
triggers to be indistinguishable to humans. However, prior work neither
directly nor comprehensively evaluated attack subtlety and invisibility with
human involvement. We bridge the gap by conducting thorough human evaluations
to assess attack subtlety. We also propose \emph{AttrBkd}, consisting of three
recipes for crafting subtle yet effective trigger attributes, such as
extracting fine-grained attributes from existing baseline backdoor attacks. Our
human evaluations find that AttrBkd with these baseline-derived attributes is
often more effective (higher attack success rate) and more subtle (fewer
instances detected by humans) than the original baseline backdoor attacks,
demonstrating that backdoor attacks can bypass detection by being inconspicuous
and appearing natural even upon close inspection, while still remaining
effective. Our human annotation also provides information not captured by
automated metrics used in prior work, and demonstrates the misalignment of
these metrics with human judgment.


http://arxiv.org/abs/2504.17461
Evaluating Time Series Models for Urban Wastewater Management: Predictive Performance, Model Complexity and Resilience. (4%)
Vipin Singh; Tianheng Ling; Teodor Chiaburu; Felix Biessmann
  Climate change increases the frequency of extreme rainfall, placing a
significant strain on urban infrastructures, especially Combined Sewer Systems
(CSS). Overflows from overburdened CSS release untreated wastewater into
surface waters, posing environmental and public health risks. Although
traditional physics-based models are effective, they are costly to maintain and
difficult to adapt to evolving system dynamics. Machine Learning (ML)
approaches offer cost-efficient alternatives with greater adaptability. To
systematically assess the potential of ML for modeling urban infrastructure
systems, we propose a protocol for evaluating Neural Network architectures for
CSS time series forecasting with respect to predictive performance, model
complexity, and robustness to perturbations. In addition, we assess model
performance on peak events and critical fluctuations, as these are the key
regimes for urban wastewater management. To investigate the feasibility of
lightweight models suitable for IoT deployment, we compare global models, which
have access to all information, with local models, which rely solely on nearby
sensor readings. Additionally, to explore the security risks posed by network
outages or adversarial attacks on urban infrastructure, we introduce error
models that assess the resilience of models. Our results demonstrate that while
global models achieve higher predictive performance, local models provide
sufficient resilience in decentralized scenarios, ensuring robust modeling of
urban infrastructure. Furthermore, models with longer native forecast horizons
exhibit greater robustness to data perturbations. These findings contribute to
the development of interpretable and reliable ML solutions for sustainable
urban wastewater management. The implementation is available in our GitHub
repository.


http://arxiv.org/abs/2504.17971
Cluster-Aware Attacks on Graph Watermarks. (2%)
Alexander Nemecek; Emre Yilmaz; Erman Ayday
  Data from domains such as social networks, healthcare, finance, and
cybersecurity can be represented as graph-structured information. Given the
sensitive nature of this data and their frequent distribution among
collaborators, ensuring secure and attributable sharing is essential. Graph
watermarking enables attribution by embedding user-specific signatures into
graph-structured data. While prior work has addressed random perturbation
attacks, the threat posed by adversaries leveraging structural properties
through community detection remains unexplored. In this work, we introduce a
cluster-aware threat model in which adversaries apply community-guided
modifications to evade detection. We propose two novel attack strategies and
evaluate them on real-world social network graphs. Our results show that
cluster-aware attacks can reduce attribution accuracy by up to 80% more than
random baselines under equivalent perturbation budgets on sparse graphs. To
mitigate this threat, we propose a lightweight embedding enhancement that
distributes watermark nodes across graph communities. This approach improves
attribution accuracy by up to 60% under attack on dense graphs, without
increasing runtime or structural distortion. Our findings underscore the
importance of cluster-topological awareness in both watermarking design and
adversarial modeling.


http://arxiv.org/abs/2504.17311
FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation. (1%)
Yulia Otmakhova; Hung Thinh Truong; Rahmad Mahendra; Zenan Zhai; Rongxin Zhu; Daniel Beck; Jey Han Lau
  We present FLUKE (Framework for LingUistically-driven and tasK-agnostic
robustness Evaluation), a task-agnostic framework for assessing model
robustness through systematic minimal variations of test data. FLUKE introduces
controlled variations across linguistic levels - from orthography to dialect
and style varieties - and leverages large language models (LLMs) with human
validation to generate modifications. We demonstrate FLUKE's utility by
evaluating both fine-tuned models and LLMs across four diverse NLP tasks, and
reveal that (1) the impact of linguistic variations is highly task-dependent,
with some tests being critical for certain tasks but irrelevant for others; (2)
while LLMs have better overall robustness compared to fine-tuned models, they
still exhibit significant brittleness to certain linguistic variations; (3) all
models show substantial vulnerability to negation modifications across most
tasks. These findings highlight the importance of systematic robustness testing
for understanding model behaviors.


http://arxiv.org/abs/2504.18598
BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Experts. (1%)
Qingyue Wang; Qi Pang; Xixun Lin; Shuai Wang; Daoyuan Wu
  Mixture-of-Experts (MoE) have emerged as a powerful architecture for large
language models (LLMs), enabling efficient scaling of model capacity while
maintaining manageable computational costs. The key advantage lies in their
ability to route different tokens to different ``expert'' networks within the
model, enabling specialization and efficient handling of diverse input.
However, the vulnerabilities of MoE-based LLMs still have barely been studied,
and the potential for backdoor attacks in this context remains largely
unexplored. This paper presents the first backdoor attack against MoE-based
LLMs where the attackers poison ``dormant experts'' (i.e., underutilized
experts) and activate them by optimizing routing triggers, thereby gaining
control over the model's output. We first rigorously prove the existence of a
few ``dominating experts'' in MoE models, whose outputs can determine the
overall MoE's output. We also show that dormant experts can serve as dominating
experts to manipulate model predictions. Accordingly, our attack, namely
BadMoE, exploits the unique architecture of MoE models by 1) identifying
dormant experts unrelated to the target task, 2) constructing a routing-aware
loss to optimize the activation triggers of these experts, and 3) promoting
dormant experts to dominating roles via poisoned training data. Extensive
experiments show that BadMoE successfully enforces malicious prediction on
attackers' target tasks while preserving overall model utility, making it a
more potent and stealthy attack than existing methods.


http://arxiv.org/abs/2504.16474
Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: A Theoretical Framework and Algorithmic Instantiation. (99%)
Meixi Zheng; Kehan Wu; Yanbo Fan; Rui Huang; Baoyuan Wu
  The transfer-based black-box adversarial attack setting poses the challenge
of crafting an adversarial example (AE) on known surrogate models that remain
effective against unseen target models. Due to the practical importance of this
task, numerous methods have been proposed to address this challenge. However,
most previous methods are heuristically designed and intuitively justified,
lacking a theoretical foundation. To bridge this gap, we derive a novel
transferability bound that offers provable guarantees for adversarial
transferability. Our theoretical analysis has the advantages of \textit{(i)}
deepening our understanding of previous methods by building a general attack
framework and \textit{(ii)} providing guidance for designing an effective
attack algorithm. Our theoretical results demonstrate that optimizing AEs
toward flat minima over the surrogate model set, while controlling the
surrogate-target model shift measured by the adversarial model discrepancy,
yields a comprehensive guarantee for AE transferability. The results further
lead to a general transfer-based attack framework, within which we observe that
previous methods consider only partial factors contributing to the
transferability. Algorithmically, inspired by our theoretical results, we first
elaborately construct the surrogate model set in which models exhibit diverse
adversarial vulnerabilities with respect to AEs to narrow an instantiated
adversarial model discrepancy. Then, a \textit{model-Diversity-compatible
Reverse Adversarial Perturbation} (DRAP) is generated to effectively promote
the flatness of AEs over diverse surrogate models to improve transferability.
Extensive experiments on NIPS2017 and CIFAR-10 datasets against various target
models demonstrate the effectiveness of our proposed attack.


http://arxiv.org/abs/2504.16907
BadVideo: Stealthy Backdoor Attack against Text-to-Video Generation. (61%)
Ruotong Wang; Mingli Zhu; Jiarong Ou; Rui Chen; Xin Tao; Pengfei Wan; Baoyuan Wu
  Text-to-video (T2V) generative models have rapidly advanced and found
widespread applications across fields like entertainment, education, and
marketing. However, the adversarial vulnerabilities of these models remain
rarely explored. We observe that in T2V generation tasks, the generated videos
often contain substantial redundant information not explicitly specified in the
text prompts, such as environmental elements, secondary objects, and additional
details, providing opportunities for malicious attackers to embed hidden
harmful content. Exploiting this inherent redundancy, we introduce BadVideo,
the first backdoor attack framework tailored for T2V generation. Our attack
focuses on designing target adversarial outputs through two key strategies: (1)
Spatio-Temporal Composition, which combines different spatiotemporal features
to encode malicious information; (2) Dynamic Element Transformation, which
introduces transformations in redundant elements over time to convey malicious
information. Based on these strategies, the attacker's malicious target
seamlessly integrates with the user's textual instructions, providing high
stealthiness. Moreover, by exploiting the temporal dimension of videos, our
attack successfully evades traditional content moderation systems that
primarily analyze spatial information within individual frames. Extensive
experiments demonstrate that BadVideo achieves high attack success rates while
preserving original semantics and maintaining excellent performance on clean
inputs. Overall, our work reveals the adversarial vulnerability of T2V models,
calling attention to potential risks and misuse. Our project page is at
https://wrt2000.github.io/BadVideo2025/.


http://arxiv.org/abs/2504.17219
Enhancing Variational Autoencoders with Smooth Robust Latent Encoding. (50%)
Hyomin Lee; Minseon Kim; Sangwon Jang; Jongheon Jeong; Sung Ju Hwang
  Variational Autoencoders (VAEs) have played a key role in scaling up
diffusion-based generative models, as in Stable Diffusion, yet questions
regarding their robustness remain largely underexplored. Although adversarial
training has been an established technique for enhancing robustness in
predictive models, it has been overlooked for generative models due to concerns
about potential fidelity degradation by the nature of trade-offs between
performance and robustness. In this work, we challenge this presumption,
introducing Smooth Robust Latent VAE (SRL-VAE), a novel adversarial training
framework that boosts both generation quality and robustness. In contrast to
conventional adversarial training, which focuses on robustness only, our
approach smooths the latent space via adversarial perturbations, promoting more
generalizable representations while regularizing with originality
representation to sustain original fidelity. Applied as a post-training step on
pre-trained VAEs, SRL-VAE improves image robustness and fidelity with minimal
computational overhead. Experiments show that SRL-VAE improves both generation
quality, in image reconstruction and text-guided image editing, and robustness,
against Nightshade attacks and image editing attacks. These results establish a
new paradigm, showing that adversarial training, once thought to be detrimental
to generative models, can instead enhance both fidelity and robustness.


http://arxiv.org/abs/2504.17179
AUTHENTICATION: Identifying Rare Failure Modes in Autonomous Vehicle Perception Systems using Adversarially Guided Diffusion Models. (1%)
Mohammad Zarei; Melanie A Jutras; Eliana Evans; Mike Tan; Omid Aaramoon
  Autonomous Vehicles (AVs) rely on artificial intelligence (AI) to accurately
detect objects and interpret their surroundings. However, even when trained
using millions of miles of real-world data, AVs are often unable to detect rare
failure modes (RFMs). The problem of RFMs is commonly referred to as the
"long-tail challenge", due to the distribution of data including many instances
that are very rarely seen. In this paper, we present a novel approach that
utilizes advanced generative and explainable AI techniques to aid in
understanding RFMs. Our methods can be used to enhance the robustness and
reliability of AVs when combined with both downstream model training and
testing. We extract segmentation masks for objects of interest (e.g., cars) and
invert them to create environmental masks. These masks, combined with carefully
crafted text prompts, are fed into a custom diffusion model. We leverage the
Stable Diffusion inpainting model guided by adversarial noise optimization to
generate images containing diverse environments designed to evade object
detection models and expose vulnerabilities in AI systems. Finally, we produce
natural language descriptions of the generated RFMs that can guide developers
and policymakers to improve the safety and reliability of AV systems.


http://arxiv.org/abs/2504.15823
Human-Imperceptible Physical Adversarial Attack for NIR Face Recognition Models. (99%)
Songyan Xie; Jinghang Wen; Encheng Su; Qiucheng Yu
  Near-infrared (NIR) face recognition systems, which can operate effectively
in low-light conditions or in the presence of makeup, exhibit vulnerabilities
when subjected to physical adversarial attacks. To further demonstrate the
potential risks in real-world applications, we design a novel, stealthy, and
practical adversarial patch to attack NIR face recognition systems in a
black-box setting. We achieved this by utilizing human-imperceptible
infrared-absorbing ink to generate multiple patches with digitally optimized
shapes and positions for infrared images. To address the optimization mismatch
between digital and real-world NIR imaging, we develop a light reflection model
for human skin to minimize pixel-level discrepancies by simulating NIR light
reflection.
  Compared to state-of-the-art (SOTA) physical attacks on NIR face recognition
systems, the experimental results show that our method improves the attack
success rate in both digital and physical domains, particularly maintaining
effectiveness across various face postures. Notably, the proposed approach
outperforms SOTA methods, achieving an average attack success rate of 82.46% in
the physical domain across different models, compared to 64.18% for existing
methods. The artifact is available at
https://anonymous.4open.science/r/Human-imperceptible-adversarial-patch-0703/.


http://arxiv.org/abs/2504.16355
Property-Preserving Hashing for $\ell_1$-Distance Predicates: Applications to Countering Adversarial Input Attacks. (75%)
Hassan Asghar; Chenhan Zhang; Dali Kaafar
  Perceptual hashing is used to detect whether an input image is similar to a
reference image with a variety of security applications. Recently, they have
been shown to succumb to adversarial input attacks which make small
imperceptible changes to the input image yet the hashing algorithm does not
detect its similarity to the original image. Property-preserving hashing (PPH)
is a recent construct in cryptography, which preserves some property
(predicate) of its inputs in the hash domain. Researchers have so far shown
constructions of PPH for Hamming distance predicates, which, for instance,
outputs 1 if two inputs are within Hamming distance $t$. A key feature of PPH
is its strong correctness guarantee, i.e., the probability that the predicate
will not be correctly evaluated in the hash domain is negligible. Motivated by
the use case of detecting similar images under adversarial setting, we propose
the first PPH construction for an $\ell_1$-distance predicate. Roughly, this
predicate checks if the two one-sided $\ell_1$-distances between two images are
within a threshold $t$. Since many adversarial attacks use $\ell_2$-distance
(related to $\ell_1$-distance) as the objective function to perturb the input
image, by appropriately choosing the threshold $t$, we can force the attacker
to add considerable noise to evade detection, and hence significantly
deteriorate the image quality. Our proposed scheme is highly efficient, and
runs in time $O(t^2)$. For grayscale images of size $28 \times 28$, we can
evaluate the predicate in $0.0784$ seconds when pixel values are perturbed by
up to $1 \%$. For larger RGB images of size $224 \times 224$, by dividing the
image into 1,000 blocks, we achieve times of $0.0128$ seconds per block for $1
\%$ change, and up to $0.2641$ seconds per block for $14\%$ change.


http://arxiv.org/abs/2504.15674
TrojanDam: Detection-Free Backdoor Defense in Federated Learning through Proactive Model Robustification utilizing OOD Data. (56%)
Yanbo Dai; Songze Li; Zihan Gan; Xueluan Gong
  Federated learning (FL) systems allow decentralized data-owning clients to
jointly train a global model through uploading their locally trained updates to
a centralized server. The property of decentralization enables adversaries to
craft carefully designed backdoor updates to make the global model misclassify
only when encountering adversary-chosen triggers. Existing defense mechanisms
mainly rely on post-training detection after receiving updates. These methods
either fail to identify updates which are deliberately fabricated statistically
close to benign ones, or show inconsistent performance in different FL training
stages. The effect of unfiltered backdoor updates will accumulate in the global
model, and eventually become functional. Given the difficulty of ruling out
every backdoor update, we propose a backdoor defense paradigm, which focuses on
proactive robustification on the global model against potential backdoor
attacks. We first reveal that the successful launching of backdoor attacks in
FL stems from the lack of conflict between malicious and benign updates on
redundant neurons of ML models. We proceed to prove the feasibility of
activating redundant neurons utilizing out-of-distribution (OOD) samples in
centralized settings, and migrating to FL settings to propose a novel backdoor
defense mechanism, TrojanDam. The proposed mechanism has the FL server
continuously inject fresh OOD mappings into the global model to activate
redundant neurons, canceling the effect of backdoor updates during aggregation.
We conduct systematic and extensive experiments to illustrate the superior
performance of TrojanDam, over several SOTA backdoor defense methods across a
wide range of FL settings.


http://arxiv.org/abs/2504.18575
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks. (4%)
Ivan Evtimov; Arman Zharmagambetov; Aaron Grattafiori; Chuan Guo; Kamalika Chaudhuri
  Web navigation AI agents use language-and-vision foundation models to enhance
productivity but these models are known to be susceptible to indirect prompt
injections that get them to follow instructions different from the legitimate
user's. Existing explorations of this threat applied to web agents often focus
on a single isolated adversarial goal, test with injected instructions that are
either too easy or not truly malicious, and often give the adversary
unreasonable access. In order to better focus adversarial research, we
construct a new benchmark called WASP (Web Agent Security against Prompt
injection attacks) that introduces realistic web agent hijacking objectives and
an isolated environment to test them in that does not affect real users or the
live web. As part of WASP, we also develop baseline attacks against three
popular web agentic systems (VisualWebArena, Claude Computer Use, and Operator)
instantiated with various state-of-the-art models. Our evaluation shows that
even AI agents backed by models with advanced reasoning capabilities and by
models with instruction hierarchy mitigations are susceptible to low-effort
human-written prompt injections. However, the realistic objectives in WASP also
allow us to observe that agents are currently not capable enough to complete
the goals of attackers end-to-end. Agents begin executing the adversarial
instruction between 16 and 86% of the time but only achieve the goal between 0
and 17% of the time. Based on these findings, we argue that adversarial
researchers should demonstrate stronger attacks that more consistently maintain
control over the agent given realistic constraints on the adversary's power.


http://arxiv.org/abs/2504.15585
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment. (4%)
Kun Wang; Guibin Zhang; Zhenhong Zhou; Jiahao Wu; Miao Yu; Shiqian Zhao; Chenlong Yin; Jinhu Fu; Yibo Yan; Hanjun Luo; Liang Lin; Zhihao Xu; Haolang Lu; Xinye Cao; Xinyun Zhou; Weifei Jin; Fanci Meng; Shicheng Xu; Junyuan Mao; Yu Wang; Hao Wu; Minghe Wang; Fan Zhang; Junfeng Fang; Wenjie Qu; Yue Liu; Chengwei Liu; Yifan Zhang; Qiankun Li; Chongye Guo; Yalan Qin; Zhaoxin Fan; Kai Wang; Yi Ding; Donghai Hong; Jiaming Ji; Yingxin Lai; Zitong Yu; Xinfeng Li; Yifan Jiang; Yanhui Li; Xinyu Deng; Junlin Wu; Dongxia Wang; Yihao Huang; Yufei Guo; Jen-tse Huang; Qiufeng Wang; Xiaolong Jin; Wenxuan Wang; Dongrui Liu; Yanwei Yue; Wenke Huang; Guancheng Wan; Heng Chang; Tianlin Li; Yi Yu; Chenghao Li; Jiawei Li; Lei Bai; Jie Zhang; Qing Guo; Jingyi Wang; Tianlong Chen; Joey Tianyi Zhou; Xiaojun Jia; Weisong Sun; Cong Wu; Jing Chen; Xuming Hu; Yiming Li; Xiao Wang; Ningyu Zhang; Luu Anh Tuan; Guowen Xu; Jiaheng Zhang; Tianwei Zhang; Xingjun Ma; Jindong Gu; Liang Pang; Xiang Wang; Bo An; Jun Sun; Mohit Bansal; Shirui Pan; Lingjuan Lyu; Yuval Elovici; Bhavya Kailkhura; Yaodong Yang; Hongwei Li; Wenyuan Xu; Yizhou Sun; Wei Wang; Qing Li; Ke Tang; Yu-Gang Jiang; Felix Juefei-Xu; Hui Xiong; Xiaofeng Wang; Dacheng Tao; Philip S. Yu; Qingsong Wen; Yang Liu
  The remarkable success of Large Language Models (LLMs) has illuminated a
promising pathway toward achieving Artificial General Intelligence for both
academic and industrial communities, owing to their unprecedented performance
across various applications. As LLMs continue to gain prominence in both
research and commercial domains, their security and safety implications have
become a growing concern, not only for researchers and corporations but also
for every nation. Currently, existing surveys on LLM safety primarily focus on
specific stages of the LLM lifecycle, e.g., deployment phase or fine-tuning
phase, lacking a comprehensive understanding of the entire "lifechain" of LLMs.
To address this gap, this paper introduces, for the first time, the concept of
"full-stack" safety to systematically consider safety issues throughout the
entire process of LLM training, deployment, and eventual commercialization.
Compared to the off-the-shelf LLM safety surveys, our work demonstrates several
distinctive advantages: (I) Comprehensive Perspective. We define the complete
LLM lifecycle as encompassing data preparation, pre-training, post-training,
deployment and final commercialization. To our knowledge, this represents the
first safety survey to encompass the entire lifecycle of LLMs. (II) Extensive
Literature Support. Our research is grounded in an exhaustive review of over
800+ papers, ensuring comprehensive coverage and systematic organization of
security issues within a more holistic understanding. (III) Unique Insights.
Through systematic literature analysis, we have developed reliable roadmaps and
perspectives for each chapter. Our work identifies promising research
directions, including safety in data generation, alignment techniques, model
editing, and LLM-based agent systems. These insights provide valuable guidance
for researchers pursuing future work in this field.


http://arxiv.org/abs/2504.15995
OPUS-VFL: Incentivizing Optimal Privacy-Utility Tradeoffs in Vertical Federated Learning. (3%)
Sindhuja Madabushi; Ahmad Faraz Khan; Haider Ali; Jin-Hee Cho
  Vertical Federated Learning (VFL) enables organizations with disjoint feature
spaces but shared user bases to collaboratively train models without sharing
raw data. However, existing VFL systems face critical limitations: they often
lack effective incentive mechanisms, struggle to balance privacy-utility
tradeoffs, and fail to accommodate clients with heterogeneous resource
capabilities. These challenges hinder meaningful participation, degrade model
performance, and limit practical deployment. To address these issues, we
propose OPUS-VFL, an Optimal Privacy-Utility tradeoff Strategy for VFL.
OPUS-VFL introduces a novel, privacy-aware incentive mechanism that rewards
clients based on a principled combination of model contribution, privacy
preservation, and resource investment. It employs a lightweight leave-one-out
(LOO) strategy to quantify feature importance per client, and integrates an
adaptive differential privacy mechanism that enables clients to dynamically
calibrate noise levels to optimize their individual utility. Our framework is
designed to be scalable, budget-balanced, and robust to inference and poisoning
attacks. Extensive experiments on benchmark datasets (MNIST, CIFAR-10, and
CIFAR-100) demonstrate that OPUS-VFL significantly outperforms state-of-the-art
VFL baselines in both efficiency and robustness. It reduces label inference
attack success rates by up to 20%, increases feature inference reconstruction
error (MSE) by over 30%, and achieves up to 25% higher incentives for clients
that contribute meaningfully while respecting privacy and cost constraints.
These results highlight the practicality and innovation of OPUS-VFL as a
secure, fair, and performance-driven solution for real-world VFL.


http://arxiv.org/abs/2504.15622
Exploring the Role of Large Language Models in Cybersecurity: A Systematic Survey. (1%)
Shuang Tian; Tao Zhang; Jiqiang Liu; Jiacheng Wang; Xuangou Wu; Xiaoqiang Zhu; Ruichen Zhang; Weiting Zhang; Zhenhui Yuan; Shiwen Mao; Dong In Kim
  With the rapid development of technology and the acceleration of
digitalisation, the frequency and complexity of cyber security threats are
increasing. Traditional cybersecurity approaches, often based on static rules
and predefined scenarios, are struggling to adapt to the rapidly evolving
nature of modern cyberattacks. There is an urgent need for more adaptive and
intelligent defence strategies. The emergence of Large Language Model (LLM)
provides an innovative solution to cope with the increasingly severe cyber
threats, and its potential in analysing complex attack patterns, predicting
threats and assisting real-time response has attracted a lot of attention in
the field of cybersecurity, and exploring how to effectively use LLM to defend
against cyberattacks has become a hot topic in the current research field. This
survey examines the applications of LLM from the perspective of the cyber
attack lifecycle, focusing on the three phases of defense reconnaissance,
foothold establishment, and lateral movement, and it analyzes the potential of
LLMs in Cyber Threat Intelligence (CTI) tasks. Meanwhile, we investigate how
LLM-based security solutions are deployed and applied in different network
scenarios. It also summarizes the internal and external risk issues faced by
LLM during its application. Finally, this survey also points out the facing
risk issues and possible future research directions in this domain.


http://arxiv.org/abs/2504.15479
Unifying Image Counterfactuals and Feature Attributions with Latent-Space Adversarial Attacks. (99%)
Jeremy Goldwasser; Giles Hooker
  Counterfactuals are a popular framework for interpreting machine learning
predictions. These what if explanations are notoriously challenging to create
for computer vision models: standard gradient-based methods are prone to
produce adversarial examples, in which imperceptible modifications to image
pixels provoke large changes in predictions. We introduce a new,
easy-to-implement framework for counterfactual images that can flexibly adapt
to contemporary advances in generative modeling. Our method, Counterfactual
Attacks, resembles an adversarial attack on the representation of the image
along a low-dimensional manifold. In addition, given an auxiliary dataset of
image descriptors, we show how to accompany counterfactuals with feature
attribution that quantify the changes between the original and counterfactual
images. These importance scores can be aggregated into global counterfactual
explanations that highlight the overall features driving model predictions.
While this unification is possible for any counterfactual method, it has
particular computational efficiency for ours. We demonstrate the efficacy of
our approach with the MNIST and CelebA datasets.


http://arxiv.org/abs/2504.14921
Fast Adversarial Training with Weak-to-Strong Spatial-Temporal Consistency in the Frequency Domain on Videos. (75%)
Songping Wang; Hanqing Liu; Yueming Lyu; Xiantao Hu; Ziwen He; Wei Wang; Caifeng Shan; Liang Wang
  Adversarial Training (AT) has been shown to significantly enhance adversarial
robustness via a min-max optimization approach. However, its effectiveness in
video recognition tasks is hampered by two main challenges. First, fast
adversarial training for video models remains largely unexplored, which
severely impedes its practical applications. Specifically, most video
adversarial training methods are computationally costly, with long training
times and high expenses. Second, existing methods struggle with the trade-off
between clean accuracy and adversarial robustness. To address these challenges,
we introduce Video Fast Adversarial Training with Weak-to-Strong consistency
(VFAT-WS), the first fast adversarial training method for video data.
Specifically, VFAT-WS incorporates the following key designs: First, it
integrates a straightforward yet effective temporal frequency augmentation
(TF-AUG), and its spatial-temporal enhanced form STF-AUG, along with a
single-step PGD attack to boost training efficiency and robustness. Second, it
devises a weak-to-strong spatial-temporal consistency regularization, which
seamlessly integrates the simpler TF-AUG and the more complex STF-AUG.
Leveraging the consistency regularization, it steers the learning process from
simple to complex augmentations. Both of them work together to achieve a better
trade-off between clean accuracy and robustness. Extensive experiments on
UCF-101 and HMDB-51 with both CNN and Transformer-based models demonstrate that
VFAT-WS achieves great improvements in adversarial robustness and corruption
robustness, while accelerating training by nearly 490%.


http://arxiv.org/abs/2504.15512
T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models. (68%)
Siyuan Liang; Jiayang Liu; Jiecheng Zhai; Tianmeng Fang; Rongcheng Tu; Aishan Liu; Xiaochun Cao; Dacheng Tao
  The rapid development of generative artificial intelligence has made text to
video models essential for building future multimodal world simulators.
However, these models remain vulnerable to jailbreak attacks, where specially
crafted prompts bypass safety mechanisms and lead to the generation of harmful
or unsafe content. Such vulnerabilities undermine the reliability and security
of simulation based applications. In this paper, we propose T2VShield, a
comprehensive and model agnostic defense framework designed to protect text to
video models from jailbreak threats. Our method systematically analyzes the
input, model, and output stages to identify the limitations of existing
defenses, including semantic ambiguities in prompts, difficulties in detecting
malicious content in dynamic video outputs, and inflexible model centric
mitigation strategies. T2VShield introduces a prompt rewriting mechanism based
on reasoning and multimodal retrieval to sanitize malicious inputs, along with
a multi scope detection module that captures local and global inconsistencies
across time and modalities. The framework does not require access to internal
model parameters and works with both open and closed source systems. Extensive
experiments on five platforms show that T2VShield can reduce jailbreak success
rates by up to 35 percent compared to strong baselines. We further develop a
human centered audiovisual evaluation protocol to assess perceptual safety,
emphasizing the importance of visual level defense in enhancing the
trustworthiness of next generation multimodal simulators.


http://arxiv.org/abs/2504.18564
DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimization. (22%)
Xinzhe Huang; Kedong Xiu; Tianhang Zheng; Churui Zeng; Wangze Ni; Zhan Qiin; Kui Ren; Chun Chen
  Recent research has focused on exploring the vulnerabilities of Large
Language Models (LLMs), aiming to elicit harmful and/or sensitive content from
LLMs. However, due to the insufficient research on dual-jailbreaking -- attacks
targeting both LLMs and Guardrails, the effectiveness of existing attacks is
limited when attempting to bypass safety-aligned LLMs shielded by guardrails.
Therefore, in this paper, we propose DualBreach, a target-driven framework for
dual-jailbreaking. DualBreach employs a Target-driven Initialization (TDI)
strategy to dynamically construct initial prompts, combined with a Multi-Target
Optimization (MTO) method that utilizes approximate gradients to jointly adapt
the prompts across guardrails and LLMs, which can simultaneously save the
number of queries and achieve a high dual-jailbreaking success rate. For
black-box guardrails, DualBreach either employs a powerful open-sourced
guardrail or imitates the target black-box guardrail by training a proxy model,
to incorporate guardrails into the MTO process.
  We demonstrate the effectiveness of DualBreach in dual-jailbreaking scenarios
through extensive evaluation on several widely-used datasets. Experimental
results indicate that DualBreach outperforms state-of-the-art methods with
fewer queries, achieving significantly higher success rates across all
settings. More specifically, DualBreach achieves an average dual-jailbreaking
success rate of 93.67% against GPT-4 with Llama-Guard-3 protection, whereas the
best success rate achieved by other methods is 88.33%. Moreover, DualBreach
only uses an average of 1.77 queries per successful dual-jailbreak,
outperforming other state-of-the-art methods. For the purpose of defense, we
propose an XGBoost-based ensemble defensive mechanism named EGuard, which
integrates the strengths of multiple guardrails, demonstrating superior
performance compared with Llama-Guard-3.


http://arxiv.org/abs/2504.14985
aiXamine: Simplified LLM Safety and Security. (13%)
Fatih Deniz; Dorde Popovic; Yazan Boshmaf; Euisuh Jeong; Minhaj Ahmad; Sanjay Chawla; Issa Khalil
  Evaluating Large Language Models (LLMs) for safety and security remains a
complex task, often requiring users to navigate a fragmented landscape of ad
hoc benchmarks, datasets, metrics, and reporting formats. To address this
challenge, we present aiXamine, a comprehensive black-box evaluation platform
for LLM safety and security. aiXamine integrates over 40 tests (i.e.,
benchmarks) organized into eight key services targeting specific dimensions of
safety and security: adversarial robustness, code security, fairness and bias,
hallucination, model and data privacy, out-of-distribution (OOD) robustness,
over-refusal, and safety alignment. The platform aggregates the evaluation
results into a single detailed report per model, providing a detailed breakdown
of model performance, test examples, and rich visualizations. We used aiXamine
to assess over 50 publicly available and proprietary LLMs, conducting over 2K
examinations. Our findings reveal notable vulnerabilities in leading models,
including susceptibility to adversarial attacks in OpenAI's GPT-4o, biased
outputs in xAI's Grok-3, and privacy weaknesses in Google's Gemini 2.0.
Additionally, we observe that open-source models can match or exceed
proprietary models in specific services such as safety alignment, fairness and
bias, and OOD robustness. Finally, we identify trade-offs between distillation
strategies, model size, training methods, and architectural choices.


http://arxiv.org/abs/2504.18563
Backdoor Defense in Diffusion Models via Spatial Attention Unlearning. (9%)
Abha Jha; Ashwath Vaithinathan Aravindan; Matthew Salaway; Atharva Sandeep Bhide; Duygu Nur Yaldiz
  Text-to-image diffusion models are increasingly vulnerable to backdoor
attacks, where malicious modifications to the training data cause the model to
generate unintended outputs when specific triggers are present. While
classification models have seen extensive development of defense mechanisms,
generative models remain largely unprotected due to their high-dimensional
output space, which complicates the detection and mitigation of subtle
perturbations. Defense strategies for diffusion models, in particular, remain
under-explored. In this work, we propose Spatial Attention Unlearning (SAU), a
novel technique for mitigating backdoor attacks in diffusion models. SAU
leverages latent space manipulation and spatial attention mechanisms to isolate
and remove the latent representation of backdoor triggers, ensuring precise and
efficient removal of malicious effects. We evaluate SAU across various types of
backdoor attacks, including pixel-based and style-based triggers, and
demonstrate its effectiveness in achieving 100% trigger removal accuracy.
Furthermore, SAU achieves a CLIP score of 0.7023, outperforming existing
methods while preserving the model's ability to generate high-quality,
semantically aligned images. Our results show that SAU is a robust, scalable,
and practical solution for securing text-to-image diffusion models against
backdoor attacks.


http://arxiv.org/abs/2504.14995
Trainable Quantum Neural Network for Multiclass Image Classification with the Power of Pre-trained Tree Tensor Networks. (1%)
Keisuke Murota; Takumi Kobori
  Tree tensor networks (TTNs) offer powerful models for image classification.
While these TTN image classifiers already show excellent performance on
classical hardware, embedding them into quantum neural networks (QNNs) may
further improve the performance by leveraging quantum resources. However,
embedding TTN classifiers into QNNs for multiclass classification remains
challenging. Key obstacles are the highorder gate operations required for large
bond dimensions and the mid-circuit postselection with exponentially low
success rates necessary for the exact embedding. In this work, to address these
challenges, we propose forest tensor network (FTN)-classifiers, which aggregate
multiple small-bond-dimension TTNs. This allows us to handle multiclass
classification without requiring large gates in the embedded circuits. We then
remove the overhead of mid-circuit postselection by extending the adiabatic
encoding framework to our setting and smoothly encode the FTN-classifiers into
a quantum forest tensor network (qFTN)- classifiers. Numerical experiments on
MNIST and CIFAR-10 demonstrate that we can successfully train FTN-classifiers
and encode them into qFTN-classifiers, while maintaining or even improving the
performance of the pre-trained FTN-classifiers. These results suggest that
synergy between TTN classification models and QNNs can provide a robust and
scalable framework for multiclass quantum-enhanced image classification.


http://arxiv.org/abs/2504.14541
Towards Model Resistant to Transferable Adversarial Examples via Trigger Activation. (99%)
Yi Yu; Song Xia; Xun Lin; Chenqi Kong; Wenhan Yang; Shijian Lu; Yap-Peng Tan; Alex C. Kot
  Adversarial examples, characterized by imperceptible perturbations, pose
significant threats to deep neural networks by misleading their predictions. A
critical aspect of these examples is their transferability, allowing them to
deceive {unseen} models in black-box scenarios. Despite the widespread
exploration of defense methods, including those on transferability, they show
limitations: inefficient deployment, ineffective defense, and degraded
performance on clean images. In this work, we introduce a novel training
paradigm aimed at enhancing robustness against transferable adversarial
examples (TAEs) in a more efficient and effective way. We propose a model that
exhibits random guessing behavior when presented with clean data
$\boldsymbol{x}$ as input, and generates accurate predictions when with
triggered data $\boldsymbol{x}+\boldsymbol{\tau}$. Importantly, the trigger
$\boldsymbol{\tau}$ remains constant for all data instances. We refer to these
models as \textbf{models with trigger activation}. We are surprised to find
that these models exhibit certain robustness against TAEs. Through the
consideration of first-order gradients, we provide a theoretical analysis of
this robustness. Moreover, through the joint optimization of the learnable
trigger and the model, we achieve improved robustness to transferable attacks.
Extensive experiments conducted across diverse datasets, evaluating a variety
of attacking methods, underscore the effectiveness and superiority of our
approach.


http://arxiv.org/abs/2504.14798
Verifying Robust Unlearning: Probing Residual Knowledge in Unlearned Models. (1%)
Hao Xuan; Xingyu Li
  Machine Unlearning (MUL) is crucial for privacy protection and content
regulation, yet recent studies reveal that traces of forgotten information
persist in unlearned models, enabling adversaries to resurface removed
knowledge. Existing verification methods only confirm whether unlearning was
executed, failing to detect such residual information leaks. To address this,
we introduce the concept of Robust Unlearning, ensuring models are
indistinguishable from retraining and resistant to adversarial recovery. To
empirically evaluate whether unlearning techniques meet this security standard,
we propose the Unlearning Mapping Attack (UMA), a post-unlearning verification
framework that actively probes models for forgotten traces using adversarial
queries. Extensive experiments on discriminative and generative tasks show that
existing unlearning techniques remain vulnerable, even when passing existing
verification metrics. By establishing UMA as a practical verification tool,
this study sets a new standard for assessing and enhancing machine unlearning
security.


http://arxiv.org/abs/2504.14395
Hydra: An Agentic Reasoning Approach for Enhancing Adversarial Robustness and Mitigating Hallucinations in Vision-Language Models. (99%)
Johnny Chung-En; Neil Yu; Neil Hsuan-Chih; Chen; Brian Jalaian; Nathaniel D. Bastian
  To develop trustworthy Vision-Language Models (VLMs), it is essential to
address adversarial robustness and hallucination mitigation, both of which
impact factual accuracy in high-stakes applications such as defense and
healthcare. Existing methods primarily focus on either adversarial defense or
hallucination post-hoc correction, leaving a gap in unified robustness
strategies. We introduce \textbf{Hydra}, an adaptive agentic framework that
enhances plug-in VLMs through iterative reasoning, structured critiques, and
cross-model verification, improving both resilience to adversarial
perturbations and intrinsic model errors. Hydra employs an Action-Critique
Loop, where it retrieves and critiques visual information, leveraging
Chain-of-Thought (CoT) and In-Context Learning (ICL) techniques to refine
outputs dynamically. Unlike static post-hoc correction methods, Hydra adapts to
both adversarial manipulations and intrinsic model errors, making it robust to
malicious perturbations and hallucination-related inaccuracies. We evaluate
Hydra on four VLMs, three hallucination benchmarks, two adversarial attack
strategies, and two adversarial defense methods, assessing performance on both
clean and adversarial inputs. Results show that Hydra surpasses plug-in VLMs
and state-of-the-art (SOTA) dehallucination methods, even without explicit
adversarial defenses, demonstrating enhanced robustness and factual
consistency. By bridging adversarial resistance and hallucination mitigation,
Hydra provides a scalable, training-free solution for improving the reliability
of VLMs in real-world applications.


http://arxiv.org/abs/2504.14423
Adversarial Attack for RGB-Event based Visual Object Tracking. (99%)
Qiang Chen; Xiao Wang; Haowen Wang; Bo Jiang; Lin Zhu; Dawei Zhang; Yonghong Tian; Jin Tang
  Visual object tracking is a crucial research topic in the fields of computer
vision and multi-modal fusion. Among various approaches, robust visual tracking
that combines RGB frames with Event streams has attracted increasing attention
from researchers. While striving for high accuracy and efficiency in tracking,
it is also important to explore how to effectively conduct adversarial attacks
and defenses on RGB-Event stream tracking algorithms, yet research in this area
remains relatively scarce. To bridge this gap, in this paper, we propose a
cross-modal adversarial attack algorithm for RGB-Event visual tracking. Because
of the diverse representations of Event streams, and given that Event voxels
and frames are more commonly used, this paper will focus on these two
representations for an in-depth study. Specifically, for the RGB-Event voxel,
we first optimize the perturbation by adversarial loss to generate RGB frame
adversarial examples. For discrete Event voxel representations, we propose a
two-step attack strategy, more in detail, we first inject Event voxels into the
target region as initialized adversarial examples, then, conduct a
gradient-guided optimization by perturbing the spatial location of the Event
voxels. For the RGB-Event frame based tracking, we optimize the cross-modal
universal perturbation by integrating the gradient information from multimodal
data. We evaluate the proposed approach against attacks on three widely used
RGB-Event Tracking datasets, i.e., COESOT, FE108, and VisEvent. Extensive
experiments show that our method significantly reduces the performance of the
tracker across numerous datasets in both unimodal and multimodal scenarios. The
source code will be released on
https://github.com/Event-AHU/Adversarial_Attack_Defense


http://arxiv.org/abs/2504.14348
Manipulating Multimodal Agents via Cross-Modal Prompt Injection. (92%)
Le Wang; Zonghao Ying; Tianyuan Zhang; Siyuan Liang; Shengshan Hu; Mingchuan Zhang; Aishan Liu; Xianglong Liu
  The emergence of multimodal large language models has redefined the agent
paradigm by integrating language and vision modalities with external data
sources, enabling agents to better interpret human instructions and execute
increasingly complex tasks. However, in this work, we identify a critical yet
previously overlooked security vulnerability in multimodal agents: cross-modal
prompt injection attacks. To exploit this vulnerability, we propose
CrossInject, a novel attack framework in which attackers embed adversarial
perturbations across multiple modalities to align with target malicious
content, allowing external instructions to hijack the agent's decision-making
process and execute unauthorized tasks. Our approach consists of two key
components. First, we introduce Visual Latent Alignment, where we optimize
adversarial features to the malicious instructions in the visual embedding
space based on a text-to-image generative model, ensuring that adversarial
images subtly encode cues for malicious task execution. Subsequently, we
present Textual Guidance Enhancement, where a large language model is leveraged
to infer the black-box defensive system prompt through adversarial meta
prompting and generate an malicious textual command that steers the agent's
output toward better compliance with attackers' requests. Extensive experiments
demonstrate that our method outperforms existing injection attacks, achieving
at least a +26.4% increase in attack success rates across diverse tasks.
Furthermore, we validate our attack's effectiveness in real-world multimodal
autonomous agents, highlighting its potential implications for safety-critical
applications.


http://arxiv.org/abs/2504.13551
Q-FAKER: Query-free Hard Black-box Attack via Controlled Generation. (99%)
CheolWon Na; YunSeok Choi; Jee-Hyong Lee
  Many adversarial attack approaches are proposed to verify the vulnerability
of language models. However, they require numerous queries and the information
on the target model. Even black-box attack methods also require the target
model's output information. They are not applicable in real-world scenarios, as
in hard black-box settings where the target model is closed and inaccessible.
Even the recently proposed hard black-box attacks still require many queries
and demand extremely high costs for training adversarial generators. To address
these challenges, we propose Q-faker (Query-free Hard Black-box Attacker), a
novel and efficient method that generates adversarial examples without
accessing the target model. To avoid accessing the target model, we use a
surrogate model instead. The surrogate model generates adversarial sentences
for a target-agnostic attack. During this process, we leverage controlled
generation techniques. We evaluate our proposed method on eight datasets.
Experimental results demonstrate our method's effectiveness including high
transferability and the high quality of the generated adversarial examples, and
prove its practical in hard black-box settings.


http://arxiv.org/abs/2504.14137
Rethinking Target Label Conditioning in Adversarial Attacks: A 2D Tensor-Guided Generative Approach. (92%)
Hangyu Liu; Bo Peng; Pengxiang Ding; Donglin Wang
  Compared to single-target adversarial attacks, multi-target attacks have
garnered significant attention due to their ability to generate adversarial
images for multiple target classes simultaneously. Existing generative
approaches for multi-target attacks mainly analyze the effect of the use of
target labels on noise generation from a theoretical perspective, lacking
practical validation and comprehensive summarization. To address this gap, we
first identify and validate that the semantic feature quality and quantity are
critical factors affecting the transferability of targeted attacks: 1) Feature
quality refers to the structural and detailed completeness of the implanted
target features, as deficiencies may result in the loss of key discriminative
information; 2) Feature quantity refers to the spatial sufficiency of the
implanted target features, as inadequacy limits the victim model's attention to
this feature. Based on these findings, we propose the 2D Tensor-Guided
Adversarial Fusion (2D-TGAF) framework, which leverages the powerful generative
capabilities of diffusion models to encode target labels into two-dimensional
semantic tensors for guiding adversarial noise generation. Additionally, we
design a novel masking strategy tailored for the training process, ensuring
that parts of the generated noise retain complete semantic information about
the target class. Extensive experiments on the standard ImageNet dataset
demonstrate that 2D-TGAF consistently surpasses state-of-the-art methods in
attack success rates, both on normally trained models and across various
defense mechanisms.


http://arxiv.org/abs/2504.13562
DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modification. (82%)
Yu Li; Han Jiang; Zhihua Wei
  With the widespread adoption of Large Language Models (LLMs), jailbreak
attacks have become an increasingly pressing safety concern. While
safety-aligned LLMs can effectively defend against normal harmful queries, they
remain vulnerable to such attacks. Existing defense methods primarily rely on
fine-tuning or input modification, which often suffer from limited
generalization and reduced utility. To address this, we introduce DETAM, a
finetuning-free defense approach that improves the defensive capabilities
against jailbreak attacks of LLMs via targeted attention modification.
Specifically, we analyze the differences in attention scores between successful
and unsuccessful defenses to identify the attention heads sensitive to
jailbreak attacks. During inference, we reallocate attention to emphasize the
user's core intention, minimizing interference from attack tokens. Our
experimental results demonstrate that DETAM outperforms various baselines in
jailbreak defense and exhibits robust generalization across different attacks
and models, maintaining its effectiveness even on in-the-wild jailbreak data.
Furthermore, in evaluating the model's utility, we incorporated over-defense
datasets, which further validate the superior performance of our approach. The
code will be released immediately upon acceptance.


http://arxiv.org/abs/2504.14096
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment. (75%)
Yogesh Kulkarni; Pooyan Fazli
  Video-language models (Video-LLMs) excel at understanding video content but
struggle with spatial relationships, temporal ordering, and cross-frame
continuity. To address these limitations, we introduce VideoPASTA (Preference
Alignment with Spatio-Temporal-Cross Frame Adversaries), a framework that
enhances Video-LLMs through targeted preference optimization. VideoPASTA trains
models to distinguish accurate video representations from carefully generated
adversarial examples that deliberately violate spatial, temporal, or
cross-frame relations. By applying Direct Preference Optimization to just 7,020
preference pairs, VideoPASTA learns robust representations that capture
fine-grained spatial relationships and long-range temporal dynamics.
Experiments on standard video benchmarks show significant relative performance
gains of 3.05% on VideoMME, 1.97% on NeXTQA, and 1.31% on LongVideoBench, over
the baseline Qwen2.5-VL model. These results demonstrate that targeted
alignment, rather than massive pretraining or architectural modifications,
effectively addresses core video-language challenges. Notably, VideoPASTA
achieves these improvements without human annotation or captioning, relying on
just 32-frame sampling, compared to the 96-frame, multi-GPU setups of prior
work. This efficiency makes our approach a scalable, plug-and-play solution
that seamlessly integrates with existing models while preserving their
capabilities.


http://arxiv.org/abs/2504.13775
BadApex: Backdoor Attack Based on Adaptive Optimization Mechanism of Black-box Large Language Models. (45%)
Zhengxian Wu; Juan Wen; Wanli Peng; Ziwei Zhang; Yinghan Zhou; Yiming Xue
  Previous insertion-based and paraphrase-based backdoors have achieved great
success in attack efficacy, but they ignore the text quality and semantic
consistency between poisoned and clean texts. Although recent studies introduce
LLMs to generate poisoned texts and improve the stealthiness, semantic
consistency, and text quality, their hand-crafted prompts rely on expert
experiences, facing significant challenges in prompt adaptability and attack
performance after defenses. In this paper, we propose a novel backdoor attack
based on adaptive optimization mechanism of black-box large language models
(BadApex), which leverages a black-box LLM to generate poisoned text through a
refined prompt. Specifically, an Adaptive Optimization Mechanism is designed to
refine an initial prompt iteratively using the generation and modification
agents. The generation agent generates the poisoned text based on the initial
prompt. Then the modification agent evaluates the quality of the poisoned text
and refines a new prompt. After several iterations of the above process, the
refined prompt is used to generate poisoned texts through LLMs. We conduct
extensive experiments on three dataset with six backdoor attacks and two
defenses. Extensive experimental results demonstrate that BadApex significantly
outperforms state-of-the-art attacks. It improves prompt adaptability, semantic
consistency, and text quality. Furthermore, when two defense methods are
applied, the average attack success rate (ASR) still up to 96.75%.


http://arxiv.org/abs/2504.14064
DoomArena: A framework for Testing AI Agents Against Evolving Security Threats. (10%)
Leo Boisvert; Mihir Bansal; Chandra Kiran Reddy Evuru; Gabriel Huang; Abhay Puri; Avinandan Bose; Maryam Fazel; Quentin Cappart; Jason Stanley; Alexandre Lacoste; Alexandre Drouin; Krishnamurthy Dvijotham
  We present DoomArena, a security evaluation framework for AI agents.
DoomArena is designed on three principles: 1) It is a plug-in framework and
integrates easily into realistic agentic frameworks like BrowserGym (for web
agents) and $\tau$-bench (for tool calling agents); 2) It is configurable and
allows for detailed threat modeling, allowing configuration of specific
components of the agentic framework being attackable, and specifying targets
for the attacker; and 3) It is modular and decouples the development of attacks
from details of the environment in which the agent is deployed, allowing for
the same attacks to be applied across multiple environments. We illustrate
several advantages of our framework, including the ability to adapt to new
threat models and environments easily, the ability to easily combine several
previously published attacks to enable comprehensive and fine-grained security
testing, and the ability to analyze trade-offs between various vulnerabilities
and performance. We apply DoomArena to state-of-the-art (SOTA) web and
tool-calling agents and find a number of surprising results: 1) SOTA agents
have varying levels of vulnerability to different threat models (malicious user
vs malicious environment), and there is no Pareto dominant agent across all
threat models; 2) When multiple attacks are applied to an agent, they often
combine constructively; 3) Guardrail model-based defenses seem to fail, while
defenses based on powerful SOTA LLMs work better. DoomArena is available at
https://github.com/ServiceNow/DoomArena.


http://arxiv.org/abs/2504.13610
Fairness and Robustness in Machine Unlearning. (9%)
Khoa Tran; Simon S. Woo
  Machine unlearning poses the challenge of ``how to eliminate the influence of
specific data from a pretrained model'' in regard to privacy concerns. While
prior research on approximated unlearning has demonstrated accuracy and
efficiency in time complexity, we claim that it falls short of achieving exact
unlearning, and we are the first to focus on fairness and robustness in machine
unlearning algorithms. Our study presents fairness Conjectures for a
well-trained model, based on the variance-bias trade-off characteristic, and
considers their relevance to robustness. Our Conjectures are supported by
experiments conducted on the two most widely used model architectures, ResNet
and ViT, demonstrating the correlation between fairness and robustness:
\textit{the higher fairness-gap is, the more the model is sensitive and
vulnerable}. In addition, our experiments demonstrate the vulnerability of
current state-of-the-art approximated unlearning algorithms to adversarial
attacks, where their unlearned models suffer a significant drop in accuracy
compared to the exact-unlearned models. We claim that our fairness-gap
measurement and robustness metric should be used to evaluate the unlearning
algorithm. Furthermore, we demonstrate that unlearning in the intermediate and
last layers is sufficient and cost-effective for time and memory complexity.


http://arxiv.org/abs/2507.01020
AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models. (8%)
Aashray Reddy; Andrew Zagula; Nicholas Saban
Large Language Models (LLMs) continue to exhibit vulnerabilities to jailbreaking attacks: carefully crafted malicious inputs intended to circumvent safety guardrails and elicit harmful responses. As such, we present AutoAdv, a novel framework that automates adversarial prompt generation to systematically evaluate and expose vulnerabilities in LLM safety mechanisms. Our approach leverages a parametric attacker LLM to produce semantically disguised malicious prompts through strategic rewriting techniques, specialized system prompts, and optimized hyperparameter configurations. The primary contribution of our work is a dynamic, multi-turn attack methodology that analyzes failed jailbreak attempts and iteratively generates refined follow-up prompts, leveraging techniques such as roleplaying, misdirection, and contextual manipulation. We quantitatively evaluate attack success rate (ASR) using the StrongREJECT (arXiv:2402.10260 [cs.CL]) framework across sequential interaction turns. Through extensive empirical evaluation of state-of-the-art models--including ChatGPT, Llama, and DeepSeek--we reveal significant vulnerabilities, with our automated attacks achieving jailbreak success rates of up to 86% for harmful content generation. Our findings reveal that current safety mechanisms remain susceptible to sophisticated multi-turn attacks, emphasizing the urgent need for more robust defense strategies.

http://arxiv.org/abs/2504.14119
CodeCrash: Stress Testing LLM Reasoning under Structural and Semantic Perturbations. (1%)
Man Ho Lam; Chaozheng Wang; Jen-tse Huang; Michael R. Lyu
  Large Language Models (LLMs) have recently demonstrated strong capabilities
in code-related tasks, yet their robustness in code comprehension and reasoning
remains insufficiently explored. We present CodeCrash, a comprehensive
stress-testing benchmark comprising 1,279 questions from two established
datasets, CruxEval and LiveCodeBench, designed to evaluate model reasoning
reliability under non-standard coding environments. We systematically evaluate
17 LLMs across input and output prediction tasks using direct and
Chain-of-Thought prompting approaches, revealing that LLMs are particularly
vulnerable to disorganized code and overly reliant on natural language cues:
aggregated structural perturbations result in over 14 percentage points (pp) of
degradation, while textual perturbations cause a performance drop of over 11
pp. Moreover, self-reflective mechanisms in state-of-the-art reasoning models
significantly increase token usage by 2-3 times, reduce output confidence, and
even lead to catastrophic reasoning failures when faced with targeted
perturbations -- for instance, QwQ-32B generates over 12,000 redundant tokens
under reasoning-level perturbations. CodeCrash provides a rigorous benchmark
for evaluating robustness in code understanding, guiding future research toward
more reliable and resilient LLMs in code reasoning. The benchmark code,
perturbed datasets, and full leaderboard are publicly available at
https://cuhk-arise.github.io/CodeCrash/ .


http://arxiv.org/abs/2504.13474
Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask. (1%)
Yue Li; Xiao Li; Hao Wu; Minghui Xu; Yue Zhang; Xiuzhen Cheng; Fengyuan Xu; Sheng Zhong
  Large Language Models are a promising tool for automated vulnerability
detection, thanks to their success in code generation and repair. However,
despite widespread adoption, a critical question remains: Are LLMs truly
effective at detecting real-world vulnerabilities? Current evaluations, which
often assess models on isolated functions or files, ignore the broader
execution and data-flow context essential for understanding vulnerabilities.
This oversight leads to two types of misleading outcomes: incorrect conclusions
and flawed rationales, collectively undermining the reliability of prior
assessments. Therefore, in this paper, we challenge three widely held community
beliefs: that LLMs are (i) unreliable, (ii) insensitive to code patches, and
(iii) performance-plateaued across model scales. We argue that these beliefs
are artifacts of context-deprived evaluations. To address this, we propose
CORRECT (Context-Rich Reasoning Evaluation of Code with Trust), a new
evaluation framework that systematically incorporates contextual information
into LLM-based vulnerability detection. We construct a context-rich dataset of
2,000 vulnerable-patched program pairs spanning 99 CWEs and evaluate 13 LLMs
across four model families. Our framework elicits both binary predictions and
natural-language rationales, which are further validated using LLM-as-a-judge
techniques. Our findings overturn existing misconceptions. When provided with
sufficient context, SOTA LLMs achieve significantly improved performance (e.g.,
0.7 F1-score on key CWEs), with 0.8 precision. We show that most false
positives stem from reasoning errors rather than misclassification, and that
while model and test-time scaling improve performance, they introduce
diminishing returns and trade-offs in recall. Finally, we uncover new flaws in
current LLM-based detection systems, such as limited generalization and
overthinking biases.


http://arxiv.org/abs/2504.12644
Quantum Computing Supported Adversarial Attack-Resilient Autonomous Vehicle Perception Module for Traffic Sign Classification. (99%)
Reek Majumder; Mashrur Chowdhury; Sakib Mahmud Khan; Zadid Khan; Fahim Ahmad; Frank Ngeni; Gurcan Comert; Judith Mwakalonge; Dimitra Michalaka
  Deep learning (DL)-based image classification models are essential for
autonomous vehicle (AV) perception modules since incorrect categorization might
have severe repercussions. Adversarial attacks are widely studied cyberattacks
that can lead DL models to predict inaccurate output, such as incorrectly
classified traffic signs by the perception module of an autonomous vehicle. In
this study, we create and compare hybrid classical-quantum deep learning
(HCQ-DL) models with classical deep learning (C-DL) models to demonstrate
robustness against adversarial attacks for perception modules. Before feeding
them into the quantum system, we used transfer learning models, alexnet and
vgg-16, as feature extractors. We tested over 1000 quantum circuits in our
HCQ-DL models for projected gradient descent (PGD), fast gradient sign attack
(FGSA), and gradient attack (GA), which are three well-known untargeted
adversarial approaches. We evaluated the performance of all models during
adversarial attacks and no-attack scenarios. Our HCQ-DL models maintain
accuracy above 95\% during a no-attack scenario and above 91\% for GA and FGSA
attacks, which is higher than C-DL models. During the PGD attack, our
alexnet-based HCQ-DL model maintained an accuracy of 85\% compared to C-DL
models that achieved accuracies below 21\%. Our results highlight that the
HCQ-DL models provide improved accuracy for traffic sign classification under
adversarial settings compared to their classical counterparts.


http://arxiv.org/abs/2504.13301
DYNAMITE: Dynamic Defense Selection for Enhancing Machine Learning-based Intrusion Detection Against Adversarial Attacks. (98%)
Jing Chen; Onat Gungor; Zhengli Shang; Elvin Li; Tajana Rosing
  The rapid proliferation of the Internet of Things (IoT) has introduced
substantial security vulnerabilities, highlighting the need for robust
Intrusion Detection Systems (IDS). Machine learning-based intrusion detection
systems (ML-IDS) have significantly improved threat detection capabilities;
however, they remain highly susceptible to adversarial attacks. While numerous
defense mechanisms have been proposed to enhance ML-IDS resilience, a
systematic approach for selecting the most effective defense against a specific
adversarial attack remains absent. To address this challenge, we propose
Dynamite, a dynamic defense selection framework that enhances ML-IDS by
intelligently identifying and deploying the most suitable defense using a
machine learning-driven selection mechanism. Our results demonstrate that
Dynamite achieves a 96.2% reduction in computational time compared to the
Oracle, significantly decreasing computational overhead while preserving strong
prediction performance. Dynamite also demonstrates an average F1-score
improvement of 76.7% over random defense and 65.8% over the best static
state-of-the-art defense.


http://arxiv.org/abs/2504.12875
A Client-level Assessment of Collaborative Backdoor Poisoning in Non-IID Federated Learning. (83%)
Phung Lai; Guanxiong Liu; Hai Phan; Issa Khalil; Abdallah Khreishah; Xintao Wu
  Federated learning (FL) enables collaborative model training using
decentralized private data from multiple clients. While FL has shown robustness
against poisoning attacks with basic defenses, our research reveals new
vulnerabilities stemming from non-independent and identically distributed
(non-IID) data among clients. These vulnerabilities pose a substantial risk of
model poisoning in real-world FL scenarios.
  To demonstrate such vulnerabilities, we develop a novel collaborative
backdoor poisoning attack called CollaPois. In this attack, we distribute a
single pre-trained model infected with a Trojan to a group of compromised
clients. These clients then work together to produce malicious gradients,
causing the FL model to consistently converge towards a low-loss region
centered around the Trojan-infected model. Consequently, the impact of the
Trojan is amplified, especially when the benign clients have diverse local data
distributions and scattered local gradients. CollaPois stands out by achieving
its goals while involving only a limited number of compromised clients, setting
it apart from existing attacks. Also, CollaPois effectively avoids noticeable
shifts or degradation in the FL model's performance on legitimate data samples,
allowing it to operate stealthily and evade detection by advanced robust FL
algorithms.
  Thorough theoretical analysis and experiments conducted on various benchmark
datasets demonstrate the superiority of CollaPois compared to state-of-the-art
backdoor attacks. Notably, CollaPois bypasses existing backdoor defenses,
especially in scenarios where clients possess diverse data distributions.
Moreover, the results show that CollaPois remains effective even when involving
a small number of compromised clients. Notably, clients whose local data is
closely aligned with compromised clients experience higher risks of backdoor
infections.


http://arxiv.org/abs/2504.13061
ArtistAuditor: Auditing Artist Style Pirate in Text-to-Image Generation Models. (41%)
Linkang Du; Zheng Zhu; Min Chen; Zhou Su; Shouling Ji; Peng Cheng; Jiming Chen; Zhikun Zhang
  Text-to-image models based on diffusion processes, such as DALL-E, Stable
Diffusion, and Midjourney, are capable of transforming texts into detailed
images and have widespread applications in art and design. As such, amateur
users can easily imitate professional-level paintings by collecting an artist's
work and fine-tuning the model, leading to concerns about artworks' copyright
infringement. To tackle these issues, previous studies either add visually
imperceptible perturbation to the artwork to change its underlying styles
(perturbation-based methods) or embed post-training detectable watermarks in
the artwork (watermark-based methods). However, when the artwork or the model
has been published online, i.e., modification to the original artwork or model
retraining is not feasible, these strategies might not be viable.
  To this end, we propose a novel method for data-use auditing in the
text-to-image generation model. The general idea of ArtistAuditor is to
identify if a suspicious model has been finetuned using the artworks of
specific artists by analyzing the features related to the style. Concretely,
ArtistAuditor employs a style extractor to obtain the multi-granularity style
representations and treats artworks as samplings of an artist's style. Then,
ArtistAuditor queries a trained discriminator to gain the auditing decisions.
The experimental results on six combinations of models and datasets show that
ArtistAuditor can achieve high AUC values (> 0.937). By studying
ArtistAuditor's transferability and core modules, we provide valuable insights
into the practical implementation. Finally, we demonstrate the effectiveness of
ArtistAuditor in real-world cases by an online platform Scenario. ArtistAuditor
is open-sourced at https://github.com/Jozenn/ArtistAuditor.


http://arxiv.org/abs/2504.12747
Privacy Protection Against Personalized Text-to-Image Synthesis via Cross-image Consistency Constraints. (3%)
Guanyu Wang; Kailong Wang; Yihao Huang; Mingyi Zhou; Zhang Qing cnwatcher; Geguang Pu; Li Li
  The rapid advancement of diffusion models and personalization techniques has
made it possible to recreate individual portraits from just a few publicly
available images. While such capabilities empower various creative
applications, they also introduce serious privacy concerns, as adversaries can
exploit them to generate highly realistic impersonations. To counter these
threats, anti-personalization methods have been proposed, which add adversarial
perturbations to published images to disrupt the training of personalization
models. However, existing approaches largely overlook the intrinsic multi-image
nature of personalization and instead adopt a naive strategy of applying
perturbations independently, as commonly done in single-image settings. This
neglects the opportunity to leverage inter-image relationships for stronger
privacy protection. Therefore, we advocate for a group-level perspective on
privacy protection against personalization. Specifically, we introduce
Cross-image Anti-Personalization (CAP), a novel framework that enhances
resistance to personalization by enforcing style consistency across perturbed
images. Furthermore, we develop a dynamic ratio adjustment strategy that
adaptively balances the impact of the consistency loss throughout the attack
iterations. Extensive experiments on the classical CelebHQ and VGGFace2
benchmarks show that CAP substantially improves existing methods.


http://arxiv.org/abs/2504.13077
Effective Dual-Region Augmentation for Reduced Reliance on Large Amounts of Labeled Data. (1%)
Prasanna Reddy Pulakurthi; Majid Rabbani; Melo Celso M. de; Sohail A. Dianat; Raghuveer M. Rao
  This paper introduces a novel dual-region augmentation approach designed to
reduce reliance on large-scale labeled datasets while improving model
robustness and adaptability across diverse computer vision tasks, including
source-free domain adaptation (SFDA) and person re-identification (ReID). Our
method performs targeted data transformations by applying random noise
perturbations to foreground objects and spatially shuffling background patches.
This effectively increases the diversity of the training data, improving model
robustness and generalization. Evaluations on the PACS dataset for SFDA
demonstrate that our augmentation strategy consistently outperforms existing
methods, achieving significant accuracy improvements in both single-target and
multi-target adaptation settings. By augmenting training data through
structured transformations, our method enables model generalization across
domains, providing a scalable solution for reducing reliance on manually
annotated datasets. Furthermore, experiments on Market-1501 and DukeMTMC-reID
datasets validate the effectiveness of our approach for person ReID, surpassing
traditional augmentation techniques. The code is available at
https://github.com/PrasannaPulakurthi/Foreground-Background-Augmentation


http://arxiv.org/abs/2504.13055
NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation. (1%)
Xiangyan Liu; Jinjie Ni; Zijian Wu; Chao Du; Longxu Dou; Haonan Wang; Tianyu Pang; Michael Qizhe Shieh
  Recent advances in reinforcement learning (RL) have strengthened the
reasoning capabilities of vision-language models (VLMs). However, enhancing
policy exploration to better scale test-time compute remains largely
underexplored. In addition, VLMs continue to struggle with imperfect visual
perception, which in turn affects the subsequent reasoning process. To this
end, we propose NoisyRollout, a simple yet effective data augmentation method
that mixes trajectories from both clean and moderately distorted images during
RL training. By injecting targeted diversity in visual perception and the
resulting reasoning patterns, NoisyRollout promotes better policy exploration
through vision-oriented inductive biases, ultimately leading to more robust
reasoning behaviors. We further adopt a noise annealing schedule that gradually
reduces distortion strength over training, leveraging noisy signals early on
while ensuring training stability in later stages. Crucially, our method is
easy-to-adopt--requiring no additional training cost and no modifications to
the RL objective. Extensive experiments on $2$ distinct training datasets
demonstrate that NoisyRollout achieves state-of-the-art performance among
open-source RL-tuned models across $5$ out-of-domain reasoning and perception
benchmarks. Furthermore, we validate the effectiveness of NoisyRollout across
model sizes ($7$B and $32$B) and data scales (from $1$K to $6$K), highlighting
its generalizability and scalability.


http://arxiv.org/abs/2504.11923
SemDiff: Generating Natural Unrestricted Adversarial Examples via Semantic Attributes Optimization in Diffusion Models. (99%)
Zeyu Dai; Shengcai Liu; Rui He; Jiahao Wu; Ning Lu; Wenqi Fan; Qing Li; Ke Tang
  Unrestricted adversarial examples (UAEs), allow the attacker to create
non-constrained adversarial examples without given clean samples, posing a
severe threat to the safety of deep learning models. Recent works utilize
diffusion models to generate UAEs. However, these UAEs often lack naturalness
and imperceptibility due to simply optimizing in intermediate latent noises. In
light of this, we propose SemDiff, a novel unrestricted adversarial attack that
explores the semantic latent space of diffusion models for meaningful
attributes, and devises a multi-attributes optimization approach to ensure
attack success while maintaining the naturalness and imperceptibility of
generated UAEs. We perform extensive experiments on four tasks on three
high-resolution datasets, including CelebA-HQ, AFHQ and ImageNet. The results
demonstrate that SemDiff outperforms state-of-the-art methods in terms of
attack success rate and imperceptibility. The generated UAEs are natural and
exhibit semantically meaningful changes, in accord with the attributes'
weights. In addition, SemDiff is found capable of evading different defenses,
which further validates its effectiveness and threatening.


http://arxiv.org/abs/2504.12255
Human Aligned Compression for Robust Models. (98%)
Samuel Räber; Andreas Plesner; Till Aczel; Roger Wattenhofer
  Adversarial attacks on image models threaten system robustness by introducing
imperceptible perturbations that cause incorrect predictions. We investigate
human-aligned learned lossy compression as a defense mechanism, comparing two
learned models (HiFiC and ELIC) against traditional JPEG across various quality
levels. Our experiments on ImageNet subsets demonstrate that learned
compression methods outperform JPEG, particularly for Vision Transformer
architectures, by preserving semantically meaningful content while removing
adversarial noise. Even in white-box settings where attackers can access the
defense, these methods maintain substantial effectiveness. We also show that
sequential compression--applying rounds of
compression/decompression--significantly enhances defense efficacy while
maintaining classification performance. Our findings reveal that human-aligned
compression provides an effective, computationally efficient defense that
protects the image features most relevant to human and machine understanding.
It offers a practical approach to improving model robustness against
adversarial threats.


http://arxiv.org/abs/2504.18556
RDI: An adversarial robustness evaluation metric for deep neural networks based on sample clustering features. (98%)
Jialei Song; Xingquan Zuo; Feiyang Wang; Hai Huang; Tianle Zhang
  Deep neural networks (DNNs) are highly susceptible to adversarial samples,
raising concerns about their reliability in safety-critical tasks. Currently,
methods of evaluating adversarial robustness are primarily categorized into
attack-based and certified robustness evaluation approaches. The former not
only relies on specific attack algorithms but also is highly time-consuming,
while the latter due to its analytical nature, is typically difficult to
implement for large and complex models. A few studies evaluate model robustness
based on the model's decision boundary, but they suffer from low evaluation
accuracy. To address the aforementioned issues, we propose a novel adversarial
robustness evaluation metric, Robustness Difference Index (RDI), which is based
on sample clustering features. RDI draws inspiration from clustering evaluation
by analyzing the intra-class and inter-class distances of feature vectors
separated by the decision boundary to quantify model robustness. It is
attack-independent and has high computational efficiency. Experiments show
that, RDI demonstrates a stronger correlation with the gold-standard
adversarial robustness metric of attack success rate (ASR). The average
computation time of RDI is only 1/30 of the evaluation method based on the PGD
attack. Our open-source code is available at:
https://anonymous.4open.science/r/RDI-B1DA.


http://arxiv.org/abs/2504.11990
Secure Transfer Learning: Training Clean Models Against Backdoor in (Both) Pre-trained Encoders and Downstream Datasets. (2%)
Yechao Zhang; Yuxuan Zhou; Tianyu Li; Minghui Li; Shengshan Hu; Wei Luo; Leo Yu Zhang
  Transfer learning from pre-trained encoders has become essential in modern
machine learning, enabling efficient model adaptation across diverse tasks.
However, this combination of pre-training and downstream adaptation creates an
expanded attack surface, exposing models to sophisticated backdoor embeddings
at both the encoder and dataset levels--an area often overlooked in prior
research. Additionally, the limited computational resources typically available
to users of pre-trained encoders constrain the effectiveness of generic
backdoor defenses compared to end-to-end training from scratch. In this work,
we investigate how to mitigate potential backdoor risks in resource-constrained
transfer learning scenarios. Specifically, we conduct an exhaustive analysis of
existing defense strategies, revealing that many follow a reactive workflow
based on assumptions that do not scale to unknown threats, novel attack types,
or different training paradigms. In response, we introduce a proactive mindset
focused on identifying clean elements and propose the Trusted Core (T-Core)
Bootstrapping framework, which emphasizes the importance of pinpointing
trustworthy data and neurons to enhance model security. Our empirical
evaluations demonstrate the effectiveness and superiority of T-Core,
specifically assessing 5 encoder poisoning attacks, 7 dataset poisoning
attacks, and 14 baseline defenses across five benchmark datasets, addressing
four scenarios of 3 potential backdoor threats.


http://arxiv.org/abs/2504.11867
MDHP-Net: Detecting an Emerging Time-exciting Threat in IVN. (1%)
Qi Liu; Yanchen Liu; Ruifeng Li; Chenhong Cao; Yufeng Li; Xingyu Li; Peng Wang; Runhan Feng; Shiyang Bu
  The integration of intelligent and connected technologies in modern vehicles,
while offering enhanced functionalities through Electronic Control Unit (ECU)
and interfaces like OBD-II and telematics, also exposes the vehicle's
in-vehicle network (IVN) to potential cyberattacks. Unlike prior work, we
identify a new time-exciting threat model against IVN. These attacks inject
malicious messages that exhibit a time-exciting effect, gradually manipulating
network traffic to disrupt vehicle operations and compromise safety-critical
functions. We systematically analyze the characteristics of the threat:
dynamism, time-exciting impact, and low prior knowledge dependency. To validate
its practicality, we replicate the attack on a real Advanced Driver Assistance
System via Controller Area Network (CAN), exploiting Unified Diagnostic Service
vulnerabilities and proposing four attack strategies. While CAN's integrity
checks mitigate attacks, Ethernet migration (e.g., DoIP/SOME/IP) introduces new
surfaces. We further investigate the feasibility of time-exciting threat under
SOME/IP. To detect time-exciting threat, we introduce MDHP-Net, leveraging
Multi-Dimentional Hawkes Process (MDHP) and temporal and message-wise feature
extracting structures. Meanwhile, to estimate MDHP parameters, we developed the
first GPU-optimized gradient descent solver for MDHP (MDHP-GDS). These modules
significantly improves the detection rate under time-exciting attacks in
multi-ECU IVN system. To address data scarcity, we release STEIA9, the first
open-source dataset for time-exciting attacks, covering 9 Ethernet-based attack
scenarios. Extensive experiments on STEIA9 (9 attack scenarios) show MDHP-Net
outperforms 3 baselines, confirming attack feasibility and detection efficacy.


http://arxiv.org/abs/2504.11707
Towards Safe Synthetic Image Generation On the Web: A Multimodal Robust NSFW Defense and Million Scale Dataset. (99%)
Muhammad Shahid Muneer; Simon S. Woo
  In the past years, we have witnessed the remarkable success of Text-to-Image
(T2I) models and their widespread use on the web. Extensive research in making
T2I models produce hyper-realistic images has led to new concerns, such as
generating Not-Safe-For-Work (NSFW) web content and polluting the web society.
To help prevent misuse of T2I models and create a safer web environment for
users features like NSFW filters and post-hoc security checks are used in these
models. However, recent work unveiled how these methods can easily fail to
prevent misuse. In particular, adversarial attacks on text and image modalities
can easily outplay defensive measures. %Exploiting such leads to the growing
concern of preventing adversarial attacks on text and image modalities.
Moreover, there is currently no robust multimodal NSFW dataset that includes
both prompt and image pairs and adversarial examples. This work proposes a
million-scale prompt and image dataset generated using open-source diffusion
models. Second, we develop a multimodal defense to distinguish safe and NSFW
text and images, which is robust against adversarial attacks and directly
alleviates current challenges. Our extensive experiments show that our model
performs well against existing SOTA NSFW detection methods in terms of accuracy
and recall, drastically reducing the Attack Success Rate (ASR) in multimodal
adversarial attack scenarios. Code:
https://github.com/shahidmuneer/multimodal-nsfw-defense.


http://arxiv.org/abs/2504.11195
R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning. (92%)
Lijun Sheng; Jian Liang; Zilei Wang; Ran He
  Vision-language models (VLMs), such as CLIP, have gained significant
popularity as foundation models, with numerous fine-tuning methods developed to
enhance performance on downstream tasks. However, due to their inherent
vulnerability and the common practice of selecting from a limited set of
open-source models, VLMs suffer from a higher risk of adversarial attacks than
traditional vision models. Existing defense techniques typically rely on
adversarial fine-tuning during training, which requires labeled data and lacks
of flexibility for downstream tasks. To address these limitations, we propose
robust test-time prompt tuning (R-TPT), which mitigates the impact of
adversarial attacks during the inference stage. We first reformulate the
classic marginal entropy objective by eliminating the term that introduces
conflicts under adversarial conditions, retaining only the pointwise entropy
minimization. Furthermore, we introduce a plug-and-play reliability-based
weighted ensembling strategy, which aggregates useful information from reliable
augmented views to strengthen the defense. R-TPT enhances defense against
adversarial attacks without requiring labeled training data while offering high
flexibility for inference tasks. Extensive experiments on widely used
benchmarks with various attacks demonstrate the effectiveness of R-TPT. The
code is available in https://github.com/TomSheng21/R-TPT.


http://arxiv.org/abs/2504.11182
Exploring Backdoor Attack and Defense for LLM-empowered Recommendations. (92%)
Liangbo Ning; Wenqi Fan; Qing Li
  The fusion of Large Language Models (LLMs) with recommender systems (RecSys)
has dramatically advanced personalized recommendations and drawn extensive
attention. Despite the impressive progress, the safety of LLM-based RecSys
against backdoor attacks remains largely under-explored. In this paper, we
raise a new problem: Can a backdoor with a specific trigger be injected into
LLM-based Recsys, leading to the manipulation of the recommendation responses
when the backdoor trigger is appended to an item's title? To investigate the
vulnerabilities of LLM-based RecSys under backdoor attacks, we propose a new
attack framework termed Backdoor Injection Poisoning for RecSys (BadRec).
BadRec perturbs the items' titles with triggers and employs several fake users
to interact with these items, effectively poisoning the training set and
injecting backdoors into LLM-based RecSys. Comprehensive experiments reveal
that poisoning just 1% of the training data with adversarial examples is
sufficient to successfully implant backdoors, enabling manipulation of
recommendations. To further mitigate such a security threat, we propose a
universal defense strategy called Poison Scanner (P-Scanner). Specifically, we
introduce an LLM-based poison scanner to detect the poisoned items by
leveraging the powerful language understanding and rich knowledge of LLMs. A
trigger augmentation agent is employed to generate diverse synthetic triggers
to guide the poison scanner in learning domain-specific knowledge of the
poisoned item detection task. Extensive experiments on three real-world
datasets validate the effectiveness of the proposed P-Scanner.


http://arxiv.org/abs/2504.11034
Defending Against Frequency-Based Attacks with Diffusion Models. (88%)
Fatemeh Amerehi; Patrick Healy
  Adversarial training is a common strategy for enhancing model robustness
against adversarial attacks. However, it is typically tailored to the specific
attack types it is trained on, limiting its ability to generalize to unseen
threat models. Adversarial purification offers an alternative by leveraging a
generative model to remove perturbations before classification. Since the
purifier is trained independently of both the classifier and the threat models,
it is better equipped to handle previously unseen attack scenarios. Diffusion
models have proven highly effective for noise purification, not only in
countering pixel-wise adversarial perturbations but also in addressing
non-adversarial data shifts. In this study, we broaden the focus beyond
pixel-wise robustness to explore the extent to which purification can mitigate
both spectral and spatial adversarial attacks. Our findings highlight its
effectiveness in handling diverse distortion patterns across low- to
high-frequency regions.


http://arxiv.org/abs/2504.11038
QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models. (86%)
Yudong Zhang; Ruobing Xie; Jiansheng Chen; Xingwu Sun; Zhanhui Kang; Yu Wang
  In typical multimodal tasks, such as Visual Question Answering (VQA),
adversarial attacks targeting a specific image and question can lead large
vision-language models (LVLMs) to provide incorrect answers. However, it is
common for a single image to be associated with multiple questions, and LVLMs
may still answer other questions correctly even for an adversarial image
attacked by a specific question. To address this, we introduce the
query-agnostic visual attack (QAVA), which aims to create robust adversarial
examples that generate incorrect responses to unspecified and unknown
questions. Compared to traditional adversarial attacks focused on specific
images and questions, QAVA significantly enhances the effectiveness and
efficiency of attacks on images when the question is unknown, achieving
performance comparable to attacks on known target questions. Our research
broadens the scope of visual adversarial attacks on LVLMs in practical
settings, uncovering previously overlooked vulnerabilities, particularly in the
context of visual adversarial threats. The code is available at
https://github.com/btzyd/qava.


http://arxiv.org/abs/2504.11510
RAID: An In-Training Defense against Attribute Inference Attacks in Recommender Systems. (81%)
Xiaohua Feng; Yuyuan Li; Fengyuan Yu; Ke Xiong; Junjie Fang; Li Zhang; Tianyu Du; Chaochao Chen
  In various networks and mobile applications, users are highly susceptible to
attribute inference attacks, with particularly prevalent occurrences in
recommender systems. Attackers exploit partially exposed user profiles in
recommendation models, such as user embeddings, to infer private attributes of
target users, such as gender and political views. The goal of defenders is to
mitigate the effectiveness of these attacks while maintaining recommendation
performance. Most existing defense methods, such as differential privacy and
attribute unlearning, focus on post-training settings, which limits their
capability of utilizing training data to preserve recommendation performance.
Although adversarial training extends defenses to in-training settings, it
often struggles with convergence due to unstable training processes. In this
paper, we propose RAID, an in-training defense method against attribute
inference attacks in recommender systems. In addition to the recommendation
objective, we define a defensive objective to ensure that the distribution of
protected attributes becomes independent of class labels, making users
indistinguishable from attribute inference attacks. Specifically, this
defensive objective aims to solve a constrained Wasserstein barycenter problem
to identify the centroid distribution that makes the attribute
indistinguishable while complying with recommendation performance constraints.
To optimize our proposed objective, we use optimal transport to align users
with the centroid distribution. We conduct extensive experiments on four
real-world datasets to evaluate RAID. The experimental results validate the
effectiveness of RAID and demonstrate its significant superiority over existing
methods in multiple aspects.


http://arxiv.org/abs/2504.10969
RF Sensing Security and Malicious Exploitation: A Comprehensive Survey. (69%)
Mingda Han; Huanqi Yang; Wenhao Li; Weitao Xu; Xiuzhen Cheng; Prasant Mohapatra; Pengfei Hu
  Radio Frequency (RF) sensing technologies have experienced significant growth
due to the widespread adoption of RF devices and the Internet of Things (IoT).
These technologies enable numerous applications across healthcare, smart homes,
industrial automation, and human-computer interaction. However, the
non-intrusive and ubiquitous nature of RF sensing - combined with its
environmental sensitivity and data dependency - makes these systems inherently
vulnerable not only as attack targets, but also as powerful attack vectors.
This survey presents a comprehensive analysis of RF sensing security, covering
both system-level vulnerabilities - such as signal spoofing, adversarial
perturbations, and model poisoning - and the misuse of sensing capabilities for
attacks like cross-boundary surveillance, side-channel inference, and semantic
privacy breaches. We propose unified threat models to structure these attack
vectors and further conduct task-specific vulnerability assessments across key
RF sensing applications, identifying their unique attack surfaces and risk
profiles. In addition, we systematically review defense strategies across
system layers and threat-specific scenarios, incorporating both active and
passive paradigms to provide a structured and practical view of protection
mechanisms. Compared to prior surveys, our work distinguishes itself by
offering a multi-dimensional classification framework based on task type,
threat vector, and sensing modality, and by providing fine-grained,
scenario-driven analysis that bridges theoretical models and real-world
implications. This survey aims to serve as a comprehensive reference for
researchers and practitioners seeking to understand, evaluate, and secure the
evolving landscape of RF sensing technologies.


http://arxiv.org/abs/2504.10850
How to Enhance Downstream Adversarial Robustness (almost) without Touching the Pre-Trained Foundation Model? (41%)
Meiqi Liu; Zhuoqun Huang; Yue Xing
  With the rise of powerful foundation models, a pre-training-fine-tuning
paradigm becomes increasingly popular these days: A foundation model is
pre-trained using a huge amount of data from various sources, and then the
downstream users only need to fine-tune and adapt it to specific downstream
tasks. However, due to the high computation complexity of adversarial training,
it is not feasible to fine-tune the foundation model to improve its robustness
on the downstream task. Observing the above challenge, we want to improve the
downstream robustness without updating/accessing the weights in the foundation
model. Inspired from existing literature in robustness inheritance (Kim et al.,
2020), through theoretical investigation, we identify a close relationship
between robust contrastive learning with the adversarial robustness of
supervised learning. To further validate and utilize this theoretical insight,
we design a simple-yet-effective robust auto-encoder as a data pre-processing
method before feeding the data into the foundation model. The proposed approach
has zero access to the foundation model when training the robust auto-encoder.
Extensive experiments demonstrate the effectiveness of the proposed method in
improving the robustness of downstream tasks, verifying the connection between
the feature robustness (implied by small adversarial contrastive loss) and the
robustness of the downstream task.


http://arxiv.org/abs/2504.11106
Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Models. (38%)
Jiangtao Liu; Zhaoxin Wang; Handing Wang; Cong Tian; Yaochu Jin
  Recent advancements in Text-to-Image (T2I) generation have significantly
enhanced the realism and creativity of generated images. However, such powerful
generative capabilities pose risks related to the production of inappropriate
or harmful content. Existing defense mechanisms, including prompt checkers and
post-hoc image checkers, are vulnerable to sophisticated adversarial attacks.
In this work, we propose TCBS-Attack, a novel query-based black-box jailbreak
attack that searches for tokens located near the decision boundaries defined by
text and image checkers. By iteratively optimizing tokens near these
boundaries, TCBS-Attack generates semantically coherent adversarial prompts
capable of bypassing multiple defensive layers in T2I models. Extensive
experiments demonstrate that our method consistently outperforms
state-of-the-art jailbreak attacks across various T2I models, including
securely trained open-source models and commercial online services like DALL-E
3. TCBS-Attack achieves an ASR-4 of 45\% and an ASR-1 of 21\% on jailbreaking
full-chain T2I models, significantly surpassing baseline methods.


http://arxiv.org/abs/2504.11281
The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injections. (9%)
Chaoran Chen; Zhiping Zhang; Bingcan Guo; Shang Ma; Ibrahim Khalilov; Simret A Gebreegziabher; Yanfang Ye; Ziang Xiao; Yaxing Yao; Tianshi Li; Toby Jia-Jun Li
  A Large Language Model (LLM) powered GUI agent is a specialized autonomous
system that performs tasks on the user's behalf according to high-level
instructions. It does so by perceiving and interpreting the graphical user
interfaces (GUIs) of relevant apps, often visually, inferring necessary
sequences of actions, and then interacting with GUIs by executing the actions
such as clicking, typing, and tapping. To complete real-world tasks, such as
filling forms or booking services, GUI agents often need to process and act on
sensitive user data. However, this autonomy introduces new privacy and security
risks. Adversaries can inject malicious content into the GUIs that alters agent
behaviors or induces unintended disclosures of private information. These
attacks often exploit the discrepancy between visual saliency for agents and
human users, or the agent's limited ability to detect violations of contextual
integrity in task automation. In this paper, we characterized six types of such
attacks, and conducted an experimental study to test these attacks with six
state-of-the-art GUI agents, 234 adversarial webpages, and 39 human
participants. Our findings suggest that GUI agents are highly vulnerable,
particularly to contextually embedded threats. Moreover, human users are also
susceptible to many of these attacks, indicating that simple human oversight
may not reliably prevent failures. This misalignment highlights the need for
privacy-aware agent design. We propose practical defense strategies to inform
the development of safer and more reliable GUI agents.


http://arxiv.org/abs/2504.11633
Chypnosis: Stealthy Secret Extraction using Undervolting-based Static Side-channel Attacks. (1%)
Kyle Mitard; Saleh Khalaj Monfared; Fatemeh Khojasteh Dana; Shahin Tajik
  There is a growing class of static physical side-channel attacks that allow
adversaries to extract secrets by probing the persistent state of a circuit.
Techniques such as laser logic state imaging (LLSI), impedance analysis (IA),
and static power analysis fall into this category. These attacks require that
the targeted data remain constant for a specific duration, which often
necessitates halting the circuit's clock. Some methods additionally rely on
modulating the chip's supply voltage to probe the circuit. However, tampering
with the clock or voltage is typically assumed to be detectable, as secure
chips often deploy sensors that erase sensitive data upon detecting such
anomalies. Furthermore, many secure devices use internal clock sources, making
external clock control infeasible. In this work, we introduce a novel class of
static side-channel attacks, called Chypnosis, that enables adversaries to
freeze a chip's internal clock by inducing a hibernation state via rapid
undervolting, and then extracting secrets using static side-channels. We
demonstrate that, by rapidly dropping a chip's voltage below the standard
nominal levels, the attacker can bypass the clock and voltage sensors and put
the chip in a so-called brownout condition, in which the chip's transistors
stop switching, but volatile memories (e.g., Flip-flops and SRAMs) still retain
their data. We test our attack on AMD FPGAs by putting them into hibernation.
We show that not only are all clock sources deactivated, but various clock and
voltage sensors also fail to detect the tamper event. Afterward, we present the
successful recovery of secret bits from a hibernated chip using two static
attacks, namely, LLSI and IA. Finally, we discuss potential countermeasures
which could be integrated into future designs.


http://arxiv.org/abs/2504.10804
The Sword of Damocles in ViTs: Computational Redundancy Amplifies Adversarial Transferability. (99%)
Jiani Liu; Zhiyuan Wang; Zeliang Zhang; Chao Huang; Susan Liang; Yunlong Tang; Chenliang Xu
  Vision Transformers (ViTs) have demonstrated impressive performance across a
range of applications, including many safety-critical tasks. However, their
unique architectural properties raise new challenges and opportunities in
adversarial robustness. In particular, we observe that adversarial examples
crafted on ViTs exhibit higher transferability compared to those crafted on
CNNs, suggesting that ViTs contain structural characteristics favorable for
transferable attacks. In this work, we investigate the role of computational
redundancy in ViTs and its impact on adversarial transferability. Unlike prior
studies that aim to reduce computation for efficiency, we propose to exploit
this redundancy to improve the quality and transferability of adversarial
examples. Through a detailed analysis, we identify two forms of redundancy,
including the data-level and model-level, that can be harnessed to amplify
attack effectiveness. Building on this insight, we design a suite of
techniques, including attention sparsity manipulation, attention head
permutation, clean token regularization, ghost MoE diversification, and
test-time adversarial training. Extensive experiments on the ImageNet-1k
dataset validate the effectiveness of our approach, showing that our methods
significantly outperform existing baselines in both transferability and
generality across diverse model architectures.


http://arxiv.org/abs/2504.13196
Investigating cybersecurity incidents using large language models in latest-generation wireless networks. (83%)
Leonid Legashev; Arthur Zhigalov
  The purpose of research: Detection of cybersecurity incidents and analysis of
decision support and assessment of the effectiveness of measures to counter
information security threats based on modern generative models. The methods of
research: Emulation of signal propagation data in MIMO systems, synthesis of
adversarial examples, execution of adversarial attacks on machine learning
models, fine tuning of large language models for detecting adversarial attacks,
explainability of decisions on detecting cybersecurity incidents based on the
prompts technique. Scientific novelty: A binary classification of data
poisoning attacks was performed using large language models, and the
possibility of using large language models for investigating cybersecurity
incidents in the latest generation wireless networks was investigated. The
result of research: Fine-tuning of large language models was performed on the
prepared data of the emulated wireless network segment. Six large language
models were compared for detecting adversarial attacks, and the capabilities of
explaining decisions made by a large language model were investigated. The
Gemma-7b model showed the best results according to the metrics Precision =
0.89, Recall = 0.89 and F1-Score = 0.89. Based on various explainability
prompts, the Gemma-7b model notes inconsistencies in the compromised data under
study, performs feature importance analysis and provides various
recommendations for mitigating the consequences of adversarial attacks. Large
language models integrated with binary classifiers of network threats have
significant potential for practical application in the field of cybersecurity
incident investigation, decision support and assessing the effectiveness of
measures to counter information security threats.


http://arxiv.org/abs/2504.10016
Quantifying Privacy Leakage in Split Inference via Fisher-Approximated Shannon Information Analysis. (73%)
Ruijun Deng; Zhihui Lu; Qiang Duan
  Split inference (SI) partitions deep neural networks into distributed
sub-models, enabling privacy-preserving collaborative learning. Nevertheless,
it remains vulnerable to Data Reconstruction Attacks (DRAs), wherein
adversaries exploit exposed smashed data to reconstruct raw inputs. Despite
extensive research on adversarial attack-defense games, a shortfall remains in
the fundamental analysis of privacy risks. This paper establishes a theoretical
framework for privacy leakage quantification using information theory, defining
it as the adversary's certainty and deriving both average-case and worst-case
error bounds. We introduce Fisher-approximated Shannon information (FSInfo), a
novel privacy metric utilizing Fisher Information (FI) for operational privacy
leakage computation. We empirically show that our privacy metric correlates
well with empirical attacks and investigate some of the factors that affect
privacy leakage, namely the data distribution, model size, and overfitting.


http://arxiv.org/abs/2504.10833
Towards Spatially-Aware and Optimally Faithful Concept-Based Explanations. (56%)
Shubham Kumar; Dwip Dalal; Narendra Ahuja
  Post-hoc, unsupervised concept-based explanation methods (U-CBEMs) are a
promising tool for generating semantic explanations of the decision-making
processes in deep neural networks, having applications in both model
improvement and understanding. It is vital that the explanation is accurate, or
faithful, to the model, yet we identify several limitations of prior
faithfulness metrics that inhibit an accurate evaluation; most notably, prior
metrics involve only the set of concepts present, ignoring how they may be
spatially distributed. We address these limitations with Surrogate Faithfulness
(SF), an evaluation method that introduces a spatially-aware surrogate and two
novel faithfulness metrics. Using SF, we produce Optimally Faithful (OF)
explanations, where concepts are found that maximize faithfulness. Our
experiments show that (1) adding spatial-awareness to prior U-CBEMs increases
faithfulness in all cases; (2) OF produces significantly more faithful
explanations than prior U-CBEMs (30% or higher improvement in error); (3) OF's
learned concepts generalize well to out-of-domain data and are more robust to
adversarial examples, where prior U-CBEMs struggle.


http://arxiv.org/abs/2504.13201
Concept Enhancement Engineering: A Lightweight and Efficient Robust Defense Against Jailbreak Attacks in Embodied AI. (45%)
Jirui Yang; Zheyu Lin; Shuhan Yang; Zhihui Lu; Xin Du
  Embodied Intelligence (EI) systems integrated with large language models
(LLMs) face significant security risks, particularly from jailbreak attacks
that manipulate models into generating harmful outputs or executing unsafe
physical actions. Traditional defense strategies, such as input filtering and
output monitoring, often introduce high computational overhead or interfere
with task performance in real-time embodied scenarios. To address these
challenges, we propose Concept Enhancement Engineering (CEE), a novel defense
framework that leverages representation engineering to enhance the safety of
embodied LLMs by dynamically steering their internal activations. CEE operates
by (1) extracting multilingual safety patterns from model activations, (2)
constructing control directions based on safety-aligned concept subspaces, and
(3) applying subspace concept rotation to reinforce safe behavior during
inference. Our experiments demonstrate that CEE effectively mitigates jailbreak
attacks while maintaining task performance, outperforming existing defense
methods in both robustness and efficiency. This work contributes a scalable and
interpretable safety mechanism for embodied AI, bridging the gap between
theoretical representation engineering and practical security applications. Our
findings highlight the potential of latent-space interventions as a viable
defense paradigm against emerging adversarial threats in physically grounded AI
systems.


http://arxiv.org/abs/2504.10318
Shield Bash: Abusing Defensive Coherence State Retrieval to Break Timing Obfuscation. (10%)
Kartik Ramkrishnan; Antonia Zhai; Stephen McCamant; Pen Chung Yew
  Microarchitectural attacks are a significant concern, leading to many
hardware-based defense proposals. However, different defenses target different
classes of attacks, and their impact on each other has not been fully
considered. To raise awareness of this problem, we study an interaction between
two state-of-the art defenses in this paper, timing obfuscations of remote
cache lines (TORC) and delaying speculative changes to remote cache lines
(DSRC). TORC mitigates cache-hit based attacks and DSRC mitigates speculative
coherence state change attacks.
  We observe that DSRC enables coherence information to be retrieved into the
processor core, where it is out of the reach of timing obfuscations to protect.
This creates an unforeseen consequence that redo operations can be triggered
within the core to detect the presence or absence of remote cache lines, which
constitutes a security vulnerability. We demonstrate that a new covert channel
attack is possible using this vulnerability. We propose two ways to mitigate
the attack, whose performance varies depending on an application's cache usage.
One way is to never send remote exclusive coherence state (E) information to
the core even if it is created. The other way is to never create a remote E
state, which is responsible for triggering redos.
  We demonstrate the timing difference caused by this microarchitectural
defense assumption violation using GEM5 simulations. Performance evaluation on
SPECrate 2017 and PARSEC benchmarks of the two fixes show less than 32\%
average overhead across both sets of benchmarks. The repair which prevented the
creation of remote E state had less than 2.8% average overhead.


http://arxiv.org/abs/2504.10782
Deep Audio Watermarks are Shallow: Limitations of Post-Hoc Watermarking Techniques for Speech. (1%)
Patrick O'Reilly; Zeyu Jin; Jiaqi Su; Bryan Pardo
  In the audio modality, state-of-the-art watermarking methods leverage deep
neural networks to allow the embedding of human-imperceptible signatures in
generated audio. The ideal is to embed signatures that can be detected with
high accuracy when the watermarked audio is altered via compression, filtering,
or other transformations. Existing audio watermarking techniques operate in a
post-hoc manner, manipulating "low-level" features of audio recordings after
generation (e.g. through the addition of a low-magnitude watermark signal). We
show that this post-hoc formulation makes existing audio watermarks vulnerable
to transformation-based removal attacks. Focusing on speech audio, we (1) unify
and extend existing evaluations of the effect of audio transformations on
watermark detectability, and (2) demonstrate that state-of-the-art post-hoc
audio watermarks can be removed with no knowledge of the watermarking scheme
and minimal degradation in audio quality.


http://arxiv.org/abs/2504.10374
Ctrl-Z: Controlling AI Agents via Resampling. (1%)
Aryan Bhatt; Cody Rushing; Adam Kaufman; Tyler Tracy; Vasil Georgiev; David Matolcsi; Akbir Khan; Buck Shlegeris
  Control evaluations measure whether monitoring and security protocols for AI
systems prevent intentionally subversive AI models from causing harm. Our work
presents the first control evaluation performed in an agent environment. We
construct BashBench, a dataset of 257 challenging multi-step system
administration tasks, and evaluate whether various safety measures can prevent
an adversarially constructed AI agent from covertly downloading and executing
malicious code in this environment. This multi-step setting introduces new
attack and defense dynamics, which we investigate in order to design novel
control protocols that prevent safety failures without hindering the ability of
non-malicious agents to perform useful work. We introduce a class of control
protocols called resample protocols that dynamically take additional samples of
certain actions. We find these protocols significantly improve on existing
techniques by selectively blocking the AI agent from executing suspicious code
and incriminating the agent by generating additional examples of dangerous
behavior. We measure the tradeoff between attack prevention and usefulness; our
best protocol combines resampling with analysis of previous steps, reducing the
success rate of attacks from 58% to 7% at a 5% cost to the performance of a
non-malicious agent.


http://arxiv.org/abs/2504.10598
Beyond Worst-Case Online Classification: VC-Based Regret Bounds for Relaxed Benchmarks. (1%)
Omar Montasser; Abhishek Shetty; Nikita Zhivotovskiy
  We revisit online binary classification by shifting the focus from competing
with the best-in-class binary loss to competing against relaxed benchmarks that
capture smoothed notions of optimality. Instead of measuring regret relative to
the exact minimal binary error -- a standard approach that leads to worst-case
bounds tied to the Littlestone dimension -- we consider comparing with
predictors that are robust to small input perturbations, perform well under
Gaussian smoothing, or maintain a prescribed output margin. Previous examples
of this were primarily limited to the hinge loss. Our algorithms achieve regret
guarantees that depend only on the VC dimension and the complexity of the
instance space (e.g., metric entropy), and notably, they incur only an
$O(\log(1/\gamma))$ dependence on the generalized margin $\gamma$. This stands
in contrast to most existing regret bounds, which typically exhibit a
polynomial dependence on $1/\gamma$. We complement this with matching lower
bounds. Our analysis connects recent ideas from adversarial robustness and
smoothed online learning.


http://arxiv.org/abs/2504.09839
SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis. (92%)
Zhisheng Zhang; Derui Wang; Qianyi Yang; Pengyang Huang; Junhan Pu; Yuxin Cao; Kai Ye; Jie Hao; Yixian Yang
  Speech synthesis technology has brought great convenience, while the
widespread usage of realistic deepfake audio has triggered hazards. Malicious
adversaries may unauthorizedly collect victims' speeches and clone a similar
voice for illegal exploitation (\textit{e.g.}, telecom fraud). However, the
existing defense methods cannot effectively prevent deepfake exploitation and
are vulnerable to robust training techniques. Therefore, a more effective and
robust data protection method is urgently needed. In response, we propose a
defensive framework, \textit{\textbf{SafeSpeech}}, which protects the users'
audio before uploading by embedding imperceptible perturbations on original
speeches to prevent high-quality synthetic speech. In SafeSpeech, we devise a
robust and universal proactive protection technique, \textbf{S}peech
\textbf{PE}rturbative \textbf{C}oncealment (\textbf{SPEC}), that leverages a
surrogate model to generate universally applicable perturbation for generative
synthetic models. Moreover, we optimize the human perception of embedded
perturbation in terms of time and frequency domains. To evaluate our method
comprehensively, we conduct extensive experiments across advanced models and
datasets, both subjectively and objectively. Our experimental results
demonstrate that SafeSpeech achieves state-of-the-art (SOTA) voice protection
effectiveness and transferability and is highly robust against advanced
adaptive adversaries. Moreover, SafeSpeech has real-time capability in
real-world tests. The source code is available at
\href{https://github.com/wxzyd123/SafeSpeech}{https://github.com/wxzyd123/SafeSpeech}.


http://arxiv.org/abs/2504.13192
CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agent. (86%)
Liang-bo Ning; Shijie Wang; Wenqi Fan; Qing Li; Xin Xu; Hao Chen; Feiran Huang
  Recently, Large Language Model (LLM)-empowered recommender systems (RecSys)
have brought significant advances in personalized user experience and have
attracted considerable attention. Despite the impressive progress, the research
question regarding the safety vulnerability of LLM-empowered RecSys still
remains largely under-investigated. Given the security and privacy concerns, it
is more practical to focus on attacking the black-box RecSys, where attackers
can only observe the system's inputs and outputs. However, traditional attack
approaches employing reinforcement learning (RL) agents are not effective for
attacking LLM-empowered RecSys due to the limited capabilities in processing
complex textual inputs, planning, and reasoning. On the other hand, LLMs
provide unprecedented opportunities to serve as attack agents to attack RecSys
because of their impressive capability in simulating human-like decision-making
processes. Therefore, in this paper, we propose a novel attack framework called
CheatAgent by harnessing the human-like capabilities of LLMs, where an
LLM-based agent is developed to attack LLM-Empowered RecSys. Specifically, our
method first identifies the insertion position for maximum impact with minimal
input modification. After that, the LLM agent is designed to generate
adversarial perturbations to insert at target positions. To further improve the
quality of generated perturbations, we utilize the prompt tuning technique to
improve attacking strategies via feedback from the victim RecSys iteratively.
Extensive experiments across three real-world datasets demonstrate the
effectiveness of our proposed attacking method.


http://arxiv.org/abs/2504.09648
RANSAC Revisited: An Improved Algorithm for Robust Subspace Recovery under Adversarial and Noisy Corruptions. (9%)
Guixian Chen; Jianhao Ma; Salar Fattahi
  In this paper, we study the problem of robust subspace recovery (RSR) in the
presence of both strong adversarial corruptions and Gaussian noise.
Specifically, given a limited number of noisy samples -- some of which are
tampered by an adaptive and strong adversary -- we aim to recover a
low-dimensional subspace that approximately contains a significant fraction of
the uncorrupted samples, up to an error that scales with the Gaussian noise.
Existing approaches to this problem often suffer from high computational costs
or rely on restrictive distributional assumptions, limiting their applicability
in truly adversarial settings. To address these challenges, we revisit the
classical random sample consensus (RANSAC) algorithm, which offers strong
robustness to adversarial outliers, but sacrifices efficiency and robustness
against Gaussian noise and model misspecification in the process. We propose a
two-stage algorithm, RANSAC+, that precisely pinpoints and remedies the failure
modes of standard RANSAC. Our method is provably robust to both Gaussian and
adversarial corruptions, achieves near-optimal sample complexity without
requiring prior knowledge of the subspace dimension, and is more efficient than
existing RANSAC-type methods.


http://arxiv.org/abs/2504.09593
ControlNET: A Firewall for RAG-based LLM System. (9%)
Hongwei Yao; Haoran Shi; Yidou Chen; Yixin Jiang; Cong Wang; Zhan Qin
  Retrieval-Augmented Generation (RAG) has significantly enhanced the factual
accuracy and domain adaptability of Large Language Models (LLMs). This
advancement has enabled their widespread deployment across sensitive domains
such as healthcare, finance, and enterprise applications. RAG mitigates
hallucinations by integrating external knowledge, yet introduces privacy risk
and security risk, notably data breaching risk and data poisoning risk. While
recent studies have explored prompt injection and poisoning attacks, there
remains a significant gap in comprehensive research on controlling inbound and
outbound query flows to mitigate these threats. In this paper, we propose an AI
firewall, ControlNET, designed to safeguard RAG-based LLM systems from these
vulnerabilities. ControlNET controls query flows by leveraging activation shift
phenomena to detect adversarial queries and mitigate their impact through
semantic divergence. We conduct comprehensive experiments on four different
benchmark datasets including Msmarco, HotpotQA, FinQA, and MedicalSys using
state-of-the-art open source LLMs (Llama3, Vicuna, and Mistral). Our results
demonstrate that ControlNET achieves over 0.909 AUROC in detecting and
mitigating security threats while preserving system harmlessness. Overall,
ControlNET offers an effective, robust, harmless defense mechanism, marking a
significant advancement toward the secure deployment of RAG-based LLM systems.


http://arxiv.org/abs/2504.09776
An Investigation of Large Language Models and Their Vulnerabilities in Spam Detection. (5%)
Qiyao Tang; Xiangyang Li
  Spam messages continue to present significant challenges to digital users,
cluttering inboxes and posing security risks. Traditional spam detection
methods, including rules-based, collaborative, and machine learning approaches,
struggle to keep up with the rapidly evolving tactics employed by spammers.
This project studies new spam detection systems that leverage Large Language
Models (LLMs) fine-tuned with spam datasets. More importantly, we want to
understand how LLM-based spam detection systems perform under adversarial
attacks that purposefully modify spam emails and data poisoning attacks that
exploit the differences between the training data and the massages in
detection, to which traditional machine learning models are shown to be
vulnerable. This experimentation employs two LLM models of GPT2 and BERT and
three spam datasets of Enron, LingSpam, and SMSspamCollection for extensive
training and testing tasks. The results show that, while they can function as
effective spam filters, the LLM models are susceptible to the adversarial and
data poisoning attacks. This research provides very useful insights for future
applications of LLM models for information security.


http://arxiv.org/abs/2504.09466
AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender. (2%)
Weixiang Zhao; Jiahe Guo; Yulin Hu; Yang Deng; An Zhang; Xingyu Sui; Xinyang Han; Yanyan Zhao; Bing Qin; Tat-Seng Chua; Ting Liu
  Despite extensive efforts in safety alignment, large language models (LLMs)
remain vulnerable to jailbreak attacks. Activation steering offers a
training-free defense method but relies on fixed steering coefficients,
resulting in suboptimal protection and increased false rejections of benign
inputs. To address this, we propose AdaSteer, an adaptive activation steering
method that dynamically adjusts model behavior based on input characteristics.
We identify two key properties: Rejection Law (R-Law), which shows that
stronger steering is needed for jailbreak inputs opposing the rejection
direction, and Harmfulness Law (H-Law), which differentiates adversarial and
benign inputs. AdaSteer steers input representations along both the Rejection
Direction (RD) and Harmfulness Direction (HD), with adaptive coefficients
learned via logistic regression, ensuring robust jailbreak defense while
preserving benign input handling. Experiments on LLaMA-3.1, Gemma-2, and
Qwen2.5 show that AdaSteer outperforms baseline methods across multiple
jailbreak attacks with minimal impact on utility. Our results highlight the
potential of interpretable model internals for real-time, flexible safety
enforcement in LLMs.


http://arxiv.org/abs/2504.09451
FractalForensics: Proactive Deepfake Detection and Localization via Fractal Watermarks. (1%)
Tianyi Wang; Harry Cheng; Ming-Hui Liu; Mohan Kankanhalli
  Proactive Deepfake detection via robust watermarks has been raised ever since
passive Deepfake detectors encountered challenges in identifying high-quality
synthetic images. However, while demonstrating reasonable detection
performance, they lack localization functionality and explainability in
detection results. Additionally, the unstable robustness of watermarks can
significantly affect the detection performance accordingly. In this study, we
propose novel fractal watermarks for proactive Deepfake detection and
localization, namely FractalForensics. Benefiting from the characteristics of
fractals, we devise a parameter-driven watermark generation pipeline that
derives fractal-based watermarks and conducts one-way encryption regarding the
parameters selected. Subsequently, we propose a semi-fragile watermarking
framework for watermark embedding and recovery, trained to be robust against
benign image processing operations and fragile when facing Deepfake
manipulations in a black-box setting. Meanwhile, we introduce an entry-to-patch
strategy that implicitly embeds the watermark matrix entries into image patches
at corresponding positions, achieving localization of Deepfake manipulations.
Extensive experiments demonstrate satisfactory robustness and fragility of our
approach against common image processing operations and Deepfake manipulations,
outperforming state-of-the-art semi-fragile watermarking algorithms and passive
detectors for Deepfake detection. Furthermore, by highlighting the areas
manipulated, our method provides explainability for the proactive Deepfake
detection results.


http://arxiv.org/abs/2504.09712
The Structural Safety Generalization Problem. (1%)
Julius Broomfield; Tom Gibbs; Ethan Kosak-Hine; George Ingebretsen; Tia Nasir; Jason Zhang; Reihaneh Iranmanesh; Sara Pieri; Reihaneh Rabbany; Kellin Pelrine
  LLM jailbreaks are a widespread safety challenge. Given this problem has not
yet been tractable, we suggest targeting a key failure mechanism: the failure
of safety to generalize across semantically equivalent inputs. We further focus
the target by requiring desirable tractability properties of attacks to study:
explainability, transferability between models, and transferability between
goals. We perform red-teaming within this framework by uncovering new
vulnerabilities to multi-turn, multi-image, and translation-based attacks.
These attacks are semantically equivalent by our design to their single-turn,
single-image, or untranslated counterparts, enabling systematic comparisons; we
show that the different structures yield different safety outcomes. We then
demonstrate the potential for this framework to enable new defenses by
proposing a Structure Rewriting Guardrail, which converts an input to a
structure more conducive to safety assessment. This guardrail significantly
improves refusal of harmful inputs, without over-refusing benign ones. Thus, by
framing this intermediate challenge - more tractable than universal defenses
but essential for long-term safety - we highlight a critical milestone for AI
safety research.


http://arxiv.org/abs/2504.09361
PapMOT: Exploring Adversarial Patch Attack against Multiple Object Tracking. (98%)
Jiahuan Long; Tingsong Jiang; Wen Yao; Shuai Jia; Weijia Zhang; Weien Zhou; Chao Ma; Xiaoqian Chen
  Tracking multiple objects in a continuous video stream is crucial for many
computer vision tasks. It involves detecting and associating objects with their
respective identities across successive frames. Despite significant progress
made in multiple object tracking (MOT), recent studies have revealed the
vulnerability of existing MOT methods to adversarial attacks. Nevertheless, all
of these attacks belong to digital attacks that inject pixel-level noise into
input images, and are therefore ineffective in physical scenarios. To fill this
gap, we propose PapMOT, which can generate physical adversarial patches against
MOT for both digital and physical scenarios. Besides attacking the detection
mechanism, PapMOT also optimizes a printable patch that can be detected as new
targets to mislead the identity association process. Moreover, we introduce a
patch enhancement strategy to further degrade the temporal consistency of
tracking results across video frames, resulting in more aggressive attacks. We
further develop new evaluation metrics to assess the robustness of MOT against
such attacks. Extensive evaluations on multiple datasets demonstrate that our
PapMOT can successfully attack various architectures of MOT trackers in digital
scenarios. We also validate the effectiveness of PapMOT for physical attacks by
deploying printed adversarial patches in the real world.


http://arxiv.org/abs/2504.09202
From Visual Explanations to Counterfactual Explanations with Latent Diffusion. (75%)
Tung Luu; Nam Le; Duc Le; Bac Le
  Visual counterfactual explanations are ideal hypothetical images that change
the decision-making of the classifier with high confidence toward the desired
class while remaining visually plausible and close to the initial image. In
this paper, we propose a new approach to tackle two key challenges in recent
prominent works: i) determining which specific counterfactual features are
crucial for distinguishing the "concept" of the target class from the original
class, and ii) supplying valuable explanations for the non-robust classifier
without relying on the support of an adversarially robust model. Our method
identifies the essential region for modification through algorithms that
provide visual explanations, and then our framework generates realistic
counterfactual explanations by combining adversarial attacks based on pruning
the adversarial gradient of the target classifier and the latent diffusion
model. The proposed method outperforms previous state-of-the-art results on
various evaluation criteria on ImageNet and CelebA-HQ datasets. In general, our
method can be applied to arbitrary classifiers, highlight the strong
association between visual and counterfactual explanations, make semantically
meaningful changes from the target classifier, and provide observers with
subtle counterfactual images.


http://arxiv.org/abs/2504.09409
Bregman Linearized Augmented Lagrangian Method for Nonconvex Constrained Stochastic Zeroth-order Optimization. (68%)
Qiankun Shi; Xiao Wang; Hao Wang
  In this paper, we study nonconvex constrained stochastic zeroth-order
optimization problems, for which we have access to exact information of
constraints and noisy function values of the objective. We propose a Bregman
linearized augmented Lagrangian method that utilizes stochastic zeroth-order
gradient estimators combined with a variance reduction technique. We analyze
its oracle complexity, in terms of the total number of stochastic function
value evaluations required to achieve an \(\epsilon\)-KKT point in
\(\ell_p\)-norm metrics with \(p \ge 2\), where \(p\) is a parameter associated
with the selected Bregman distance. In particular, starting from a
near-feasible initial point and using Rademacher smoothing, the oracle
complexity is in order \(O(p d^{2/p} \epsilon^{-3})\) for \(p \in [2, 2 \ln
d]\), and \(O(\ln d \cdot \epsilon^{-3})\) for \(p > 2 \ln d\), where \(d\)
denotes the problem dimension. Those results show that the complexity of the
proposed method can achieve a dimensional dependency lower than \(O(d)\)
without requiring additional assumptions, provided that a Bregman distance is
chosen properly. This offers a significant improvement in the high-dimensional
setting over existing work, and matches the lowest complexity order with
respect to the tolerance \(\epsilon\) reported in the literature. Numerical
experiments on constrained Lasso and black-box adversarial attack problems
highlight the promising performances of the proposed method.


http://arxiv.org/abs/2504.09192
Towards More Efficient, Robust, Instance-adaptive, and Generalizable Sequential Decision making. (1%)
Zhiyong Wang
  The primary goal of my Ph.D. study is to develop provably efficient and
practical algorithms for data-driven sequential decision-making under
uncertainty. My work focuses on reinforcement learning (RL), multi-armed
bandits, and their applications, including recommendation systems, computer
networks, video analytics, and large language models (LLMs). Sequential
decision-making methods, such as bandits and RL, have demonstrated remarkable
success - ranging from outperforming human players in complex games like Atari
and Go to advancing robotics, recommendation systems, and fine-tuning LLMs.
Despite these successes, many established algorithms rely on idealized models
that can fail under model misspecifications or adversarial perturbations,
particularly in settings where accurate prior knowledge of the underlying model
class is unavailable or where malicious users operate within dynamic systems.
These challenges are pervasive in real-world applications, where robust and
adaptive solutions are critical. Furthermore, while worst-case guarantees
provide theoretical reliability, they often fail to capture instance-dependent
performance, which can lead to more efficient and practical solutions. Another
key challenge lies in generalizing to new, unseen environments, a crucial
requirement for deploying these methods in dynamic and unpredictable settings.
To address these limitations, my research aims to develop more efficient,
robust, instance-adaptive, and generalizable sequential decision-making
algorithms for both reinforcement learning and bandits. Towards this end, I
focus on developing more efficient, robust, instance-adaptive, and
generalizable for both general reinforcement learning (RL) and bandits.


http://arxiv.org/abs/2504.08414
Adversarial Examples in Environment Perception for Automated Driving (Review). (99%)
Jun Yan; Huilin Yin
  The renaissance of deep learning has led to the massive development of
automated driving. However, deep neural networks are vulnerable to adversarial
examples. The perturbations of adversarial examples are imperceptible to human
eyes but can lead to the false predictions of neural networks. It poses a huge
risk to artificial intelligence (AI) applications for automated driving. This
survey systematically reviews the development of adversarial robustness
research over the past decade, including the attack and defense methods and
their applications in automated driving. The growth of automated driving pushes
forward the realization of trustworthy AI applications. This review lists
significant references in the research history of adversarial examples.


http://arxiv.org/abs/2504.08480
Toward Realistic Adversarial Attacks in IDS: A Novel Feasibility Metric for Transferability. (99%)
Sabrine Ennaji; Elhadj Benkhelifa; Luigi Vincenzo Mancini
  Transferability-based adversarial attacks exploit the ability of adversarial
examples, crafted to deceive a specific source Intrusion Detection System (IDS)
model, to also mislead a target IDS model without requiring access to the
training data or any internal model parameters. These attacks exploit common
vulnerabilities in machine learning models to bypass security measures and
compromise systems. Although the transferability concept has been widely
studied, its practical feasibility remains limited due to assumptions of high
similarity between source and target models. This paper analyzes the core
factors that contribute to transferability, including feature alignment, model
architectural similarity, and overlap in the data distributions that each IDS
examines. We propose a novel metric, the Transferability Feasibility Score
(TFS), to assess the feasibility and reliability of such attacks based on these
factors. Through experimental evidence, we demonstrate that TFS and actual
attack success rates are highly correlated, addressing the gap between
theoretical understanding and real-world impact. Our findings provide needed
guidance for designing more realistic transferable adversarial attacks,
developing robust defenses, and ultimately improving the security of machine
learning-based IDS in critical systems.


http://arxiv.org/abs/2504.08906
Robust SAM: On the Adversarial Robustness of Vision Foundation Models. (98%)
Jiahuan Long; Zhengqin Xu; Tingsong Jiang; Wen Yao; Shuai Jia; Chao Ma; Xiaoqian Chen
  The Segment Anything Model (SAM) is a widely used vision foundation model
with diverse applications, including image segmentation, detection, and
tracking. Given SAM's wide applications, understanding its robustness against
adversarial attacks is crucial for real-world deployment. However, research on
SAM's robustness is still in its early stages. Existing attacks often overlook
the role of prompts in evaluating SAM's robustness, and there has been
insufficient exploration of defense methods to balance the robustness and
accuracy. To address these gaps, this paper proposes an adversarial robustness
framework designed to evaluate and enhance the robustness of SAM. Specifically,
we introduce a cross-prompt attack method to enhance the attack transferability
across different prompt types. Besides attacking, we propose a few-parameter
adaptation strategy to defend SAM against various adversarial attacks. To
balance robustness and accuracy, we use the singular value decomposition (SVD)
to constrain the space of trainable parameters, where only singular values are
adaptable. Experiments demonstrate that our cross-prompt attack method
outperforms previous approaches in terms of attack success rate on both SAM and
SAM 2. By adapting only 512 parameters, we achieve at least a 15\% improvement
in mean intersection over union (mIoU) against various adversarial attacks.
Compared to previous defense methods, our approach enhances the robustness of
SAM while maximally maintaining its original performance.


http://arxiv.org/abs/2504.08866
On Transfer-based Universal Attacks in Pure Black-box Setting. (92%)
Mohammad A. A. K. Jalwana; Naveed Akhtar; Ajmal Mian; Nazanin Rahnavard; Mubarak Shah
  Despite their impressive performance, deep visual models are susceptible to
transferable black-box adversarial attacks. Principally, these attacks craft
perturbations in a target model-agnostic manner. However, surprisingly, we find
that existing methods in this domain inadvertently take help from various
priors that violate the black-box assumption such as the availability of the
dataset used to train the target model, and the knowledge of the number of
classes in the target model. Consequently, the literature fails to articulate
the true potency of transferable black-box attacks. We provide an empirical
study of these biases and propose a framework that aids in a prior-free
transparent study of this paradigm. Using our framework, we analyze the role of
prior knowledge of the target model data and number of classes in attack
performance. We also provide several interesting insights based on our
analysis, and demonstrate that priors cause overestimation in transferability
scores. Finally, we extend our framework to query-based attacks. This extension
inspires a novel image-blending technique to prepare data for effective
surrogate model training.


http://arxiv.org/abs/2504.08411
A Knowledge-guided Adversarial Defense for Resisting Malicious Visual Manipulation. (83%)
Dawei Zhou; Suzhi Gang; Decheng Liu; Tongliang Liu; Nannan Wang; Xinbo Gao
  Malicious applications of visual manipulation have raised serious threats to
the security and reputation of users in many fields. To alleviate these issues,
adversarial noise-based defenses have been enthusiastically studied in recent
years. However, ``data-only" methods tend to distort fake samples in the
low-level feature space rather than the high-level semantic space, leading to
limitations in resisting malicious manipulation. Frontier research has shown
that integrating knowledge in deep learning can produce reliable and
generalizable solutions. Inspired by these, we propose a knowledge-guided
adversarial defense (KGAD) to actively force malicious manipulation models to
output semantically confusing samples. Specifically, in the process of
generating adversarial noise, we focus on constructing significant semantic
confusions at the domain-specific knowledge level, and exploit a metric closely
related to visual perception to replace the general pixel-wise metrics. The
generated adversarial noise can actively interfere with the malicious
manipulation model by triggering knowledge-guided and perception-related
disruptions in the fake samples. To validate the effectiveness of the proposed
method, we conduct qualitative and quantitative experiments on human perception
and visual quality assessment. The results on two different tasks both show
that our defense provides better protection compared to state-of-the-art
methods and achieves great generalizability.


http://arxiv.org/abs/2504.09047
Multi-Robot Coordination with Adversarial Perception. (10%)
Rayan Bahrami; Hamidreza Jafarnejadsani
  This paper investigates the resilience of perception-based multi-robot
coordination with wireless communication to online adversarial perception. A
systematic study of this problem is essential for many safety-critical robotic
applications that rely on the measurements from learned perception modules. We
consider a (small) team of quadrotor robots that rely only on an Inertial
Measurement Unit (IMU) and the visual data measurements obtained from a learned
multi-task perception module (e.g., object detection) for downstream tasks,
including relative localization and coordination. We focus on a class of
adversarial perception attacks that cause misclassification, mislocalization,
and latency. We propose that the effects of adversarial misclassification and
mislocalization can be modeled as sporadic (intermittent) and spurious
measurement data for the downstream tasks. To address this, we present a
framework for resilience analysis of multi-robot coordination with adversarial
measurements. The framework integrates data from Visual-Inertial Odometry (VIO)
and the learned perception model for robust relative localization and state
estimation in the presence of adversarially sporadic and spurious measurements.
The framework allows for quantifying the degradation in system observability
and stability in relation to the success rate of adversarial perception.
Finally, experimental results on a multi-robot platform demonstrate the
real-world applicability of our methodology for resource-constrained robotic
platforms.


http://arxiv.org/abs/2504.08977
Robust Steganography from Large Language Models. (5%)
Neil Perry; Sanket Gupte; Nishant Pitta; Lior Rotem
  Recent steganographic schemes, starting with Meteor (CCS'21), rely on
leveraging large language models (LLMs) to resolve a historically-challenging
task of disguising covert communication as ``innocent-looking''
natural-language communication. However, existing methods are vulnerable to
``re-randomization attacks,'' where slight changes to the communicated text,
that might go unnoticed, completely destroy any hidden message. This is also a
vulnerability in more traditional encryption-based stegosystems, where
adversaries can modify the randomness of an encryption scheme to destroy the
hidden message while preserving an acceptable covertext to ordinary users. In
this work, we study the problem of robust steganography. We introduce formal
definitions of weak and strong robust LLM-based steganography, corresponding to
two threat models in which natural language serves as a covertext channel
resistant to realistic re-randomization attacks. We then propose two
constructions satisfying these notions. We design and implement our
steganographic schemes that embed arbitrary secret messages into natural
language text generated by LLMs, ensuring recoverability even under adversarial
paraphrasing and rewording attacks. To support further research and real-world
deployment, we release our implementation and datasets for public use.


http://arxiv.org/abs/2504.08205
EO-VLM: VLM-Guided Energy Overload Attacks on Vision Models. (69%)
Minjae Seo; Myoungsung You; Junhee Lee; Jaehan Kim; Hwanjo Heo; Jintae Oh; Jinwoo Kim
  Vision models are increasingly deployed in critical applications such as
autonomous driving and CCTV monitoring, yet they remain susceptible to
resource-consuming attacks. In this paper, we introduce a novel
energy-overloading attack that leverages vision language model (VLM) prompts to
generate adversarial images targeting vision models. These images, though
imperceptible to the human eye, significantly increase GPU energy consumption
across various vision models, threatening the availability of these systems.
Our framework, EO-VLM (Energy Overload via VLM), is model-agnostic, meaning it
is not limited by the architecture or type of the target vision model. By
exploiting the lack of safety filters in VLMs like DALL-E 3, we create
adversarial noise images without requiring prior knowledge or internal
structure of the target vision models. Our experiments demonstrate up to a 50%
increase in energy consumption, revealing a critical vulnerability in current
vision models.


http://arxiv.org/abs/2504.07887
Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge. (68%)
Riccardo Cantini; Alessio Orsino; Massimo Ruggiero; Domenico Talia
  Large Language Models (LLMs) have revolutionized artificial intelligence,
driving advancements in machine translation, summarization, and conversational
agents. However, their increasing integration into critical societal domains
has raised concerns about embedded biases, which can perpetuate stereotypes and
compromise fairness. These biases stem from various sources, including
historical inequalities in training data, linguistic imbalances, and
adversarial manipulation. Despite mitigation efforts, recent studies indicate
that LLMs remain vulnerable to adversarial attacks designed to elicit biased
responses. This work proposes a scalable benchmarking framework to evaluate LLM
robustness against adversarial bias elicitation. Our methodology involves (i)
systematically probing models with a multi-task approach targeting biases
across various sociocultural dimensions, (ii) quantifying robustness through
safety scores using an LLM-as-a-Judge approach for automated assessment of
model responses, and (iii) employing jailbreak techniques to investigate
vulnerabilities in safety mechanisms. Our analysis examines prevalent biases in
both small and large state-of-the-art models and their impact on model safety.
Additionally, we assess the safety of domain-specific models fine-tuned for
critical fields, such as medicine. Finally, we release a curated dataset of
bias-related prompts, CLEAR-Bias, to facilitate systematic vulnerability
benchmarking. Our findings reveal critical trade-offs between model size and
safety, aiding the development of fairer and more robust future language
models.


http://arxiv.org/abs/2504.07467
Defense against Prompt Injection Attacks via Mixture of Encodings. (38%)
Ruiyi Zhang; David Sullivan; Kyle Jackson; Pengtao Xie; Mei Chen
  Large Language Models (LLMs) have emerged as a dominant approach for a wide
range of NLP tasks, with their access to external information further enhancing
their capabilities. However, this introduces new vulnerabilities, known as
prompt injection attacks, where external content embeds malicious instructions
that manipulate the LLM's output. Recently, the Base64 defense has been
recognized as one of the most effective methods for reducing success rate of
prompt injection attacks. Despite its efficacy, this method can degrade LLM
performance on certain NLP tasks. To address this challenge, we propose a novel
defense mechanism: mixture of encodings, which utilizes multiple character
encodings, including Base64. Extensive experimental results show that our
method achieves one of the lowest attack success rates under prompt injection
attacks, while maintaining high performance across all NLP tasks, outperforming
existing character encoding-based defense methods. This underscores the
effectiveness of our mixture of encodings strategy for both safety and task
performance metrics.


http://arxiv.org/abs/2504.08848
X-Guard: Multilingual Guard Agent for Content Moderation. (13%)
Bibek Upadhayay; Vahid Behzadan; Ph. D
  Large Language Models (LLMs) have rapidly become integral to numerous
applications in critical domains where reliability is paramount. Despite
significant advances in safety frameworks and guardrails, current protective
measures exhibit crucial vulnerabilities, particularly in multilingual
contexts. Existing safety systems remain susceptible to adversarial attacks in
low-resource languages and through code-switching techniques, primarily due to
their English-centric design. Furthermore, the development of effective
multilingual guardrails is constrained by the scarcity of diverse cross-lingual
training data. Even recent solutions like Llama Guard-3, while offering
multilingual support, lack transparency in their decision-making processes. We
address these challenges by introducing X-Guard agent, a transparent
multilingual safety agent designed to provide content moderation across diverse
linguistic contexts. X-Guard effectively defends against both conventional
low-resource language attacks and sophisticated code-switching attacks. Our
approach includes: curating and enhancing multiple open-source safety datasets
with explicit evaluation rationales; employing a jury of judges methodology to
mitigate individual judge LLM provider biases; creating a comprehensive
multilingual safety dataset spanning 132 languages with 5 million data points;
and developing a two-stage architecture combining a custom-finetuned mBART-50
translation module with an evaluation X-Guard 3B model trained through
supervised finetuning and GRPO training. Our empirical evaluations demonstrate
X-Guard's effectiveness in detecting unsafe content across multiple languages
while maintaining transparency throughout the safety evaluation process. Our
work represents a significant advancement in creating robust, transparent, and
linguistically inclusive safety systems for LLMs and its integrated systems.


http://arxiv.org/abs/2504.12321
AttentionDefense: Leveraging System Prompt Attention for Explainable Defense Against Novel Jailbreaks. (13%)
Charlotte Siska; Anush Sankaran
  In the past few years, Language Models (LMs) have shown par-human
capabilities in several domains. Despite their practical applications and
exceeding user consumption, they are susceptible to jailbreaks when malicious
input exploits the LM's weaknesses, causing it to deviate from its intended
behavior. Current defensive strategies either classify the input prompt as
adversarial or prevent LMs from generating harmful outputs. However, it is
challenging to explain the reason behind the malicious nature of the jailbreak,
which results in a wide variety of closed-box approaches. In this research, we
propose and demonstrate that system-prompt attention from Small Language Models
(SLMs) can be used to characterize adversarial prompts, providing a novel,
explainable, and cheaper defense approach called AttentionDefense. Our research
suggests that the attention mechanism is an integral component in understanding
and explaining how LMs respond to malicious input that is not captured in the
semantic meaning of text embeddings. The proposed AttentionDefense is evaluated
against existing jailbreak benchmark datasets. Ablation studies show that
SLM-based AttentionDefense has equivalent or better jailbreak detection
performance compared to text embedding-based classifiers and GPT-4 zero-shot
detectors.To further validate the efficacy of the proposed approach, we
generate a dataset of novel jailbreak variants of the existing benchmark
dataset using a closed-loop LLM-based multi-agent system. We demonstrate that
the proposed AttentionDefense approach performs robustly on this novel
jailbreak dataset while existing approaches suffer in performance.
Additionally, for practical purposes AttentionDefense is an ideal solution as
it has the computation requirements of a small LM but the performance of a LLM
detector.


http://arxiv.org/abs/2504.07717
PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimization. (11%)
Yang Jiao; Xiaodong Wang; Kai Yang
  Large Language Models (LLMs) have demonstrated remarkable performance across
a wide range of applications, e.g., medical question-answering, mathematical
sciences, and code generation. However, they also exhibit inherent limitations,
such as outdated knowledge and susceptibility to hallucinations.
Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm to
address these issues, but it also introduces new vulnerabilities. Recent
efforts have focused on the security of RAG-based LLMs, yet existing attack
methods face three critical challenges: (1) their effectiveness declines
sharply when only a limited number of poisoned texts can be injected into the
knowledge database, (2) they lack sufficient stealth, as the attacks are often
detectable by anomaly detection systems, which compromises their effectiveness,
and (3) they rely on heuristic approaches to generate poisoned texts, lacking
formal optimization frameworks and theoretic guarantees, which limits their
effectiveness and applicability. To address these issues, we propose
coordinated Prompt-RAG attack (PR-attack), a novel optimization-driven attack
that introduces a small number of poisoned texts into the knowledge database
while embedding a backdoor trigger within the prompt. When activated, the
trigger causes the LLM to generate pre-designed responses to targeted queries,
while maintaining normal behavior in other contexts. This ensures both high
effectiveness and stealth. We formulate the attack generation process as a
bilevel optimization problem leveraging a principled optimization framework to
develop optimal poisoned texts and triggers. Extensive experiments across
diverse LLMs and datasets demonstrate the effectiveness of PR-Attack, achieving
a high attack success rate even with a limited number of poisoned texts and
significantly improved stealth compared to existing methods.


http://arxiv.org/abs/2504.07831
Deceptive Automated Interpretability: Language Models Coordinating to Fool Oversight Systems. (10%)
Simon Lermen; Mateusz Dziemian; Natalia Pérez-Campanero Antolín
  We demonstrate how AI agents can coordinate to deceive oversight systems
using automated interpretability of neural networks. Using sparse autoencoders
(SAEs) as our experimental framework, we show that language models (Llama,
DeepSeek R1, and Claude 3.7 Sonnet) can generate deceptive explanations that
evade detection. Our agents employ steganographic methods to hide information
in seemingly innocent explanations, successfully fooling oversight models while
achieving explanation quality comparable to reference labels. We further find
that models can scheme to develop deceptive strategies when they believe the
detection of harmful features might lead to negative consequences for
themselves. All tested LLM agents were capable of deceiving the overseer while
achieving high interpretability scores comparable to those of reference labels.
We conclude by proposing mitigation strategies, emphasizing the critical need
for robust understanding and defenses against deception.


http://arxiv.org/abs/2504.07461
Achilles Heel of Distributed Multi-Agent Systems. (1%)
Yiting Zhang; Yijiang Li; Tianwei Zhao; Kaijie Zhu; Haohan Wang; Nuno Vasconcelos
  Multi-agent system (MAS) has demonstrated exceptional capabilities in
addressing complex challenges, largely due to the integration of multiple large
language models (LLMs). However, the heterogeneity of LLMs, the scalability of
quantities of LLMs, and local computational constraints pose significant
challenges to hosting these models locally. To address these issues, we propose
a new framework termed Distributed Multi-Agent System (DMAS). In DMAS,
heterogeneous third-party agents function as service providers managed remotely
by a central MAS server and each agent offers its services through API
interfaces. However, the distributed nature of DMAS introduces several concerns
about trustworthiness. In this paper, we study the Achilles heel of distributed
multi-agent systems, identifying four critical trustworthiness challenges: free
riding, susceptibility to malicious attacks, communication inefficiencies, and
system instability. Extensive experiments across seven frameworks and four
datasets reveal significant vulnerabilities of the DMAS. These attack
strategies can lead to a performance degradation of up to 80% and attain a 100%
success rate in executing free riding and malicious attacks. We envision our
work will serve as a useful red-teaming tool for evaluating future multi-agent
systems and spark further research on trustworthiness challenges in distributed
multi-agent systems.


http://arxiv.org/abs/2504.08813
SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models. (1%)
Junfeng Fang; Yukai Wang; Ruipeng Wang; Zijun Yao; Kun Wang; An Zhang; Xiang Wang; Tat-Seng Chua
  The rapid advancement of multi-modal large reasoning models (MLRMs) --
enhanced versions of multimodal language models (MLLMs) equipped with reasoning
capabilities -- has revolutionized diverse applications. However, their safety
implications remain underexplored. While prior work has exposed critical
vulnerabilities in unimodal reasoning models, MLRMs introduce distinct risks
from cross-modal reasoning pathways. This work presents the first systematic
safety analysis of MLRMs through large-scale empirical studies comparing MLRMs
with their base MLLMs. Our experiments reveal three critical findings: (1) The
Reasoning Tax: Acquiring reasoning capabilities catastrophically degrades
inherited safety alignment. MLRMs exhibit 37.44% higher jailbreaking success
rates than base MLLMs under adversarial attacks. (2) Safety Blind Spots: While
safety degradation is pervasive, certain scenarios (e.g., Illegal Activity)
suffer 25 times higher attack rates -- far exceeding the average 3.4 times
increase, revealing scenario-specific vulnerabilities with alarming cross-model
and datasets consistency. (3) Emergent Self-Correction: Despite tight
reasoning-answer safety coupling, MLRMs demonstrate nascent self-correction --
16.9% of jailbroken reasoning steps are overridden by safe answers, hinting at
intrinsic safeguards. These findings underscore the urgency of scenario-aware
safety auditing and mechanisms to amplify MLRMs' self-correction potential. To
catalyze research, we open-source OpenSafeMLRM, the first toolkit for MLRM
safety evaluation, providing unified interface for mainstream models, datasets,
and jailbreaking methods. Our work calls for immediate efforts to harden
reasoning-augmented AI, ensuring its transformative potential aligns with
ethical safeguards.


http://arxiv.org/abs/2504.06575
Defending LLM Watermarking Against Spoofing Attacks with Contrastive Representation Learning. (1%)
Li An; Yujian Liu; Yepeng Liu; Yang Zhang; Yuheng Bu; Shiyu Chang
  Watermarking has emerged as a promising technique for detecting texts
generated by LLMs. Current research has primarily focused on three design
criteria: high quality of the watermarked text, high detectability, and
robustness against removal attack. However, the security against spoofing
attacks remains relatively understudied. For example, a piggyback attack can
maliciously alter the meaning of watermarked text-transforming it into hate
speech-while preserving the original watermark, thereby damaging the reputation
of the LLM provider. We identify two core challenges that make defending
against spoofing difficult: (1) the need for watermarks to be both sensitive to
semantic-distorting changes and insensitive to semantic-preserving edits, and
(2) the contradiction between the need to detect global semantic shifts and the
local, auto-regressive nature of most watermarking schemes. To address these
challenges, we propose a semantic-aware watermarking algorithm that post-hoc
embeds watermarks into a given target text while preserving its original
meaning. Our method introduces a semantic mapping model, which guides the
generation of a green-red token list, contrastively trained to be sensitive to
semantic-distorting changes and insensitive to semantic-preserving changes.
Experiments on two standard benchmarks demonstrate strong robustness against
removal attacks and security against spoofing attacks, including sentiment
reversal and toxic content insertion, while maintaining high watermark
detectability. Our approach offers a significant step toward more secure and
semantically aware watermarking for LLMs. Our code is available at
https://github.com/UCSB-NLP-Chang/contrastive-watermark.


http://arxiv.org/abs/2504.05838
Mind the Trojan Horse: Image Prompt Adapter Enabling Scalable and Deceptive Jailbreaking. (97%)
Junxi Chen; Junhao Dong; Xiaohua Xie
  Recently, the Image Prompt Adapter (IP-Adapter) has been increasingly
integrated into text-to-image diffusion models (T2I-DMs) to improve
controllability. However, in this paper, we reveal that T2I-DMs equipped with
the IP-Adapter (T2I-IP-DMs) enable a new jailbreak attack named the hijacking
attack. We demonstrate that, by uploading imperceptible image-space adversarial
examples (AEs), the adversary can hijack massive benign users to jailbreak an
Image Generation Service (IGS) driven by T2I-IP-DMs and mislead the public to
discredit the service provider. Worse still, the IP-Adapter's dependency on
open-source image encoders reduces the knowledge required to craft AEs.
Extensive experiments verify the technical feasibility of the hijacking attack.
In light of the revealed threat, we investigate several existing defenses and
explore combining the IP-Adapter with adversarially trained models to overcome
existing defenses' limitations. Our code is available at
https://github.com/fhdnskfbeuv/attackIPA.


http://arxiv.org/abs/2504.06358
Towards Calibration Enhanced Network by Inverse Adversarial Attack. (92%)
Yupeng Cheng; Zi Pong Lim; Sarthak Ketanbhai Modi; Yon Shin Teo; Yushi Cao; Shang-Wei Lin
  Test automation has become increasingly important as the complexity of both
design and content in Human Machine Interface (HMI) software continues to grow.
Current standard practice uses Optical Character Recognition (OCR) techniques
to automatically extract textual information from HMI screens for validation.
At present, one of the key challenges faced during the automation of HMI screen
validation is the noise handling for the OCR models. In this paper, we propose
to utilize adversarial training techniques to enhance OCR models in HMI testing
scenarios. More specifically, we design a new adversarial attack objective for
OCR models to discover the decision boundaries in the context of HMI testing.
We then adopt adversarial training to optimize the decision boundaries towards
a more robust and accurate OCR model. In addition, we also built an HMI screen
dataset based on real-world requirements and applied multiple types of
perturbation onto the clean HMI dataset to provide a more complete coverage for
the potential scenarios. We conduct experiments to demonstrate how using
adversarial training techniques yields more robust OCR models against various
kinds of noises, while still maintaining high OCR model accuracy. Further
experiments even demonstrate that the adversarial training models exhibit a
certain degree of robustness against perturbations from other patterns.


http://arxiv.org/abs/2504.06141
Adversarial Training of Reward Models. (88%)
Alexander Bukharin; Haifeng Qian; Shengyang Sun; Adithya Renduchintala; Soumye Singhal; Zhilin Wang; Oleksii Kuchaiev; Olivier Delalleau; Tuo Zhao
  Reward modeling has emerged as a promising approach for the scalable
alignment of language models. However, contemporary reward models (RMs) often
lack robustness, awarding high rewards to low-quality, out-of-distribution
(OOD) samples. This can lead to reward hacking, where policies exploit
unintended shortcuts to maximize rewards, undermining alignment. To address
this challenge, we introduce Adv-RM, a novel adversarial training framework
that automatically identifies adversarial examples -- responses that receive
high rewards from the target RM but are OOD and of low quality. By leveraging
reinforcement learning, Adv-RM trains a policy to generate adversarial examples
that reliably expose vulnerabilities in large state-of-the-art reward models
such as Nemotron 340B RM. Incorporating these adversarial examples into the
reward training process improves the robustness of RMs, mitigating reward
hacking and enhancing downstream performance in RLHF. We demonstrate that
Adv-RM significantly outperforms conventional RM training, increasing stability
and enabling more effective RLHF training in both synthetic and real-data
settings.


http://arxiv.org/abs/2504.08798
Exploring Gradient-Guided Masked Language Model to Detect Textual Adversarial Attacks. (86%)
Xiaomei Zhang; Zhaoxi Zhang; Yanjun Zhang; Xufei Zheng; Leo Yu Zhang; Shengshan Hu; Shirui Pan
  Textual adversarial examples pose serious threats to the reliability of
natural language processing systems. Recent studies suggest that adversarial
examples tend to deviate from the underlying manifold of normal texts, whereas
pre-trained masked language models can approximate the manifold of normal data.
These findings inspire the exploration of masked language models for detecting
textual adversarial attacks. We first introduce Masked Language Model-based
Detection (MLMD), leveraging the mask and unmask operations of the masked
language modeling (MLM) objective to induce the difference in manifold changes
between normal and adversarial texts. Although MLMD achieves competitive
detection performance, its exhaustive one-by-one masking strategy introduces
significant computational overhead. Our posterior analysis reveals that a
significant number of non-keywords in the input are not important for detection
but consume resources. Building on this, we introduce Gradient-guided MLMD
(GradMLMD), which leverages gradient information to identify and skip
non-keywords during detection, significantly reducing resource consumption
without compromising detection performance.


http://arxiv.org/abs/2504.05804
StealthRank: LLM Ranking Manipulation via Stealthy Prompt Optimization. (83%)
Yiming Tang; Yi Fan; Chenxiao Yu; Tiankai Yang; Yue Zhao; Xiyang Hu
  The integration of large language models (LLMs) into information retrieval
systems introduces new attack surfaces, particularly for adversarial ranking
manipulations. We present $\textbf{StealthRank}$, a novel adversarial attack
method that manipulates LLM-driven ranking systems while maintaining textual
fluency and stealth. Unlike existing methods that often introduce detectable
anomalies, StealthRank employs an energy-based optimization framework combined
with Langevin dynamics to generate StealthRank Prompts (SRPs)-adversarial text
sequences embedded within item or document descriptions that subtly yet
effectively influence LLM ranking mechanisms. We evaluate StealthRank across
multiple LLMs, demonstrating its ability to covertly boost the ranking of
target items while avoiding explicit manipulation traces. Our results show that
StealthRank consistently outperforms state-of-the-art adversarial ranking
baselines in both effectiveness and stealth, highlighting critical
vulnerabilities in LLM-driven ranking systems. Our code is publicly available
at $\href{https://github.com/Tangyiming205069/controllable-seo}{here}$.


http://arxiv.org/abs/2504.06492
Exploiting Meta-Learning-based Poisoning Attacks for Graph Link Prediction. (67%)
Mingchen Li; Di Zhuang; Keyu Chen; Dumindu Samaraweera; Morris Chang
  Link prediction in graph data utilizes various algorithms and machine
learning/deep learning models to predict potential relationships between graph
nodes. This technique has found widespread use in numerous real-world
applications, including recommendation systems, community networks, and
biological structures. However, recent research has highlighted the
vulnerability of link prediction models to adversarial attacks, such as
poisoning and evasion attacks. Addressing the vulnerability of these models is
crucial to ensure stable and robust performance in link prediction
applications. While many works have focused on enhancing the robustness of the
Graph Convolution Network (GCN) model, the Variational Graph Auto-Encoder
(VGAE), a sophisticated model for link prediction, has not been thoroughly
investigated in the context of graph adversarial attacks. To bridge this gap,
this article proposes an unweighted graph poisoning attack approach using
meta-learning techniques to undermine VGAE's link prediction performance. We
conducted comprehensive experiments on diverse datasets to evaluate the
proposed method and its parameters, comparing it with existing approaches in
similar settings. Our results demonstrate that our approach significantly
diminishes link prediction performance and outperforms other state-of-the-art
methods.


http://arxiv.org/abs/2504.05815
Parasite: A Steganography-based Backdoor Attack Framework for Diffusion Models. (13%)
Jiahao Chen; Yu Pan; Yi Du; Chunkai Wu; Lin Wang
  Recently, the diffusion model has gained significant attention as one of the
most successful image generation models, which can generate high-quality images
by iteratively sampling noise. However, recent studies have shown that
diffusion models are vulnerable to backdoor attacks, allowing attackers to
enter input data containing triggers to activate the backdoor and generate
their desired output. Existing backdoor attack methods primarily focused on
target noise-to-image and text-to-image tasks, with limited work on backdoor
attacks in image-to-image tasks. Furthermore, traditional backdoor attacks
often rely on a single, conspicuous trigger to generate a fixed target image,
lacking concealability and flexibility. To address these limitations, we
propose a novel backdoor attack method called "Parasite" for image-to-image
tasks in diffusion models, which not only is the first to leverage
steganography for triggers hiding, but also allows attackers to embed the
target content as a backdoor trigger to achieve a more flexible attack.
"Parasite" as a novel attack method effectively bypasses existing detection
frameworks to execute backdoor attacks. In our experiments, "Parasite" achieved
a 0 percent backdoor detection rate against the mainstream defense frameworks.
In addition, in the ablation study, we discuss the influence of different
hiding coefficients on the attack results. You can find our code at
https://anonymous.4open.science/r/Parasite-1715/.


http://arxiv.org/abs/2504.05902
Defending Deep Neural Networks against Backdoor Attacks via Module Switching. (10%)
Weijun Li; Ansh Arora; Xuanli He; Mark Dras; Qiongkai Xu
  The exponential increase in the parameters of Deep Neural Networks (DNNs) has
significantly raised the cost of independent training, particularly for
resource-constrained entities. As a result, there is a growing reliance on
open-source models. However, the opacity of training processes exacerbates
security risks, making these models more vulnerable to malicious threats, such
as backdoor attacks, while simultaneously complicating defense mechanisms.
Merging homogeneous models has gained attention as a cost-effective
post-training defense. However, we notice that existing strategies, such as
weight averaging, only partially mitigate the influence of poisoned parameters
and remain ineffective in disrupting the pervasive spurious correlations
embedded across model parameters. We propose a novel module-switching strategy
to break such spurious correlations within the model's propagation path. By
leveraging evolutionary algorithms to optimize fusion strategies, we validate
our approach against backdoor attacks targeting text and vision domains. Our
method achieves effective backdoor mitigation even when incorporating a couple
of compromised models, e.g., reducing the average attack success rate (ASR) to
22% compared to 31.9% with the best-performing baseline on SST-2.


http://arxiv.org/abs/2504.05222
Security Risks in Vision-Based Beam Prediction: From Spatial Proxy Attacks to Feature Refinement. (99%)
Avi Deb Raha; Kitae Kim; Mrityunjoy Gain; Apurba Adhikary; Zhu Han; Eui-Nam Huh; Choong Seon Hong
  The rapid evolution towards the sixth-generation (6G) networks demands
advanced beamforming techniques to address challenges in dynamic, high-mobility
scenarios, such as vehicular communications. Vision-based beam prediction
utilizing RGB camera images emerges as a promising solution for accurate and
responsive beam selection. However, reliance on visual data introduces unique
vulnerabilities, particularly susceptibility to adversarial attacks, thus
potentially compromising beam accuracy and overall network reliability. In this
paper, we conduct the first systematic exploration of adversarial threats
specifically targeting vision-based mmWave beam selection systems. Traditional
white-box attacks are impractical in this context because ground-truth beam
indices are inaccessible and spatial dynamics are complex. To address this, we
propose a novel black-box adversarial attack strategy, termed Spatial Proxy
Attack (SPA), which leverages spatial correlations between user positions and
beam indices to craft effective perturbations without requiring access to model
parameters or labels. To counteract these adversarial vulnerabilities, we
formulate an optimization framework aimed at simultaneously enhancing beam
selection accuracy under clean conditions and robustness against adversarial
perturbations. We introduce a hybrid deep learning architecture integrated with
a dedicated Feature Refinement Module (FRM), designed to systematically filter
irrelevant, noisy and adversarially perturbed visual features. Evaluations
using standard backbone models such as ResNet-50 and MobileNetV2 demonstrate
that our proposed method significantly improves performance, achieving up to an
+21.07\% gain in Top-K accuracy under clean conditions and a 41.31\% increase
in Top-1 adversarial robustness compared to different baseline models.


http://arxiv.org/abs/2504.04858
Don't Lag, RAG: Training-Free Adversarial Detection Using RAG. (88%)
Roie Kazoom; Raz Lapid; Moshe Sipper; Ofer Hadar
  Adversarial patch attacks pose a major threat to vision systems by embedding
localized perturbations that mislead deep models. Traditional defense methods
often require retraining or fine-tuning, making them impractical for real-world
deployment. We propose a training-free Visual Retrieval-Augmented Generation
(VRAG) framework that integrates Vision-Language Models (VLMs) for adversarial
patch detection. By retrieving visually similar patches and images that
resemble stored attacks in a continuously expanding database, VRAG performs
generative reasoning to identify diverse attack types, all without additional
training or fine-tuning. We extensively evaluate open-source large-scale VLMs,
including Qwen-VL-Plus, Qwen2.5-VL-72B, and UI-TARS-72B-DPO, alongside
Gemini-2.0, a closed-source model. Notably, the open-source UI-TARS-72B-DPO
model achieves up to 95 percent classification accuracy, setting a new
state-of-the-art for open-source adversarial patch detection. Gemini-2.0
attains the highest overall accuracy, 98 percent, but remains closed-source.
Experimental results demonstrate VRAG's effectiveness in identifying a variety
of adversarial patches with minimal human annotation, paving the way for
robust, practical defenses against evolving adversarial patch attacks.


http://arxiv.org/abs/2504.05483
Secure Diagnostics: Adversarial Robustness Meets Clinical Interpretability. (81%)
Mohammad Hossein Najafi; Mohammad Morsali; Mohammadreza Pashanejad; Saman Soleimani Roudi; Mohammad Norouzi; Saeed Bagheri Shouraki
  Deep neural networks for medical image classification often fail to
generalize consistently in clinical practice due to violations of the i.i.d.
assumption and opaque decision-making. This paper examines interpretability in
deep neural networks fine-tuned for fracture detection by evaluating model
performance against adversarial attack and comparing interpretability methods
to fracture regions annotated by an orthopedic surgeon. Our findings prove that
robust models yield explanations more aligned with clinically meaningful areas,
indicating that robustness encourages anatomically relevant feature
prioritization. We emphasize the value of interpretability for facilitating
human-AI collaboration, in which models serve as assistants under a
human-in-the-loop paradigm: clinically plausible explanations foster trust,
enable error correction, and discourage reliance on AI for high-stakes
decisions. This paper investigates robustness and interpretability as
complementary benchmarks for bridging the gap between benchmark performance and
safe, actionable clinical deployment.


http://arxiv.org/abs/2504.04747
Two is Better than One: Efficient Ensemble Defense for Robust and Compact Models. (80%)
Yoojin Jung; Byung Cheol Song
  Deep learning-based computer vision systems adopt complex and large
architectures to improve performance, yet they face challenges in deployment on
resource-constrained mobile and edge devices. To address this issue, model
compression techniques such as pruning, quantization, and matrix factorization
have been proposed; however, these compressed models are often highly
vulnerable to adversarial attacks. We introduce the \textbf{Efficient Ensemble
Defense (EED)} technique, which diversifies the compression of a single base
model based on different pruning importance scores and enhances ensemble
diversity to achieve high adversarial robustness and resource efficiency. EED
dynamically determines the number of necessary sub-models during the inference
stage, minimizing unnecessary computations while maintaining high robustness.
On the CIFAR-10 and SVHN datasets, EED demonstrated state-of-the-art robustness
performance compared to existing adversarial pruning techniques, along with an
inference speed improvement of up to 1.86 times. This proves that EED is a
powerful defense solution in resource-constrained environments.


http://arxiv.org/abs/2504.05652
Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking. (54%)
Yu-Hang Wu; Yu-Jie Xiong; Jie-Zhang
  Large Language Models (LLMs) have become increasingly integral to a wide
range of applications. However, they still remain the threat of jailbreak
attacks, where attackers manipulate designed prompts to make the models elicit
malicious outputs. Analyzing jailbreak methods can help us delve into the
weakness of LLMs and improve it. In this paper, We reveal a vulnerability in
large language models (LLMs), which we term Defense Threshold Decay (DTD), by
analyzing the attention weights of the model's output on input and subsequent
output on prior output: as the model generates substantial benign content, its
attention weights shift from the input to prior output, making it more
susceptible to jailbreak attacks. To demonstrate the exploitability of DTD, we
propose a novel jailbreak attack method, Sugar-Coated Poison (SCP), which
induces the model to generate substantial benign content through benign input
and adversarial reasoning, subsequently producing malicious content. To
mitigate such attacks, we introduce a simple yet effective defense strategy,
POSD, which significantly reduces jailbreak success rates while preserving the
model's generalization capabilities.


http://arxiv.org/abs/2504.07135
SINCon: Mitigate LLM-Generated Malicious Message Injection Attack for Rumor Detection. (10%)
Mingqing Zhang; Qiang Liu; Xiang Tao; Shu Wu; Liang Wang
  In the era of rapidly evolving large language models (LLMs), state-of-the-art
rumor detection systems, particularly those based on Message Propagation Trees
(MPTs), which represent a conversation tree with the post as its root and the
replies as its descendants, are facing increasing threats from adversarial
attacks that leverage LLMs to generate and inject malicious messages. Existing
methods are based on the assumption that different nodes exhibit varying
degrees of influence on predictions. They define nodes with high predictive
influence as important nodes and target them for attacks. If the model treats
nodes' predictive influence more uniformly, attackers will find it harder to
target high predictive influence nodes. In this paper, we propose Similarizing
the predictive Influence of Nodes with Contrastive Learning (SINCon), a defense
mechanism that encourages the model to learn graph representations where nodes
with varying importance have a more uniform influence on predictions. Extensive
experiments on the Twitter and Weibo datasets demonstrate that SINCon not only
preserves high classification accuracy on clean data but also significantly
enhances resistance against LLM-driven message injection attacks.


http://arxiv.org/abs/2504.04809
Select Me! When You Need a Tool: A Black-box Text Attack on Tool Selection. (9%)
Liuji Chen; Hao Gao; Jinghao Zhang; Qiang Liu; Shu Wu; Liang Wang
  Tool learning serves as a powerful auxiliary mechanism that extends the
capabilities of large language models (LLMs), enabling them to tackle complex
tasks requiring real-time relevance or high precision operations. Behind its
powerful capabilities lie some potential security issues. However, previous
work has primarily focused on how to make the output of the invoked tools
incorrect or malicious, with little attention given to the manipulation of tool
selection. To fill this gap, we introduce, for the first time, a black-box
text-based attack that can significantly increase the probability of the target
tool being selected in this paper. We propose a two-level text perturbation
attack witha coarse-to-fine granularity, attacking the text at both the word
level and the character level. We conduct comprehensive experiments that
demonstrate the attacker only needs to make some perturbations to the tool's
textual information to significantly increase the possibility of the target
tool being selected and ranked higher among the candidate tools. Our research
reveals the vulnerability of the tool selection process and paves the way for
future research on protecting this process.


http://arxiv.org/abs/2504.05197
P2Mark: Plug-and-play Parameter-intrinsic Watermarking for Neural Speech Generation. (1%)
Yong Ren; Jiangyan Yi; Tao Wang; Jianhua Tao; Zhengqi Wen; Chenxing Li; Zheng Lian; Ruibo Fu; Ye Bai; Xiaohui Zhang
  Recently, a large number of advanced neural speech generation methods have
emerged in the open-source community. Although this has facilitated the
application and development of technology, it has also increased the difficulty
of preventing the abuse of generated speech and protecting copyrights. Audio
watermarking technology is an effective method for proactively protecting
generated speech, but when the source codes and model weights of the neural
speech generation methods are open-sourced, audio watermarks based on previous
watermarking methods can be easily removed or manipulated. This paper proposes
a Plug-and-play Parameter-intrinsic WaterMarking (P2Mark) method for neural
speech generation system protection. The main advantage of P2Mark is that the
watermark information is flexibly integrated into the neural speech generation
model in the form of parameters by training a watermark adapter rather than
injecting the watermark into the model in the form of features. After the
watermark adapter with the watermark embedding is merged with the pre-trained
generation model, the watermark information cannot be easily removed or
manipulated. Therefore, P2Mark will be a reliable choice for proactively
tracing and protecting the copyrights of neural speech generation models in
open-source white-box scenarios. We validated P2Mark on two main types of
decoders in neural speech generation: vocoder and codec. Experimental results
show that P2Mark achieves performance comparable to state-of-the-art audio
watermarking methods that cannot be used for open-source white-box protection
scenarios in terms of watermark extraction accuracy, watermark
imperceptibility, and robustness.


http://arxiv.org/abs/2504.05050
Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models. (1%)
Jiawei Lian; Jianhong Pan; Lefan Wang; Yi Wang; Shaohui Mei; Lap-Pui Chau
  Large language models (LLMs) are foundational explorations to artificial
general intelligence, yet their alignment with human values via instruction
tuning and preference learning achieves only superficial compliance. Here, we
demonstrate that harmful knowledge embedded during pretraining persists as
indelible "dark patterns" in LLMs' parametric memory, evading alignment
safeguards and resurfacing under adversarial inducement at distributional
shifts. In this study, we first theoretically analyze the intrinsic ethical
vulnerability of aligned LLMs by proving that current alignment methods yield
only local "safety regions" in the knowledge manifold. In contrast, pretrained
knowledge remains globally connected to harmful concepts via high-likelihood
adversarial trajectories. Building on this theoretical insight, we empirically
validate our findings by employing semantic coherence inducement under
distributional shifts--a method that systematically bypasses alignment
constraints through optimized adversarial prompts. This combined theoretical
and empirical approach achieves a 100% attack success rate across 19 out of 23
state-of-the-art aligned LLMs, including DeepSeek-R1 and LLaMA-3, revealing
their universal vulnerabilities.


http://arxiv.org/abs/2504.05504
SelfMAD: Enhancing Generalization and Robustness in Morphing Attack Detection via Self-Supervised Learning. (1%)
Marija Ivanovska; Leon Todorov; Naser Damer; Deepak Kumar Jain; Peter Peer; Vitomir Štruc
  With the continuous advancement of generative models, face morphing attacks
have become a significant challenge for existing face verification systems due
to their potential use in identity fraud and other malicious activities.
Contemporary Morphing Attack Detection (MAD) approaches frequently rely on
supervised, discriminative models trained on examples of bona fide and morphed
images. These models typically perform well with morphs generated with
techniques seen during training, but often lead to sub-optimal performance when
subjected to novel unseen morphing techniques. While unsupervised models have
been shown to perform better in terms of generalizability, they typically
result in higher error rates, as they struggle to effectively capture features
of subtle artifacts. To address these shortcomings, we present SelfMAD, a novel
self-supervised approach that simulates general morphing attack artifacts,
allowing classifiers to learn generic and robust decision boundaries without
overfitting to the specific artifacts induced by particular face morphing
methods. Through extensive experiments on widely used datasets, we demonstrate
that SelfMAD significantly outperforms current state-of-the-art MADs, reducing
the detection error by more than 64% in terms of EER when compared to the
strongest unsupervised competitor, and by more than 66%, when compared to the
best performing discriminative MAD model, tested in cross-morph settings. The
source code for SelfMAD is available at https://github.com/LeonTodorov/SelfMAD.


http://arxiv.org/abs/2504.05618
Technical Report: Full Version of Analyzing and Optimizing Perturbation of DP-SGD Geometrically. (1%)
Jiawei Duan; Haibo Hu; Qingqing Ye; Xinyue Sun
  Differential privacy (DP) has become a prevalent privacy model in a wide
range of machine learning tasks, especially after the debut of DP-SGD. However,
DP-SGD, which directly perturbs gradients in the training iterations, fails to
mitigate the negative impacts of noise on gradient direction. As a result,
DP-SGD is often inefficient. Although various solutions (e.g., clipping to
reduce the sensitivity of gradients and amplifying privacy bounds to save
privacy budgets) are proposed to trade privacy for model efficiency, the root
cause of its inefficiency is yet unveiled.
  In this work, we first generalize DP-SGD and theoretically derive the impact
of DP noise on the training process. Our analysis reveals that, in terms of a
perturbed gradient, only the noise on direction has eminent impact on the model
efficiency while that on magnitude can be mitigated by optimization techniques,
i.e., fine-tuning gradient clipping and learning rate. Besides, we confirm that
traditional DP introduces biased noise on the direction when adding unbiased
noise to the gradient itself. Overall, the perturbation of DP-SGD is actually
sub-optimal from a geometric perspective. Motivated by this, we design a
geometric perturbation strategy GeoDP within the DP framework, which perturbs
the direction and the magnitude of a gradient, respectively. By directly
reducing the noise on the direction, GeoDP mitigates the negative impact of DP
noise on model efficiency with the same DP guarantee. Extensive experiments on
two public datasets (i.e., MNIST and CIFAR-10), one synthetic dataset and three
prevalent models (i.e., Logistic Regression, CNN and ResNet) confirm the
effectiveness and generality of our strategy.


http://arxiv.org/abs/2504.04394
Selective Masking Adversarial Attack on Automatic Speech Recognition Systems. (99%)
Zheng Fang; Shenyi Zhang; Tao Wang; Bowen Li; Lingchen Zhao; Zhangyi Wang
  Extensive research has shown that Automatic Speech Recognition (ASR) systems
are vulnerable to audio adversarial attacks. Current attacks mainly focus on
single-source scenarios, ignoring dual-source scenarios where two people are
speaking simultaneously. To bridge the gap, we propose a Selective Masking
Adversarial attack, namely SMA attack, which ensures that one audio source is
selected for recognition while the other audio source is muted in dual-source
scenarios. To better adapt to the dual-source scenario, our SMA attack
constructs the normal dual-source audio from the muted audio and selected
audio. SMA attack initializes the adversarial perturbation with a small
Gaussian noise and iteratively optimizes it using a selective masking
optimization algorithm. Extensive experiments demonstrate that the SMA attack
can generate effective and imperceptible audio adversarial examples in the
dual-source scenario, achieving an average success rate of attack of 100% and
signal-to-noise ratio of 37.15dB on Conformer-CTC, outperforming the baselines.


http://arxiv.org/abs/2504.04367
WeiDetect: Weibull Distribution-Based Defense against Poisoning Attacks in Federated Learning for Network Intrusion Detection Systems. (82%)
Sameera K. M.; Vinod P.; Anderson Rocha; Rafidha Rehiman K. A.; Mauro Conti
  In the era of data expansion, ensuring data privacy has become increasingly
critical, posing significant challenges to traditional AI-based applications.
In addition, the increasing adoption of IoT devices has introduced significant
cybersecurity challenges, making traditional Network Intrusion Detection
Systems (NIDS) less effective against evolving threats, and privacy concerns
and regulatory restrictions limit their deployment. Federated Learning (FL) has
emerged as a promising solution, allowing decentralized model training while
maintaining data privacy to solve these issues. However, despite implementing
privacy-preserving technologies, FL systems remain vulnerable to adversarial
attacks. Furthermore, data distribution among clients is not heterogeneous in
the FL scenario. We propose WeiDetect, a two-phase, server-side defense
mechanism for FL-based NIDS that detects malicious participants to address
these challenges. In the first phase, local models are evaluated using a
validation dataset to generate validation scores. These scores are then
analyzed using a Weibull distribution, identifying and removing malicious
models. We conducted experiments to evaluate the effectiveness of our approach
in diverse attack settings. Our evaluation included two popular datasets,
CIC-Darknet2020 and CSE-CIC-IDS2018, tested under non-IID data distributions.
Our findings highlight that WeiDetect outperforms state-of-the-art defense
approaches, improving higher target class recall up to 70% and enhancing the
global model's F1 score by 1% to 14%.


http://arxiv.org/abs/2504.04716
On the Robustness of GUI Grounding Models Against Image Attacks. (81%)
Haoren Zhao; Tianyi Chen; Zhen Wang
  Graphical User Interface (GUI) grounding models are crucial for enabling
intelligent agents to understand and interact with complex visual interfaces.
However, these models face significant robustness challenges in real-world
scenarios due to natural noise and adversarial perturbations, and their
robustness remains underexplored. In this study, we systematically evaluate the
robustness of state-of-the-art GUI grounding models, such as UGround, under
three conditions: natural noise, untargeted adversarial attacks, and targeted
adversarial attacks. Our experiments, which were conducted across a wide range
of GUI environments, including mobile, desktop, and web interfaces, have
clearly demonstrated that GUI grounding models exhibit a high degree of
sensitivity to adversarial perturbations and low-resolution conditions. These
findings provide valuable insights into the vulnerabilities of GUI grounding
models and establish a strong benchmark for future research aimed at enhancing
their robustness in practical applications. Our code is available at
https://github.com/ZZZhr-1/Robust_GUI_Grounding.


http://arxiv.org/abs/2504.04645
Here Comes the Explanation: A Shapley Perspective on Multi-contrast Medical Image Segmentation. (1%)
Tianyi Ren; Juampablo Heras Rivera; Hitender Oswal; Yutong Pan; Agamdeep Chopra; Jacob Ruzevick; Mehmet Kurt
  Deep learning has been successfully applied to medical image segmentation,
enabling accurate identification of regions of interest such as organs and
lesions. This approach works effectively across diverse datasets, including
those with single-image contrast, multi-contrast, and multimodal imaging data.
To improve human understanding of these black-box models, there is a growing
need for Explainable AI (XAI) techniques for model transparency and
accountability. Previous research has primarily focused on post hoc pixel-level
explanations, using methods gradient-based and perturbation-based apporaches.
These methods rely on gradients or perturbations to explain model predictions.
However, these pixel-level explanations often struggle with the complexity
inherent in multi-contrast magnetic resonance imaging (MRI) segmentation tasks,
and the sparsely distributed explanations have limited clinical relevance. In
this study, we propose using contrast-level Shapley values to explain
state-of-the-art models trained on standard metrics used in brain tumor
segmentation. Our results demonstrate that Shapley analysis provides valuable
insights into different models' behavior used for tumor segmentation. We
demonstrated a bias for U-Net towards over-weighing T1-contrast and FLAIR,
while Swin-UNETR provided a cross-contrast understanding with balanced Shapley
distribution.


http://arxiv.org/abs/2504.08782
Embedding Hidden Adversarial Capabilities in Pre-Trained Diffusion Models. (13%)
Lucas Beerens; Desmond J. Higham
  We introduce a new attack paradigm that embeds hidden adversarial
capabilities directly into diffusion models via fine-tuning, without altering
their observable behavior or requiring modifications during inference. Unlike
prior approaches that target specific images or adjust the generation process
to produce adversarial outputs, our method integrates adversarial functionality
into the model itself. The resulting tampered model generates high-quality
images indistinguishable from those of the original, yet these images cause
misclassification in downstream classifiers at a high rate. The
misclassification can be targeted to specific output classes. Users can employ
this compromised model unaware of its embedded adversarial nature, as it
functions identically to a standard diffusion model. We demonstrate the
effectiveness and stealthiness of our approach, uncovering a covert attack
vector that raises new security concerns. These findings expose a risk arising
from the use of externally-supplied models and highlight the urgent need for
robust model verification and defense mechanisms against hidden threats in
generative models. The code is available at
https://github.com/LucasBeerens/CRAFTed-Diffusion .


http://arxiv.org/abs/2504.04062
QE-RAG: A Robust Retrieval-Augmented Generation Benchmark for Query Entry Errors. (1%)
Kepu Zhang; Zhongxiang Sun; Weijie Yu; Xiaoxue Zang; Kai Zheng; Yang Song; Han Li; Jun Xu
  Retriever-augmented generation (RAG) has become a widely adopted approach for
enhancing the factual accuracy of large language models (LLMs). While current
benchmarks evaluate the performance of RAG methods from various perspectives,
they share a common assumption that user queries used for retrieval are
error-free. However, in real-world interactions between users and LLMs, query
entry errors such as keyboard proximity errors, visual similarity errors, and
spelling errors are frequent. The impact of these errors on current RAG methods
against such errors remains largely unexplored. To bridge this gap, we propose
QE-RAG, the first robust RAG benchmark designed specifically to evaluate
performance against query entry errors. We augment six widely used datasets by
injecting three common types of query entry errors into randomly selected user
queries at rates of 20\% and 40\%, simulating typical user behavior in
real-world scenarios. We analyze the impact of these errors on LLM outputs and
find that corrupted queries degrade model performance, which can be mitigated
through query correction and training a robust retriever for retrieving
relevant documents. Based on these insights, we propose a contrastive
learning-based robust retriever training method and a retrieval-augmented query
correction method. Extensive in-domain and cross-domain experiments reveal
that: (1) state-of-the-art RAG methods including sequential, branching, and
iterative methods, exhibit poor robustness to query entry errors; (2) our
method significantly enhances the robustness of RAG when handling query entry
errors and it's compatible with existing RAG methods, further improving their
robustness.


http://arxiv.org/abs/2504.04285
Impact of Error Rate Misreporting on Resource Allocation in Multi-tenant Quantum Computing and Defense. (1%)
Subrata Das; Swaroop Ghosh
  Cloud-based quantum service providers allow multiple users to run programs on
shared hardware concurrently to maximize resource utilization and minimize
operational costs. This multi-tenant computing (MTC) model relies on the error
parameters of the hardware for fair qubit allocation and scheduling, as
error-prone qubits can degrade computational accuracy asymmetrically for users
sharing the hardware. To maintain low error rates, quantum providers perform
periodic hardware calibration, often relying on third-party calibration
services. If an adversary within this calibration service misreports error
rates, the allocator can be misled into making suboptimal decisions even when
the physical hardware remains unchanged. We demonstrate such an attack model in
which an adversary strategically misreports qubit error rates to reduce
hardware throughput, and probability of successful trial (PST) for two
previously proposed allocation frameworks, i.e. Greedy and Community-Based
Dynamic Allocation Partitioning (COMDAP). Experimental results show that
adversarial misreporting increases execution latency by 24% and reduces PST by
7.8%. We also propose to identify inconsistencies in reported error rates by
analyzing statistical deviations in error rates across calibration cycles.


http://arxiv.org/abs/2504.04033
Disparate Privacy Vulnerability: Targeted Attribute Inference Attacks and Defenses. (10%)
Ehsanul Kabir; Lucas Craig; Shagufta Mehnaz
  As machine learning (ML) technologies become more prevalent in
privacy-sensitive areas like healthcare and finance, eventually incorporating
sensitive information in building data-driven algorithms, it is vital to
scrutinize whether these data face any privacy leakage risks. One potential
threat arises from an adversary querying trained models using the public,
non-sensitive attributes of entities in the training data to infer their
private, sensitive attributes, a technique known as the attribute inference
attack. This attack is particularly deceptive because, while it may perform
poorly in predicting sensitive attributes across the entire dataset, it excels
at predicting the sensitive attributes of records from a few vulnerable groups,
a phenomenon known as disparate vulnerability. This paper illustrates that an
adversary can take advantage of this disparity to carry out a series of new
attacks, showcasing a threat level beyond previous imagination. We first
develop a novel inference attack called the disparity inference attack, which
targets the identification of high-risk groups within the dataset. We then
introduce two targeted variations of the attribute inference attack that can
identify and exploit a vulnerable subset of the training data, marking the
first instances of targeted attacks in this category, achieving significantly
higher accuracy than untargeted versions. We are also the first to introduce a
novel and effective disparity mitigation technique that simultaneously
preserves model performance and prevents any risk of targeted attacks.


http://arxiv.org/abs/2504.03957
Practical Poisoning Attacks against Retrieval-Augmented Generation. (4%)
Baolei Zhang; Yuxi Chen; Minghong Fang; Zhuqing Liu; Lihai Nie; Tong Li; Zheli Liu
  Large language models (LLMs) have demonstrated impressive natural language
processing abilities but face challenges such as hallucination and outdated
knowledge. Retrieval-Augmented Generation (RAG) has emerged as a
state-of-the-art approach to mitigate these issues. While RAG enhances LLM
outputs, it remains vulnerable to poisoning attacks. Recent studies show that
injecting poisoned text into the knowledge database can compromise RAG systems,
but most existing attacks assume that the attacker can insert a sufficient
number of poisoned texts per query to outnumber correct-answer texts in
retrieval, an assumption that is often unrealistic. To address this limitation,
we propose CorruptRAG, a practical poisoning attack against RAG systems in
which the attacker injects only a single poisoned text, enhancing both
feasibility and stealth. Extensive experiments across multiple datasets
demonstrate that CorruptRAG achieves higher attack success rates compared to
existing baselines.


http://arxiv.org/abs/2504.03624
Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models. (1%)
NVIDIA; :; Aaron Blakeman; Aarti Basant; Abhinav Khattar; Adithya Renduchintala; Akhiad Bercovich; Aleksander Ficek; Alexis Bjorlin; Ali Taghibakhshi; Amala Sanjay Deshmukh; Ameya Sunil Mahabaleshwarkar; Andrew Tao; Anna Shors; Ashwath Aithal; Ashwin Poojary; Ayush Dattagupta; Balaram Buddharaju; Bobby Chen; Boris Ginsburg; Boxin Wang; Brandon Norick; Brian Butterfield; Bryan Catanzaro; Mundo Carlo del; Chengyu Dong; Christine Harvey; Christopher Parisien; Dan Su; Daniel Korzekwa; Danny Yin; Daria Gitman; David Mosallanezhad; Deepak Narayanan; Denys Fridman; Dima Rekesh; Ding Ma; Dmytro Pykhtar; Dong Ahn; Duncan Riach; Dusan Stosic; Eileen Long; Elad Segal; Ellie Evans; Eric Chung; Erick Galinkin; Evelina Bakhturina; Ewa Dobrowolska; Fei Jia; Fuxiao Liu; Gargi Prasad; Gerald Shen; Guilin Liu; Guo Chen; Haifeng Qian; Helen Ngo; Hongbin Liu; Hui Li; Igor Gitman; Ilia Karmanov; Ivan Moshkov; Izik Golan; Jan Kautz; Jane Polak Scowcroft; Jared Casper; Jarno Seppanen; Jason Lu; Jason Sewall; Jiaqi Zeng; Jiaxuan You; Jimmy Zhang; Jing Zhang; Jining Huang; Jinze Xue; Jocelyn Huang; Joey Conway; John Kamalu; Jon Barker; Jonathan Cohen; Joseph Jennings; Jupinder Parmar; Karan Sapra; Kari Briski; Kateryna Chumachenko; Katherine Luna; Keshav Santhanam; Kezhi Kong; Kirthi Sivamani; Krzysztof Pawelec; Kumar Anik; Kunlun Li; Lawrence McAfee; Leon Derczynski; Lindsey Pavao; Luis Vega; Lukas Voegtle; Maciej Bala; Melo Maer Rodrigues de; Makesh Narsimhan Sreedhar; Marcin Chochowski; Markus Kliegl; Marta Stepniewska-Dziubinska; Matthieu Le; Matvei Novikov; Mehrzad Samadi; Michael Andersch; Michael Evans; Miguel Martinez; Mike Chrzanowski; Mike Ranzinger; Mikolaj Blaz; Misha Smelyanskiy; Mohamed Fawzy; Mohammad Shoeybi; Mostofa Patwary; Nayeon Lee; Nima Tajbakhsh; Ning Xu; Oleg Rybakov; Oleksii Kuchaiev; Olivier Delalleau; Osvald Nitski; Parth Chadha; Pasha Shamis; Paulius Micikevicius; Pavlo Molchanov; Peter Dykas; Philipp Fischer; Pierre-Yves Aquilanti; Piotr Bialecki; Prasoon Varshney; Pritam Gundecha; Przemek Tredak; Rabeeh Karimi; Rahul Kandu; Ran El-Yaniv; Raviraj Joshi; Roger Waleffe; Ruoxi Zhang; Sabrina Kavanaugh; Sahil Jain; Samuel Kriman; Sangkug Lym; Sanjeev Satheesh; Saurav Muralidharan; Sean Narenthiran; Selvaraj Anandaraj; Seonmyeong Bak; Sergey Kashirsky; Seungju Han; Shantanu Acharya; Shaona Ghosh; Sharath Turuvekere Sreenivas; Sharon Clay; Shelby Thomas; Shrimai Prabhumoye; Shubham Pachori; Shubham Toshniwal; Shyamala Prayaga; Siddhartha Jain; Sirshak Das; Slawek Kierat; Somshubra Majumdar; Song Han; Soumye Singhal; Sriharsha Niverty; Stefania Alborghetti; Suseella Panguluri; Swetha Bhendigeri; Syeda Nahida Akter; Szymon Migacz; Tal Shiri; Terry Kong; Timo Roman; Tomer Ronen; Trisha Saar; Tugrul Konuk; Tuomas Rintamaki; Tyler Poon; Ushnish De; Vahid Noroozi; Varun Singh; Vijay Korthikanti; Vitaly Kurin; Wasi Uddin Ahmad; Wei Du; Wei Ping; Wenliang Dai; Wonmin Byeon; Xiaowei Ren; Yao Xu; Yejin Choi; Yian Zhang; Ying Lin; Yoshi Suhara; Zhiding Yu; Zhiqi Li; Zhiyu Li; Zhongbo Zhu; Zhuolin Yang; Zijia Chen
  As inference-time scaling becomes critical for enhanced reasoning
capabilities, it is increasingly becoming important to build models that are
efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid
Mamba-Transformer models designed to reduce inference cost for a given accuracy
level. To achieve this goal, we replace the majority of self-attention layers
in the common Transformer model architecture with Mamba layers that perform
constant computation and require constant memory per generated token. We show
that Nemotron-H models offer either better or on-par accuracy compared to other
similarly-sized state-of-the-art open-sourced Transformer models (e.g.,
Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3$\times$ faster at
inference. To further increase inference speed and reduce the memory required
at inference time, we created Nemotron-H-47B-Base from the 56B model using a
new compression via pruning and distillation technique called MiniPuzzle.
Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20%
faster to infer. In addition, we introduce an FP8-based training recipe and
show that it can achieve on par results with BF16-based training. This recipe
is used to train the 56B model. All Nemotron-H models will be released, with
support in Hugging Face, NeMo, and Megatron-LM.


http://arxiv.org/abs/2504.03173
PPFPL: Cross-silo Privacy-preserving Federated Prototype Learning Against Data Poisoning Attacks on Non-IID Data. (1%)
Hongliang Zhang; Jiguo Yu; Fenghua Xu; Chunqiang Hu; Yongzhao Zhang; Xiaofen Wang; Zhongyuan Yu; Xiaosong Zhang
  Privacy-Preserving Federated Learning (PPFL) allows multiple clients to
collaboratively train a deep learning model by submitting hidden model updates.
Nonetheless, PPFL is vulnerable to data poisoning attacks due to the
distributed training nature of clients. Existing solutions have struggled to
improve the performance of cross-silo PPFL in poisoned Non-IID data. To address
the issues, this paper proposes a privacy-preserving federated prototype
learning framework, named PPFPL, which enhances the cross-silo FL performance
in poisoned Non-IID data while effectively resisting data poisoning attacks.
Specifically, we adopt prototypes as client-submitted model updates to
eliminate the impact of tampered data distribution on federated learning.
Moreover, we utilize two servers to achieve Byzantine-robust aggregation by
secure aggregation protocol, which greatly reduces the impact of malicious
clients. Theoretical analyses confirm the convergence of PPFPL, and
experimental results on publicly available datasets show that PPFPL is
effective for resisting data poisoning attacks with Non-IID conditions.


http://arxiv.org/abs/2504.03065
Moving Target Defense Against Adversarial False Data Injection Attacks In Power Grids. (99%)
Yexiang Chen; Subhash Lakshminarayana; H. Vincent Poor
  Machine learning (ML)-based detectors have been shown to be effective in
detecting stealthy false data injection attacks (FDIAs) that can bypass
conventional bad data detectors (BDDs) in power systems. However, ML models are
also vulnerable to adversarial attacks. A sophisticated perturbation signal
added to the original BDD-bypassing FDIA can conceal the attack from ML-based
detectors. In this paper, we develop a moving target defense (MTD) strategy to
defend against adversarial FDIAs in power grids. We first develop an
MTD-strengthened deep neural network (DNN) model, which deploys a pool of DNN
models rather than a single static model that cooperate to detect the
adversarial attack jointly. The MTD model pool introduces randomness to the ML
model's decision boundary, thereby making the adversarial attacks detectable.
Furthermore, to increase the effectiveness of the MTD strategy and reduce the
computational costs associated with developing the MTD model pool, we combine
this approach with the physics-based MTD, which involves dynamically perturbing
the transmission line reactance and retraining the DNN-based detector to adapt
to the new system topology. Simulations conducted on IEEE test bus systems
demonstrate that the MTD-strengthened DNN achieves up to 94.2% accuracy in
detecting adversarial FDIAs. When combined with a physics-based MTD, the
detection accuracy surpasses 99%, while significantly reducing the
computational costs of updating the DNN models. This approach requires only
moderate perturbations to transmission line reactances, resulting in minimal
increases in OPF cost.


http://arxiv.org/abs/2504.02335
Evaluating and Enhancing Segmentation Model Robustness with Metamorphic Testing. (98%)
Seif Mzoughi; Mohamed Elshafeia; Foutse Khomh
  Image segmentation is critical for applications such as medical imaging,
augmented reality, and video surveillance. However, segmentation models often
lack robustness, making them vulnerable to adversarial perturbations from
subtle image distortions. In this work, we propose SegRMT, a metamorphic
testing approach that leverages genetic algorithms (GA) to optimize sequences
of spatial and spectral transformations while preserving image fidelity via a
predefined PSNR threshold. Using the Cityscapes dataset, our method generates
adversarial examples that effectively challenge the DeepLabV3 segmentation
model. Our experiments show that SegRMT reduces DeepLabV3's mean Intersection
over Union (mIoU) to 6.4%, outperforming other adversarial baselines that
decrease mIoU to between 8.5% and 21.7%. Furthermore, when used for adversarial
training, SegRMT boosts model performance, achieving mIoU improvements up to
73% on dedicated adversarial datasets and increasing cross-adversarial mIoU to
53.8%, compared to only 2%-10% for other methods. These findings demonstrate
that SegRMT not only simulates realistic image distortions but also enhances
the robustness of segmentation models, making it a valuable tool for ensuring
reliable performance in safety-critical applications.


http://arxiv.org/abs/2504.03089
SLACK: Attacking LiDAR-based SLAM with Adversarial Point Injections. (22%)
Prashant Kumar; Dheeraj Vattikonda; Kshitij Madhav Bhat; Kunal Dargan; Prem Kalra
  The widespread adoption of learning-based methods for the LiDAR makes
autonomous vehicles vulnerable to adversarial attacks through adversarial
\textit{point injections (PiJ)}. It poses serious security challenges for
navigation and map generation. Despite its critical nature, no major work
exists that studies learning-based attacks on LiDAR-based SLAM. Our work
proposes SLACK, an end-to-end deep generative adversarial model to attack LiDAR
scans with several point injections without deteriorating LiDAR quality. To
facilitate SLACK, we design a novel yet simple autoencoder that augments
contrastive learning with segmentation-based attention for precise
reconstructions. SLACK demonstrates superior performance on the task of
\textit{point injections (PiJ)} compared to the best baselines on KITTI and
CARLA-64 dataset while maintaining accurate scan quality. We qualitatively and
quantitatively demonstrate PiJ attacks using a fraction of LiDAR points. It
severely degrades navigation and map quality without deteriorating the LiDAR
scan quality.


http://arxiv.org/abs/2504.02458
Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendation. (12%)
Liangbo Ning; Wenqi Fan; Qing Li
  Recently, Large Language Model (LLM)-empowered recommender systems have
revolutionized personalized recommendation frameworks and attracted extensive
attention. Despite the remarkable success, existing LLM-empowered RecSys have
been demonstrated to be highly vulnerable to minor perturbations. To mitigate
the negative impact of such vulnerabilities, one potential solution is to
employ collaborative signals based on item-item co-occurrence to purify the
malicious collaborative knowledge from the user's historical interactions
inserted by attackers. On the other hand, due to the capabilities to expand
insufficient internal knowledge of LLMs, Retrieval-Augmented Generation (RAG)
techniques provide unprecedented opportunities to enhance the robustness of
LLM-empowered recommender systems by introducing external collaborative
knowledge. Therefore, in this paper, we propose a novel framework (RETURN) by
retrieving external collaborative signals to purify the poisoned user profiles
and enhance the robustness of LLM-empowered RecSys in a plug-and-play manner.
Specifically, retrieval-augmented perturbation positioning is proposed to
identify potential perturbations within the users' historical sequences by
retrieving external knowledge from collaborative item graphs. After that, we
further retrieve the collaborative knowledge to cleanse the perturbations by
using either deletion or replacement strategies and introduce a robust ensemble
recommendation strategy to generate final robust predictions. Extensive
experiments on three real-world datasets demonstrate the effectiveness of the
proposed RETURN.


http://arxiv.org/abs/2504.02412
Bridging the Theoretical Gap in Randomized Smoothing. (10%)
Blaise Delattre; Paul Caillon; Quentin Barthélemy; Erwan Fagnou; Alexandre Allauzen
  Randomized smoothing has become a leading approach for certifying adversarial
robustness in machine learning models. However, a persistent gap remains
between theoretical certified robustness and empirical robustness accuracy.
This paper introduces a new framework that bridges this gap by leveraging
Lipschitz continuity for certification and proposing a novel, less conservative
method for computing confidence intervals in randomized smoothing. Our approach
tightens the bounds of certified robustness, offering a more accurate
reflection of model robustness in practice. Through rigorous experimentation we
show that our method improves the robust accuracy, compressing the gap between
empirical findings and previous theoretical results. We argue that
investigating local Lipschitz constants and designing ad-hoc confidence
intervals can further enhance the performance of randomized smoothing. These
results pave the way for a deeper understanding of the relationship between
Lipschitz continuity and certified robustness.


http://arxiv.org/abs/2504.03770
JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model. (2%)
Yi Nian; Shenzhe Zhu; Yuehan Qin; Li Li; Ziyi Wang; Chaowei Xiao; Yue Zhao
  Multimodal large language models (MLLMs) excel in vision-language tasks but
also pose significant risks of generating harmful content, particularly through
jailbreak attacks. Jailbreak attacks refer to intentional manipulations that
bypass safety mechanisms in models, leading to the generation of inappropriate
or unsafe content. Detecting such attacks is critical to ensuring the
responsible deployment of MLLMs. Existing jailbreak detection methods face
three primary challenges: (1) Many rely on model hidden states or gradients,
limiting their applicability to white-box models, where the internal workings
of the model are accessible; (2) They involve high computational overhead from
uncertainty-based analysis, which limits real-time detection, and (3) They
require fully labeled harmful datasets, which are often scarce in real-world
settings. To address these issues, we introduce a test-time adaptive framework
called JAILDAM. Our method leverages a memory-based approach guided by
policy-driven unsafe knowledge representations, eliminating the need for
explicit exposure to harmful data. By dynamically updating unsafe knowledge
during test-time, our framework improves generalization to unseen jailbreak
strategies while maintaining efficiency. Experiments on multiple VLM jailbreak
benchmarks demonstrate that JAILDAM delivers state-of-the-art performance in
harmful content detection, improving both accuracy and speed.


http://arxiv.org/abs/2504.02725
ERPO: Advancing Safety Alignment via Ex-Ante Reasoning Preference Optimization. (1%)
Kehua Feng; Keyan Ding; Jing Yu; Menghan Li; Yuhao Wang; Tong Xu; Xinda Wang; Qiang Zhang; Huajun Chen
  Recent advancements in large language models (LLMs) have accelerated progress
toward artificial general intelligence, yet their potential to generate harmful
content poses critical safety challenges. Existing alignment methods often
struggle to cover diverse safety scenarios and remain vulnerable to adversarial
attacks. In this work, we propose Ex-Ante Reasoning Preference Optimization
(ERPO), a novel safety alignment framework that equips LLMs with explicit
preemptive reasoning through Chain-of-Thought and provides clear evidence for
safety judgments by embedding predefined safety rules. Specifically, our
approach consists of three stages: first, equipping the model with Ex-Ante
reasoning through supervised fine-tuning (SFT) using a constructed reasoning
module; second, enhancing safety, usefulness, and efficiency via Direct
Preference Optimization (DPO); and third, mitigating inference latency with a
length-controlled iterative preference optimization strategy. Experiments on
multiple open-source LLMs demonstrate that ERPO significantly enhances safety
performance while maintaining response efficiency.


http://arxiv.org/abs/2504.01735
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization. (99%)
Chaohu Liu; Tianyi Gui; Yu Liu; Linli Xu
  Large Vision-Language Models (LVLMs), such as GPT-4o and LLaVA, have recently
witnessed remarkable advancements and are increasingly being deployed in
real-world applications. However, inheriting the sensitivity of visual neural
networks, LVLMs remain vulnerable to adversarial attacks, which can result in
erroneous or malicious outputs. While existing efforts utilize adversarial
fine-tuning to enhance robustness, they often suffer from performance
degradation on clean inputs. In this paper, we proposes AdPO, a novel
adversarial defense strategy for LVLMs based on preference optimization. For
the first time, we reframe adversarial training as a preference optimization
problem, aiming to enhance the model's preference for generating normal outputs
on clean inputs while rejecting the potential misleading outputs for
adversarial examples. Notably, AdPO achieves this by solely modifying the image
encoder, e.g., CLIP ViT, resulting in superior clean and adversarial
performance in a variety of downsream tasks. Considering that training involves
large language models (LLMs), the computational cost increases significantly.
We validate that training on smaller LVLMs and subsequently transferring to
larger models can achieve competitive performance while maintaining efficiency
comparable to baseline methods. Our comprehensive experiments confirm the
effectiveness of the proposed AdPO, which provides a novel perspective for
future adversarial defense research.


http://arxiv.org/abs/2504.01399
Leveraging Generalizability of Image-to-Image Translation for Enhanced Adversarial Defense. (98%)
Haibo Zhang; Zhihua Yao; Kouichi Sakurai; Takeshi Saitoh
  In the rapidly evolving field of artificial intelligence, machine learning
emerges as a key technology characterized by its vast potential and inherent
risks. The stability and reliability of these models are important, as they are
frequent targets of security threats. Adversarial attacks, first rigorously
defined by Ian Goodfellow et al. in 2013, highlight a critical vulnerability:
they can trick machine learning models into making incorrect predictions by
applying nearly invisible perturbations to images. Although many studies have
focused on constructing sophisticated defensive mechanisms to mitigate such
attacks, they often overlook the substantial time and computational costs of
training and maintaining these models. Ideally, a defense method should be able
to generalize across various, even unseen, adversarial attacks with minimal
overhead. Building on our previous work on image-to-image translation-based
defenses, this study introduces an improved model that incorporates residual
blocks to enhance generalizability. The proposed method requires training only
a single model, effectively defends against diverse attack types, and is
well-transferable between different target models. Experiments show that our
model can restore the classification accuracy from near zero to an average of
72\% while maintaining competitive performance compared to state-of-the-art
methods.


http://arxiv.org/abs/2504.01632
Benchmarking the Spatial Robustness of DNNs via Natural and Adversarial Localized Corruptions. (93%)
Giulia Marchiori Pietrosanti; Giulio Rossolini; Alessandro Biondi; Giorgio Buttazzo
  The robustness of DNNs is a crucial factor in safety-critical applications,
particularly in complex and dynamic environments where localized corruptions
can arise. While previous studies have evaluated the robustness of semantic
segmentation (SS) models under whole-image natural or adversarial corruptions,
a comprehensive investigation into the spatial robustness of dense vision
models under localized corruptions remained underexplored. This paper fills
this gap by introducing specialized metrics for benchmarking the spatial
robustness of segmentation models, alongside with an evaluation framework to
assess the impact of localized corruptions. Furthermore, we uncover the
inherent complexity of characterizing worst-case robustness using a single
localized adversarial perturbation. To address this, we propose region-aware
multi-attack adversarial analysis, a method that enables a deeper understanding
of model robustness against adversarial perturbations applied to specific
regions. The proposed metrics and analysis were exploited to evaluate 14
segmentation models in driving scenarios, uncovering key insights into the
effects of localized corruption in both natural and adversarial forms. The
results reveal that models respond to these two types of threats differently;
for instance, transformer-based segmentation models demonstrate notable
robustness to localized natural corruptions but are highly vulnerable to
adversarial ones and vice-versa for CNN-based models. Consequently, we also
address the challenge of balancing robustness to both natural and adversarial
localized corruptions by means of ensemble models, thereby achieving a broader
threat coverage and improved reliability for dense vision tasks.


http://arxiv.org/abs/2504.01345
Breaking BERT: Gradient Attack on Twitter Sentiment Analysis for Targeted Misclassification. (89%)
Akil Raj Subedi; Taniya Shah; Aswani Kumar Cherukuri; Thanos Vasilakos
  Social media platforms like Twitter have increasingly relied on Natural
Language Processing NLP techniques to analyze and understand the sentiments
expressed in the user generated content. One such state of the art NLP model is
Bidirectional Encoder Representations from Transformers BERT which has been
widely adapted in sentiment analysis. BERT is susceptible to adversarial
attacks. This paper aims to scrutinize the inherent vulnerabilities of such
models in Twitter sentiment analysis. It aims to formulate a framework for
constructing targeted adversarial texts capable of deceiving these models,
while maintaining stealth. In contrast to conventional methodologies, such as
Importance Reweighting, this framework core idea resides in its reliance on
gradients to prioritize the importance of individual words within the text. It
uses a whitebox approach to attain fine grained sensitivity, pinpointing words
that exert maximal influence on the classification outcome. This paper is
organized into three interdependent phases. It starts with fine-tuning a
pre-trained BERT model on Twitter data. It then analyzes gradients of the model
to rank words on their importance, and iteratively replaces those with feasible
candidates until an acceptable solution is found. Finally, it evaluates the
effectiveness of the adversarial text against the custom trained sentiment
classification model. This assessment would help in gauging the capacity of the
adversarial text to successfully subvert classification without raising any
alarm.


http://arxiv.org/abs/2504.01668
Overlap-Aware Feature Learning for Robust Unsupervised Domain Adaptation for 3D Semantic Segmentation. (83%)
Junjie Chen; Yuecong Xu; Haosheng Li; Kemi Ding
  3D point cloud semantic segmentation (PCSS) is a cornerstone for
environmental perception in robotic systems and autonomous driving, enabling
precise scene understanding through point-wise classification. While
unsupervised domain adaptation (UDA) mitigates label scarcity in PCSS, existing
methods critically overlook the inherent vulnerability to real-world
perturbations (e.g., snow, fog, rain) and adversarial distortions. This work
first identifies two intrinsic limitations that undermine current PCSS-UDA
robustness: (a) unsupervised features overlap from unaligned boundaries in
shared-class regions and (b) feature structure erosion caused by
domain-invariant learning that suppresses target-specific patterns. To address
the proposed problems, we propose a tripartite framework consisting of: 1) a
robustness evaluation model quantifying resilience against adversarial
attack/corruption types through robustness metrics; 2) an invertible attention
alignment module (IAAM) enabling bidirectional domain mapping while preserving
discriminative structure via attention-guided overlap suppression; and 3) a
contrastive memory bank with quality-aware contrastive learning that
progressively refines pseudo-labels with feature quality for more
discriminative representations. Extensive experiments on
SynLiDAR-to-SemanticPOSS adaptation demonstrate a maximum mIoU improvement of
14.3\% under adversarial attack.


http://arxiv.org/abs/2504.02132
One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Image. (80%)
Ezzeldin Shereen; Dan Ristea; Burak Hasircioglu; Shae McFadden; Vasilios Mavroudis; Chris Hicks
  Multimodal retrieval augmented generation (M-RAG) has recently emerged as a
method to inhibit hallucinations of large multimodal models (LMMs) through a
factual knowledge base (KB). However, M-RAG also introduces new attack vectors
for adversaries that aim to disrupt the system by injecting malicious entries
into the KB. In this work, we present a poisoning attack against M-RAG
targeting visual document retrieval applications, where the KB contains images
of document pages. Our objective is to craft a single image that is retrieved
for a variety of different user queries, and consistently influences the output
produced by the generative model, thus creating a universal denial-of-service
(DoS) attack against the M-RAG system. We demonstrate that while our attack is
effective against a diverse range of widely-used, state-of-the-art retrievers
(embedding models) and generators (LMMs), it can also be ineffective against
robust embedding models. Our attack not only highlights the vulnerability of
M-RAG pipelines to poisoning attacks, but also sheds light on a fundamental
weakness that potentially hinders their performance even in benign settings.


http://arxiv.org/abs/2504.01659
Robust Unsupervised Domain Adaptation for 3D Point Cloud Segmentation Under Source Adversarial Attacks. (80%)
Haosheng Li; Yuecong Xu; Junjie Chen; Kemi Ding
  Unsupervised domain adaptation (UDA) frameworks have shown good
generalization capabilities for 3D point cloud semantic segmentation models on
clean data. However, existing works overlook adversarial robustness when the
source domain itself is compromised. To comprehensively explore the robustness
of the UDA frameworks, we first design a stealthy adversarial point cloud
generation attack that can significantly contaminate datasets with only minor
perturbations to the point cloud surface. Based on that, we propose a novel
dataset, AdvSynLiDAR, comprising synthesized contaminated LiDAR point clouds.
With the generated corrupted data, we further develop the Adversarial
Adaptation Framework (AAF) as the countermeasure. Specifically, by extending
the key point sensitive (KPS) loss towards the Robust Long-Tail loss (RLT loss)
and utilizing a decoder branch, our approach enables the model to focus on
long-tail classes during the pre-training phase and leverages high-confidence
decoded point cloud information to restore point cloud structures during the
adaptation phase. We evaluated our AAF method on the AdvSynLiDAR dataset, where
the results demonstrate that our AAF method can mitigate performance
degradation under source adversarial perturbations for UDA in the 3D point
cloud segmentation application.


http://arxiv.org/abs/2504.01589
Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models. (67%)
Zhaochen Wang; Bryan Hooi; Yiwei Wang; Ming-Hsuan Yang; Zi Huang; Yujun Cai
  Vision-language models (VLMs) have advanced rapidly in processing multimodal
information, but their ability to reconcile conflicting signals across
modalities remains underexplored. This work investigates how VLMs process ASCII
art, a unique medium where textual elements collectively form visual patterns,
potentially creating semantic-visual conflicts. We introduce a novel evaluation
framework that systematically challenges five state-of-the-art models
(including GPT-4o, Claude, and Gemini) using adversarial ASCII art, where
character-level semantics deliberately contradict global visual patterns. Our
experiments reveal a strong text-priority bias: VLMs consistently prioritize
textual information over visual patterns, with visual recognition ability
declining dramatically as semantic complexity increases. Various mitigation
attempts through visual parameter tuning and prompt engineering yielded only
modest improvements, suggesting that this limitation requires
architectural-level solutions. These findings uncover fundamental flaws in how
current VLMs integrate multimodal information, providing important guidance for
future model development while highlighting significant implications for
content moderation systems vulnerable to adversarial examples.


http://arxiv.org/abs/2504.16936
Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness. (22%)
Yusheng Zhao; Junyu Luo; Xiao Luo; Weizhi Zhang; Zhiping Xiao; Wei Ju; Philip S. Yu; Ming Zhang
  Multi-modal large language models (MLLMs) have recently achieved great
success in processing and understanding information from diverse modalities
(e.g., text, audio, and visual signals). Despite their growing popularity,
there remains a lack of comprehensive evaluation measuring the audio-visual
capabilities of these models, especially in diverse scenarios (e.g.,
distribution shifts and adversarial attacks). In this paper, we present a
multifaceted evaluation of the audio-visual capability of MLLMs, focusing on
four key dimensions: effectiveness, efficiency, generalizability, and
robustness. Through extensive experiments, we find that MLLMs exhibit strong
zero-shot and few-shot generalization abilities, enabling them to achieve great
performance with limited data. However, their success relies heavily on the
vision modality, which impairs performance when visual input is corrupted or
missing. Additionally, while MLLMs are susceptible to adversarial samples, they
demonstrate greater robustness compared to traditional models. The experimental
results and our findings provide insights into the audio-visual capabilities of
MLLMs, highlighting areas for improvement and offering guidance for future
research.


http://arxiv.org/abs/2504.01550
Representation Bending for Large Language Model Safety. (12%)
Ashkan Yousefpour; Taeheon Kim; Ryan S. Kwon; Seungbeen Lee; Wonje Jeung; Seungju Han; Alvin Wan; Harrison Ngan; Youngjae Yu; Jonghyun Choi
  Large Language Models (LLMs) have emerged as powerful tools, but their
inherent safety risks - ranging from harmful content generation to broader
societal harms - pose significant challenges. These risks can be amplified by
the recent adversarial attacks, fine-tuning vulnerabilities, and the increasing
deployment of LLMs in high-stakes environments. Existing safety-enhancing
techniques, such as fine-tuning with human feedback or adversarial training,
are still vulnerable as they address specific threats and often fail to
generalize across unseen attacks, or require manual system-level defenses. This
paper introduces RepBend, a novel approach that fundamentally disrupts the
representations underlying harmful behaviors in LLMs, offering a scalable
solution to enhance (potentially inherent) safety. RepBend brings the idea of
activation steering - simple vector arithmetic for steering model's behavior
during inference - to loss-based fine-tuning. Through extensive evaluation,
RepBend achieves state-of-the-art performance, outperforming prior methods such
as Circuit Breaker, RMU, and NPO, with up to 95% reduction in attack success
rates across diverse jailbreak benchmarks, all with negligible reduction in
model usability and general capabilities.


http://arxiv.org/abs/2504.01533
LightDefense: A Lightweight Uncertainty-Driven Defense against Jailbreaks via Shifted Token Distribution. (9%)
Zhuoran Yang; Jie Peng; Zhen Tan; Tianlong Chen; Yanyong Zhang
  Large Language Models (LLMs) face threats from jailbreak prompts. Existing
methods for defending against jailbreak attacks are primarily based on
auxiliary models. These strategies, however, often require extensive data
collection or training. We propose LightDefense, a lightweight defense
mechanism targeted at white-box models, which utilizes a safety-oriented
direction to adjust the probabilities of tokens in the vocabulary, making
safety disclaimers appear among the top tokens after sorting tokens by
probability in descending order. We further innovatively leverage LLM's
uncertainty about prompts to measure their harmfulness and adaptively adjust
defense strength, effectively balancing safety and helpfulness. The
effectiveness of LightDefense in defending against 5 attack methods across 2
target LLMs, without compromising helpfulness to benign user queries,
highlights its potential as a novel and lightweight defense mechanism,
enhancing security of LLMs.


http://arxiv.org/abs/2504.02080
Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses. (3%)
Zhengchun Shang; Wenlan Wei
  Large Language Models (LLMs) are increasingly popular, powering a wide range
of applications. Their widespread use has sparked concerns, especially through
jailbreak attacks that bypass safety measures to produce harmful content.
  In this paper, we present a comprehensive security analysis of large language
models (LLMs), addressing critical research questions on the evolution and
determinants of model safety.
  Specifically, we begin by identifying the most effective techniques for
detecting jailbreak attacks. Next, we investigate whether newer versions of
LLMs offer improved security compared to their predecessors. We also assess the
impact of model size on overall security and explore the potential benefits of
integrating multiple defense strategies to enhance model robustness.
  Our study evaluates both open-source models (e.g., LLaMA and Mistral) and
closed-source systems (e.g., GPT-4) by employing four state-of-the-art attack
techniques and assessing the efficacy of three new defensive approaches.


http://arxiv.org/abs/2504.02142
Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds. (2%)
Michael-Andrei Panaitescu-Liess; Yigitcan Kaya; Sicheng Zhu; Furong Huang; Tudor Dumitras
  Group robustness has become a major concern in machine learning (ML) as
conventional training paradigms were found to produce high error on minority
groups. Without explicit group annotations, proposed solutions rely on
heuristics that aim to identify and then amplify the minority samples during
training. In our work, we first uncover a critical shortcoming of these
methods: an inability to distinguish legitimate minority samples from poison
samples in the training set. By amplifying poison samples as well, group
robustness methods inadvertently boost the success rate of an adversary --
e.g., from $0\%$ without amplification to over $97\%$ with it. Notably, we
supplement our empirical evidence with an impossibility result proving this
inability of a standard heuristic under some assumptions. Moreover,
scrutinizing recent poisoning defenses both in centralized and federated
learning, we observe that they rely on similar heuristics to identify which
samples should be eliminated as poisons. In consequence, minority samples are
eliminated along with poisons, which damages group robustness -- e.g., from
$55\%$ without the removal of the minority samples to $41\%$ with it. Finally,
as they pursue opposing goals using similar heuristics, our attempt to
alleviate the trade-off by combining group robustness methods and poisoning
defenses falls short. By exposing this tension, we also hope to highlight how
benchmark-driven ML scholarship can obscure the trade-offs among different
metrics with potentially detrimental consequences.


http://arxiv.org/abs/2504.02213
Secure Generalization through Stochastic Bidirectional Parameter Updates Using Dual-Gradient Mechanism. (2%)
Shourya Goel; Himanshi Tibrewal; Anant Jain; Anshul Pundhir; Pravendra Singh
  Federated learning (FL) has gained increasing attention due to
privacy-preserving collaborative training on decentralized clients, mitigating
the need to upload sensitive data to a central server directly. Nonetheless,
recent research has underscored the risk of exposing private data to
adversaries, even within FL frameworks. In general, existing methods sacrifice
performance while ensuring resistance to privacy leakage in FL. We overcome
these issues and generate diverse models at a global server through the
proposed stochastic bidirectional parameter update mechanism. Using diverse
models, we improved the generalization and feature representation in the FL
setup, which also helped to improve the robustness of the model against privacy
leakage without hurting the model's utility. We use global models from past FL
rounds to follow systematic perturbation in parameter space at the server to
ensure model generalization and resistance against privacy attacks. We generate
diverse models (in close neighborhoods) for each client by using systematic
perturbations in model parameters at a fine-grained level (i.e., altering each
convolutional filter across the layers of the model) to improve the
generalization and security perspective. We evaluated our proposed approach on
four benchmark datasets to validate its superiority. We surpassed the
state-of-the-art methods in terms of model utility and robustness towards
privacy leakage. We have proven the effectiveness of our method by evaluating
performance using several quantitative and qualitative results.


http://arxiv.org/abs/2504.03767
MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits. (1%)
Brandon Radosevich; John Halloran
  To reduce development overhead and enable seamless integration between
potential components comprising any given generative AI application, the Model
Context Protocol (MCP) (Anthropic, 2024) has recently been released and
subsequently widely adopted. The MCP is an open protocol that standardizes API
calls to large language models (LLMs), data sources, and agentic tools. By
connecting multiple MCP servers, each defined with a set of tools, resources,
and prompts, users are able to define automated workflows fully driven by LLMs.
However, we show that the current MCP design carries a wide range of security
risks for end users. In particular, we demonstrate that industry-leading LLMs
may be coerced into using MCP tools to compromise an AI developer's system
through various attacks, such as malicious code execution, remote access
control, and credential theft. To proactively mitigate these and related
attacks, we introduce a safety auditing tool, MCPSafetyScanner, the first
agentic tool to assess the security of an arbitrary MCP server. MCPScanner uses
several agents to (a) automatically determine adversarial samples given an MCP
server's tools and resources; (b) search for related vulnerabilities and
remediations based on those samples; and (c) generate a security report
detailing all findings. Our work highlights serious security issues with
general-purpose agentic workflows while also providing a proactive tool to
audit MCP server safety and address detected vulnerabilities before deployment.
  The described MCP server auditing tool, MCPSafetyScanner, is freely available
at: https://github.com/leidosinc/McpSafetyScanner


http://arxiv.org/abs/2504.01444
PiCo: Jailbreaking Multimodal Large Language Models via $\textbf{Pi}$ctorial $\textbf{Co}$de Contextualization. (1%)
Aofan Liu; Lulu Tang; Ting Pan; Yuguo Yin; Bin Wang; Ao Yang
  Multimodal Large Language Models (MLLMs), which integrate vision and other
modalities into Large Language Models (LLMs), significantly enhance AI
capabilities but also introduce new security vulnerabilities. By exploiting the
vulnerabilities of the visual modality and the long-tail distribution
characteristic of code training data, we present PiCo, a novel jailbreaking
framework designed to progressively bypass multi-tiered defense mechanisms in
advanced MLLMs. PiCo employs a tier-by-tier jailbreak strategy, using
token-level typographic attacks to evade input filtering and embedding harmful
intent within programming context instructions to bypass runtime monitoring. To
comprehensively assess the impact of attacks, a new evaluation metric is
further proposed to assess both the toxicity and helpfulness of model outputs
post-attack. By embedding harmful intent within code-style visual instructions,
PiCo achieves an average Attack Success Rate (ASR) of 84.13% on Gemini-Pro
Vision and 52.66% on GPT-4, surpassing previous methods. Experimental results
highlight the critical gaps in current defenses, underscoring the need for more
robust strategies to secure advanced MLLMs.


http://arxiv.org/abs/2504.00429
Unleashing the Power of Pre-trained Encoders for Universal Adversarial Attack Detection. (99%)
Yinghe Zhang; Chi Liu; Shuai Zhou; Sheng Shen; Peng Gui
  Adversarial attacks pose a critical security threat to real-world AI systems
by injecting human-imperceptible perturbations into benign samples to induce
misclassification in deep learning models. While existing detection methods,
such as Bayesian uncertainty estimation and activation pattern analysis, have
achieved progress through feature engineering, their reliance on handcrafted
feature design and prior knowledge of attack patterns limits generalization
capabilities and incurs high engineering costs. To address these limitations,
this paper proposes a lightweight adversarial detection framework based on the
large-scale pre-trained vision-language model CLIP. Departing from conventional
adversarial feature characterization paradigms, we innovatively adopt an
anomaly detection perspective. By jointly fine-tuning CLIP's dual visual-text
encoders with trainable adapter networks and learnable prompts, we construct a
compact representation space tailored for natural images. Notably, our
detection architecture achieves substantial improvements in generalization
capability across both known and unknown attack patterns compared to
traditional methods, while significantly reducing training overhead. This study
provides a novel technical pathway for establishing a parameter-efficient and
attack-agnostic defense paradigm, markedly enhancing the robustness of vision
systems against evolving adversarial threats.


http://arxiv.org/abs/2504.00858
Whispering Under the Eaves: Protecting User Privacy Against Commercial and LLM-powered Automatic Speech Recognition Systems. (99%)
Weifei Jin; Yuxin Cao; Junjie Su; Derui Wang; Yedi Zhang; Minhui Xue; Jie Hao; Jin Song Dong; Yixian Yang
  The widespread application of automatic speech recognition (ASR) supports
large-scale voice surveillance, raising concerns about privacy among users. In
this paper, we concentrate on using adversarial examples to mitigate
unauthorized disclosure of speech privacy thwarted by potential eavesdroppers
in speech communications. While audio adversarial examples have demonstrated
the capability to mislead ASR models or evade ASR surveillance, they are
typically constructed through time-intensive offline optimization, restricting
their practicality in real-time voice communication. Recent work overcame this
limitation by generating universal adversarial perturbations (UAPs) and
enhancing their transferability for black-box scenarios. However, they
introduced excessive noise that significantly degrades audio quality and
affects human perception, thereby limiting their effectiveness in practical
scenarios. To address this limitation and protect live users' speech against
ASR systems, we propose a novel framework, AudioShield. Central to this
framework is the concept of Transferable Universal Adversarial Perturbations in
the Latent Space (LS-TUAP). By transferring the perturbations to the latent
space, the audio quality is preserved to a large extent. Additionally, we
propose target feature adaptation to enhance the transferability of UAPs by
embedding target text features into the perturbations. Comprehensive evaluation
on four commercial ASR APIs (Google, Amazon, iFlytek, and Alibaba), three voice
assistants, two LLM-powered ASR and one NN-based ASR demonstrates the
protection superiority of AudioShield over existing competitors, and both
objective and subjective evaluations indicate that AudioShield significantly
improves the audio quality. Moreover, AudioShield also shows high effectiveness
in real-time end-to-end scenarios, and demonstrates strong resilience against
adaptive countermeasures.


http://arxiv.org/abs/2504.01228
TenAd: A Tensor-based Low-rank Black Box Adversarial Attack for Video Classification. (99%)
Kimia haghjooei; Mansoor Rezghi
  Deep learning models have achieved remarkable success in computer vision but
remain vulnerable to adversarial attacks, particularly in black-box settings
where model details are unknown. Existing adversarial attack methods(even those
works with key frames) often treat video data as simple vectors, ignoring their
inherent multi-dimensional structure, and require a large number of queries,
making them inefficient and detectable. In this paper, we propose
\textbf{TenAd}, a novel tensor-based low-rank adversarial attack that leverages
the multi-dimensional properties of video data by representing videos as
fourth-order tensors. By exploiting low-rank attack, our method significantly
reduces the search space and the number of queries needed to generate
adversarial examples in black-box settings. Experimental results on standard
video classification datasets demonstrate that \textbf{TenAd} effectively
generates imperceptible adversarial perturbations while achieving higher attack
success rates and query efficiency compared to state-of-the-art methods. Our
approach outperforms existing black-box adversarial attacks in terms of success
rate, query efficiency, and perturbation imperceptibility, highlighting the
potential of tensor-based methods for adversarial attacks on video models.


http://arxiv.org/abs/2504.01308
Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks. (78%)
Jiawei Wang; Yushen Zuo; Yuanjun Chai; Zhendong Liu; Yicheng Fu; Yichun Feng; Kin-Man Lam
  Vision-Language Models (VLMs) extend the capabilities of Large Language
Models (LLMs) by incorporating visual information, yet they remain vulnerable
to jailbreak attacks, especially when processing noisy or corrupted images.
Although existing VLMs adopt security measures during training to mitigate such
attacks, vulnerabilities associated with noise-augmented visual inputs are
overlooked. In this work, we identify that missing noise-augmented training
causes critical security gaps: many VLMs are susceptible to even simple
perturbations such as Gaussian noise. To address this challenge, we propose
Robust-VLGuard, a multimodal safety dataset with aligned / misaligned
image-text pairs, combined with noise-augmented fine-tuning that reduces attack
success rates while preserving functionality of VLM. For stronger
optimization-based visual perturbation attacks, we propose DiffPure-VLM,
leveraging diffusion models to convert adversarial perturbations into
Gaussian-like noise, which can be defended by VLMs with noise-augmented safety
fine-tuning. Experimental results demonstrate that the distribution-shifting
property of diffusion model aligns well with our fine-tuned VLMs, significantly
mitigating adversarial perturbations across varying intensities. The dataset
and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.


http://arxiv.org/abs/2504.00721
Alleviating Performance Disparity in Adversarial Spatiotemporal Graph Learning Under Zero-Inflated Distribution. (68%)
Songran Bai; Yuheng Ji; Yue Liu; Xingwei Zhang; Xiaolong Zheng; Daniel Dajun Zeng
  Spatiotemporal Graph Learning (SGL) under Zero-Inflated Distribution (ZID) is
crucial for urban risk management tasks, including crime prediction and traffic
accident profiling. However, SGL models are vulnerable to adversarial attacks,
compromising their practical utility. While adversarial training (AT) has been
widely used to bolster model robustness, our study finds that traditional AT
exacerbates performance disparities between majority and minority classes under
ZID, potentially leading to irreparable losses due to underreporting critical
risk events. In this paper, we first demonstrate the smaller top-k gradients
and lower separability of minority class are key factors contributing to this
disparity. To address these issues, we propose MinGRE, a framework for Minority
Class Gradients and Representations Enhancement. MinGRE employs a
multi-dimensional attention mechanism to reweight spatiotemporal gradients,
minimizing the gradient distribution discrepancies across classes.
Additionally, we introduce an uncertainty-guided contrastive loss to improve
the inter-class separability and intra-class compactness of minority
representations with higher uncertainty. Extensive experiments demonstrate that
the MinGRE framework not only significantly reduces the performance disparity
across classes but also achieves enhanced robustness compared to existing
baselines. These findings underscore the potential of our method in fostering
the development of more equitable and robust models.


http://arxiv.org/abs/2504.00638
Impact of Data Duplication on Deep Neural Network-Based Image Classifiers: Robust vs. Standard Models. (47%)
Alireza Aghabagherloo; Aydin Abadi; Sumanta Sarkar; Vishnu Asutosh Dasu; Bart Preneel
  The accuracy and robustness of machine learning models against adversarial
attacks are significantly influenced by factors such as training data quality,
model architecture, the training process, and the deployment environment. In
recent years, duplicated data in training sets, especially in language models,
has attracted considerable attention. It has been shown that deduplication
enhances both training performance and model accuracy in language models. While
the importance of data quality in training image classifier Deep Neural
Networks (DNNs) is widely recognized, the impact of duplicated images in the
training set on model generalization and performance has received little
attention.
  In this paper, we address this gap and provide a comprehensive study on the
effect of duplicates in image classification. Our analysis indicates that the
presence of duplicated images in the training set not only negatively affects
the efficiency of model training but also may result in lower accuracy of the
image classifier. This negative impact of duplication on accuracy is
particularly evident when duplicated data is non-uniform across classes or when
duplication, whether uniform or non-uniform, occurs in the training set of an
adversarially trained model. Even when duplicated samples are selected in a
uniform way, increasing the amount of duplication does not lead to a
significant improvement in accuracy.


http://arxiv.org/abs/2504.01094
Multilingual and Multi-Accent Jailbreaking of Audio LLMs. (38%)
Jaechul Roh; Virat Shejwalkar; Amir Houmansadr
  Large Audio Language Models (LALMs) have significantly advanced audio
understanding but introduce critical security risks, particularly through audio
jailbreaks. While prior work has focused on English-centric attacks, we expose
a far more severe vulnerability: adversarial multilingual and multi-accent
audio jailbreaks, where linguistic and acoustic variations dramatically amplify
attack success. In this paper, we introduce Multi-AudioJail, the first
systematic framework to exploit these vulnerabilities through (1) a novel
dataset of adversarially perturbed multilingual/multi-accent audio jailbreaking
prompts, and (2) a hierarchical evaluation pipeline revealing that how acoustic
perturbations (e.g., reverberation, echo, and whisper effects) interacts with
cross-lingual phonetics to cause jailbreak success rates (JSRs) to surge by up
to +57.25 percentage points (e.g., reverberated Kenyan-accented attack on
MERaLiON). Crucially, our work further reveals that multimodal LLMs are
inherently more vulnerable than unimodal systems: attackers need only exploit
the weakest link (e.g., non-English audio inputs) to compromise the entire
model, which we empirically show by multilingual audio-only attacks achieving
3.1x higher success rates than text-only attacks. We plan to release our
dataset to spur research into cross-modal defenses, urging the community to
address this expanding attack surface in multimodality as LALMs evolve.


http://arxiv.org/abs/2504.00988
Safety and Security Risk Mitigation in Satellite Missions via Attack-Fault-Defense Trees. (5%)
Reza Soltani; Pablo Diale; Milan Lopuhaä-Zwakenberg; Mariëlle Stoelinga
  Cyber-physical systems, such as self-driving cars or digitized electrical
grids, often involve complex interactions between security, safety, and
defense. Proper risk management strategies must account for these three
critical domains and their interaction because the failure to address one
domain can exacerbate risks in the others, leading to cascading effects that
compromise the overall system resilience. This work presents a case study from
Ascentio Technologies, a mission-critical system company in Argentina
specializing in aerospace, where the interplay between safety, security, and
defenses is critical for ensuring the resilience and reliability of their
systems. The main focus will be on the Ground Segment for the satellite project
currently developed by the company. Analyzing safety, security, and defense
mechanisms together in the Ground Segment of a satellite project is crucial
because these domains are deeply interconnected--for instance, a security
breach could disable critical safety functions, or a safety failure could
create opportunities for attackers to exploit vulnerabilities, amplifying the
risks to the entire system. This paper showcases the application of the
Attack-Fault-Defense Tree (AFDT) framework, which integrates attack trees,
fault trees, and defense mechanisms into a unified model. AFDT provides an
intuitive visual language that facilitates interdisciplinary collaboration,
enabling experts from various fields to better assess system vulnerabilities
and defenses. By applying AFDT to the Ground Segment of the satellite project,
we demonstrate how qualitative analyses can be performed to identify weaknesses
and enhance the overall system's security and safety. This case highlights the
importance of jointly analyzing attacks, faults, and defenses to improve
resilience in complex cyber-physical environments.


http://arxiv.org/abs/2504.00454
FA^{3}-CLIP: Frequency-Aware Cues Fusion and Attack-Agnostic Prompt Learning for Unified Face Attack Detection. (2%)
Yongze Li; Ning Li; Ajian Liu; Hui Ma; Liying Yang; Xihong Chen; Zhiyao Liang; Yanyan Liang; Jun Wan; Zhen Lei
  Facial recognition systems are vulnerable to physical (e.g., printed photos)
and digital (e.g., DeepFake) face attacks. Existing methods struggle to
simultaneously detect physical and digital attacks due to: 1) significant
intra-class variations between these attack types, and 2) the inadequacy of
spatial information alone to comprehensively capture live and fake cues. To
address these issues, we propose a unified attack detection model termed
Frequency-Aware and Attack-Agnostic CLIP (FA\textsuperscript{3}-CLIP), which
introduces attack-agnostic prompt learning to express generic live and fake
cues derived from the fusion of spatial and frequency features, enabling
unified detection of live faces and all categories of attacks. Specifically,
the attack-agnostic prompt module generates generic live and fake prompts
within the language branch to extract corresponding generic representations
from both live and fake faces, guiding the model to learn a unified feature
space for unified attack detection. Meanwhile, the module adaptively generates
the live/fake conditional bias from the original spatial and frequency
information to optimize the generic prompts accordingly, reducing the impact of
intra-class variations. We further propose a dual-stream cues fusion framework
in the vision branch, which leverages frequency information to complement
subtle cues that are difficult to capture in the spatial domain. In addition, a
frequency compression block is utilized in the frequency stream, which reduces
redundancy in frequency features while preserving the diversity of crucial
cues. We also establish new challenging protocols to facilitate unified face
attack detection effectiveness. Experimental results demonstrate that the
proposed method significantly improves performance in detecting physical and
digital face attacks, achieving state-of-the-art results.


http://arxiv.org/abs/2504.00446
Exposing the Ghost in the Transformer: Abnormal Detection for Large Language Models via Hidden State Forensics. (1%)
Shide Zhou; Kailong Wang; Ling Shi; Haoyu Wang
  The widespread adoption of Large Language Models (LLMs) in critical
applications has introduced severe reliability and security risks, as LLMs
remain vulnerable to notorious threats such as hallucinations, jailbreak
attacks, and backdoor exploits. These vulnerabilities have been weaponized by
malicious actors, leading to unauthorized access, widespread misinformation,
and compromised LLM-embedded system integrity. In this work, we introduce a
novel approach to detecting abnormal behaviors in LLMs via hidden state
forensics. By systematically inspecting layer-specific activation patterns, we
develop a unified framework that can efficiently identify a range of security
threats in real-time without imposing prohibitive computational costs.
Extensive experiments indicate detection accuracies exceeding 95% and
consistently robust performance across multiple models in most scenarios, while
preserving the ability to detect novel attacks effectively. Furthermore, the
computational overhead remains minimal, with merely fractions of a second. The
significance of this work lies in proposing a promising strategy to reinforce
the security of LLM-integrated systems, paving the way for safer and more
reliable deployment in high-stakes domains. By enabling real-time detection
that can also support the mitigation of abnormal behaviors, it represents a
meaningful step toward ensuring the trustworthiness of AI systems amid rising
security challenges.


http://arxiv.org/abs/2504.01278
Strategize Globally, Adapt Locally: A Multi-Turn Red Teaming Agent with Dual-Level Learning. (1%)
Si Chen; Xiao Yu; Ninareh Mehrabi; Rahul Gupta; Zhou Yu; Ruoxi Jia
  The exploitation of large language models (LLMs) for malicious purposes poses
significant security risks as these models become more powerful and widespread.
While most existing red-teaming frameworks focus on single-turn attacks,
real-world adversaries typically operate in multi-turn scenarios, iteratively
probing for vulnerabilities and adapting their prompts based on threat model
responses. In this paper, we propose \AlgName, a novel multi-turn red-teaming
agent that emulates sophisticated human attackers through complementary
learning dimensions: global tactic-wise learning that accumulates knowledge
over time and generalizes to new attack goals, and local prompt-wise learning
that refines implementations for specific goals when initial attempts fail.
Unlike previous multi-turn approaches that rely on fixed strategy sets,
\AlgName enables the agent to identify new jailbreak tactics, develop a
goal-based tactic selection framework, and refine prompt formulations for
selected tactics. Empirical evaluations on JailbreakBench demonstrate our
framework's superior performance, achieving over 90\% attack success rates
against GPT-3.5-Turbo and Llama-3.1-70B within 5 conversation turns,
outperforming state-of-the-art baselines. These results highlight the
effectiveness of dynamic learning in identifying and exploiting model
vulnerabilities in realistic multi-turn scenarios.


http://arxiv.org/abs/2504.00218
$\textit{Agents Under Siege}$: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks. (96%)
Rana Muhammad Shahroz Khan; Zhen Tan; Sukwon Yun; Charles Flemming; Tianlong Chen
  Most discussions about Large Language Model (LLM) safety have focused on
single-agent settings but multi-agent LLM systems now create novel adversarial
risks because their behavior depends on communication between agents and
decentralized reasoning. In this work, we innovatively focus on attacking
pragmatic systems that have constrains such as limited token bandwidth, latency
between message delivery, and defense mechanisms. We design a
$\textit{permutation-invariant adversarial attack}$ that optimizes prompt
distribution across latency and bandwidth-constraint network topologies to
bypass distributed safety mechanisms within the system. Formulating the attack
path as a problem of $\textit{maximum-flow minimum-cost}$, coupled with the
novel $\textit{Permutation-Invariant Evasion Loss (PIEL)}$, we leverage
graph-based optimization to maximize attack success rate while minimizing
detection risk. Evaluating across models including $\texttt{Llama}$,
$\texttt{Mistral}$, $\texttt{Gemma}$, $\texttt{DeepSeek}$ and other variants on
various datasets like $\texttt{JailBreakBench}$ and
$\texttt{AdversarialBench}$, our method outperforms conventional attacks by up
to $7\times$, exposing critical vulnerabilities in multi-agent systems.
Moreover, we demonstrate that existing defenses, including variants of
$\texttt{Llama-Guard}$ and $\texttt{PromptGuard}$, fail to prohibit our attack,
emphasizing the urgent need for multi-agent specific safety mechanisms.


http://arxiv.org/abs/2504.02859
AI-Enhanced Resilience in Power Systems: Adversarial Deep Learning for Robust Short-Term Voltage Stability Assessment under Cyber-Attacks. (96%)
Yang Li; Shitu Zhang; Yuanzheng Li
  In the era of Industry 4.0, ensuring the resilience of cyber-physical systems
against sophisticated cyber threats is increasingly critical. This study
proposes a pioneering AI-based control framework that enhances short-term
voltage stability assessments (STVSA) in power systems under complex composite
cyber-attacks. First, by incorporating white-box and black-box adversarial
attacks with Denial-of-Service (DoS) perturbations during training, composite
adversarial attacks are implemented. Second, the application of Spectral
Normalized Conditional Wasserstein Generative Adversarial Network with Gradient
Penalty (SNCWGAN-GP) and Fast Gradient Sign Method (FGSM) strengthens the
model's resistance to adversarial disturbances, improving data quality and
training stability. Third, an assessment model based on Long Short-Term Memory
(LSTM)-enhanced Graph Attention Network (L-GAT) is developed to capture dynamic
relationships between the post-fault dynamic trajectories and electrical grid
topology. Experimental results on the IEEE 39-bus test system demonstrate the
efficacy and superiority of the proposed method in composite cyber-attack
scenarios. This contribution is pivotal to advancing AI-based resilient control
strategies for nonlinear dynamical systems, marking a substantial enhancement
in the security of cyber-physical systems.


http://arxiv.org/abs/2504.03735
Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots. (87%)
Erfan Shayegani; G M Shahariar; Sara Abdali; Lei Yu; Nael Abu-Ghazaleh; Yue Dong
  Multimodal Language Models (MMLMs) typically undergo post-training alignment
to prevent harmful content generation. However, these alignment stages focus
primarily on the assistant role, leaving the user role unaligned, and stick to
a fixed input prompt structure of special tokens, leaving the model vulnerable
when inputs deviate from these expectations. We introduce Role-Modality Attacks
(RMA), a novel class of adversarial attacks that exploit role confusion between
the user and assistant and alter the position of the image token to elicit
harmful outputs. Unlike existing attacks that modify query content, RMAs
manipulate the input structure without altering the query itself. We
systematically evaluate these attacks across multiple Vision Language Models
(VLMs) on eight distinct settings, showing that they can be composed to create
stronger adversarial prompts, as also evidenced by their increased projection
in the negative refusal direction in the residual stream, a property observed
in prior successful attacks. Finally, for mitigation, we propose an adversarial
training approach that makes the model robust against input prompt
perturbations. By training the model on a range of harmful and benign prompts
all perturbed with different RMA settings, it loses its sensitivity to Role
Confusion and Modality Manipulation attacks and is trained to only pay
attention to the content of the query in the input prompt structure,
effectively reducing Attack Success Rate (ASR) while preserving the model's
general utility.


http://arxiv.org/abs/2503.23708
Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios. (69%)
Jingzheng Li; Xianglong Liu; Shikui Wei; Zhijun Chen; Bing Li; Qing Guo; Xianqi Yang; Yanjun Pu; Jiakai Wang
  Autonomous driving has made significant progress in both academia and
industry, including performance improvements in perception task and the
development of end-to-end autonomous driving systems. However, the safety and
robustness assessment of autonomous driving has not received sufficient
attention. Current evaluations of autonomous driving are typically conducted in
natural driving scenarios. However, many accidents often occur in edge cases,
also known as safety-critical scenarios. These safety-critical scenarios are
difficult to collect, and there is currently no clear definition of what
constitutes a safety-critical scenario. In this work, we explore the safety and
robustness of autonomous driving in safety-critical scenarios. First, we
provide a definition of safety-critical scenarios, including static traffic
scenarios such as adversarial attack scenarios and natural distribution shifts,
as well as dynamic traffic scenarios such as accident scenarios. Then, we
develop an autonomous driving safety testing platform to comprehensively
evaluate autonomous driving systems, encompassing not only the assessment of
perception modules but also system-level evaluations. Our work systematically
constructs a safety verification process for autonomous driving, providing
technical support for the industry to establish standardized test framework and
reduce risks in real-world road deployment.


http://arxiv.org/abs/2503.23866
A Channel-Triggered Backdoor Attack on Wireless Semantic Image Reconstruction. (68%)
Jialin Wan; Nan Cheng; Jinglong Shen
  Despite the transformative impact of deep learning (DL) on wireless
communication systems through data-driven end-to-end (E2E) learning, the
security vulnerabilities of these systems have been largely overlooked. Unlike
the extensively studied image domain, limited research has explored the threat
of backdoor attacks on the reconstruction of symbols in semantic communication
(SemCom) systems. Previous work has investigated such backdoor attacks at the
input level, but these approaches are infeasible in applications with strict
input control. In this paper, we propose a novel attack paradigm, termed
Channel-Triggered Backdoor Attack (CT-BA), where the backdoor trigger is a
specific wireless channel. This attack leverages fundamental physical layer
characteristics, making it more covert and potentially more threatening
compared to previous input-level attacks. Specifically, we utilize channel gain
with different fading distributions or channel noise with different power
spectral densities as potential triggers. This approach establishes
unprecedented attack flexibility as the adversary can select backdoor triggers
from both fading characteristics and noise variations in diverse channel
environments. Moreover, during the testing phase, CT-BA enables automatic
trigger activation through natural channel variations without requiring active
adversary participation. We evaluate the robustness of CT-BA on a ViT-based
Joint Source-Channel Coding (JSCC) model across three datasets: MNIST,
CIFAR-10, and ImageNet. Furthermore, we apply CT-BA to three typical E2E SemCom
systems: BDJSCC, ADJSCC, and JSCCOFDM. Experimental results demonstrate that
our attack achieves near-perfect attack success rate (ASR) while maintaining
effective stealth. Finally, we discuss potential defense mechanisms against
such attacks.


http://arxiv.org/abs/2503.23718
Detecting Functional Bugs in Smart Contracts through LLM-Powered and Bug-Oriented Composite Analysis. (1%)
Binbin Zhao; Xingshuang Lin; Yuan Tian; Saman Zonouz; Na Ruan; Jiliang Li; Raheem Beyah; Shouling Ji
  Smart contracts are fundamental pillars of the blockchain, playing a crucial
role in facilitating various business transactions. However, these smart
contracts are vulnerable to exploitable bugs that can lead to substantial
monetary losses. A recent study reveals that over 80% of these exploitable
bugs, which are primarily functional bugs, can evade the detection of current
tools. The primary issue is the significant gap between understanding the
high-level logic of the business model and checking the low-level
implementations in smart contracts. Furthermore, identifying deeply rooted
functional bugs in smart contracts requires the automated generation of
effective detection oracles based on various bug features. To address these
challenges, we design and implement PROMFUZZ, an automated and scalable system
to detect functional bugs, in smart contracts. In PROMFUZZ, we first propose a
novel Large Language Model (LLM)-driven analysis framework, which leverages a
dual-agent prompt engineering strategy to pinpoint potentially vulnerable
functions for further scrutiny. We then implement a dual-stage coupling
approach, which focuses on generating invariant checkers that leverage logic
information extracted from potentially vulnerable functions. Finally, we design
a bug-oriented fuzzing engine, which maps the logical information from the
high-level business model to the low-level smart contract implementations, and
performs the bug-oriented fuzzing on targeted functions. We compare PROMFUZZ
with multiple state-of-the-art methods. The results show that PROMFUZZ achieves
86.96% recall and 93.02% F1-score in detecting functional bugs, marking at
least a 50% improvement in both metrics over state-of-the-art methods.
Moreover, we perform an in-depth analysis on real-world DeFi projects and
detect 30 zero-day bugs. Up to now, 24 zero-day bugs have been assigned CVE
IDs.


http://arxiv.org/abs/2504.00038
Revisiting the Relationship between Adversarial and Clean Training: Why Clean Training Can Make Adversarial Training Better. (50%)
MingWei Zhou; Xiaobing Pei
  Adversarial training (AT) is an effective technique for enhancing adversarial
robustness, but it usually comes at the cost of a decline in generalization
ability. Recent studies have attempted to use clean training to assist
adversarial training, yet there are contradictions among the conclusions. We
comprehensively summarize the representative strategies and, with a focus on
the multi - view hypothesis, provide a unified explanation for the
contradictory phenomena among different studies. In addition, we conduct an in
- depth analysis of the knowledge combinations transferred from clean - trained
models to adversarially - trained models in previous studies, and find that
they can be divided into two categories: reducing the learning difficulty and
providing correct guidance. Based on this finding, we propose a new idea of
leveraging clean training to further improve the performance of advanced AT
methods.We reveal that the problem of generalization degradation faced by AT
partly stems from the difficulty of adversarial training in learning certain
sample features, and this problem can be alleviated by making full use of clean
training.


http://arxiv.org/abs/2503.23511
Buffer is All You Need: Defending Federated Learning against Backdoor Attacks under Non-iids via Buffering. (15%)
Xingyu Lyu; Ning Wang; Yang Xiao; Shixiong Li; Tao Li; Danjue Chen; Yimin Chen
  Federated Learning (FL) is a popular paradigm enabling clients to jointly
train a global model without sharing raw data. However, FL is known to be
vulnerable towards backdoor attacks due to its distributed nature. As
participants, attackers can upload model updates that effectively compromise
FL. What's worse, existing defenses are mostly designed under
independent-and-identically-distributed (iid) settings, hence neglecting the
fundamental non-iid characteristic of FL. Here we propose FLBuff for tackling
backdoor attacks even under non-iids. The main challenge for such defenses is
that non-iids bring benign and malicious updates closer, hence harder to
separate. FLBuff is inspired by our insight that non-iids can be modeled as
omni-directional expansion in representation space while backdoor attacks as
uni-directional. This leads to the key design of FLBuff, i.e., a
supervised-contrastive-learning model extracting penultimate-layer
representations to create a large in-between buffer layer. Comprehensive
evaluations demonstrate that FLBuff consistently outperforms state-of-the-art
defenses.


http://arxiv.org/abs/2503.23536
A Survey on Unlearnable Data. (5%)
Jiahao Li; Yiqiang Chen; Yunbing Xing; Yang Gu; Xiangyuan Lan
  Unlearnable data (ULD) has emerged as an innovative defense technique to
prevent machine learning models from learning meaningful patterns from specific
data, thus protecting data privacy and security. By introducing perturbations
to the training data, ULD degrades model performance, making it difficult for
unauthorized models to extract useful representations. Despite the growing
significance of ULD, existing surveys predominantly focus on related fields,
such as adversarial attacks and machine unlearning, with little attention given
to ULD as an independent area of study. This survey fills that gap by offering
a comprehensive review of ULD, examining unlearnable data generation methods,
public benchmarks, evaluation metrics, theoretical foundations and practical
applications. We compare and contrast different ULD approaches, analyzing their
strengths, limitations, and trade-offs related to unlearnability,
imperceptibility, efficiency and robustness. Moreover, we discuss key
challenges, such as balancing perturbation imperceptibility with model
degradation and the computational complexity of ULD generation. Finally, we
highlight promising future research directions to advance the effectiveness and
applicability of ULD, underscoring its potential to become a crucial tool in
the evolving landscape of data protection in machine learning.


http://arxiv.org/abs/2503.23288
Two Heads Are Better than One: Model-Weight and Latent-Space Analysis for Federated Learning on Non-iid Data against Poisoning Attacks. (80%)
Xingyu Lyu; Ning Wang; Yang Xiao; Shixiong Li; Tao Li; Danjue Chen; Yimin Chen
  Federated Learning is a popular paradigm that enables remote clients to
jointly train a global model without sharing their raw data. However, FL has
been shown to be vulnerable towards model poisoning attacks due to its
distributed nature. Particularly, attackers acting as participants can upload
arbitrary model updates that effectively compromise the global model of FL.
While extensive research has been focusing on fighting against these attacks,
we find that most of them assume data at remote clients are under iid while in
practice they are inevitably non-iid. Our benchmark evaluations reveal that
existing defenses generally fail to live up to their reputation when applied to
various non-iid scenarios. In this paper, we propose a novel approach,
GeminiGuard, that aims to address such a significant gap. We design GeminiGuard
to be lightweight, versatile, and unsupervised so that it aligns well with the
practical requirements of deploying such defenses. The key challenge from
non-iids is that they make benign model updates look more similar to malicious
ones. GeminiGuard is mainly built on two fundamental observations: (1) existing
defenses based on either model-weight analysis or latent-space analysis face
limitations in covering different MPAs and non-iid scenarios, and (2)
model-weight and latent-space analysis are sufficiently different yet
potentially complementary methods as MPA defenses. We hence incorporate a novel
model-weight analysis component as well as a custom latent-space analysis
component in GeminiGuard, aiming to further enhance its defense performance. We
conduct extensive experiments to evaluate our defense across various settings,
demonstrating its effectiveness in countering multiple types of untargeted and
targeted MPAs, including adaptive ones. Our comprehensive evaluations show that
GeminiGuard consistently outperforms SOTA defenses under various settings.


http://arxiv.org/abs/2503.22998
AuditVotes: A Framework Towards More Deployable Certified Robustness for Graph Neural Networks. (76%)
Yuni Lai; Yulin Zhu; Yixuan Sun; Yulun Wu; Bin Xiao; Gaolei Li; Jianhua Li; Kai Zhou
  Despite advancements in Graph Neural Networks (GNNs), adaptive attacks
continue to challenge their robustness. Certified robustness based on
randomized smoothing has emerged as a promising solution, offering provable
guarantees that a model's predictions remain stable under adversarial
perturbations within a specified range. However, existing methods face a
critical trade-off between accuracy and robustness, as achieving stronger
robustness requires introducing greater noise into the input graph. This
excessive randomization degrades data quality and disrupts prediction
consistency, limiting the practical deployment of certifiably robust GNNs in
real-world scenarios where both accuracy and robustness are essential. To
address this challenge, we propose \textbf{AuditVotes}, the first framework to
achieve both high clean accuracy and certifiably robust accuracy for GNNs. It
integrates randomized smoothing with two key components,
\underline{au}gmentation and con\underline{dit}ional smoothing, aiming to
improve data quality and prediction consistency. The augmentation, acting as a
pre-processing step, de-noises the randomized graph, significantly improving
data quality and clean accuracy. The conditional smoothing, serving as a
post-processing step, employs a filtering function to selectively count votes,
thereby filtering low-quality predictions and improving voting consistency.
Extensive experimental results demonstrate that AuditVotes significantly
enhances clean accuracy, certified robustness, and empirical robustness while
maintaining high computational efficiency. Notably, compared to baseline
randomized smoothing, AuditVotes improves clean accuracy by $437.1\%$ and
certified accuracy by $409.3\%$ when the attacker can arbitrarily insert $20$
edges on the Cora-ML datasets, representing a substantial step toward deploying
certifiably robust GNNs in real-world applications.


http://arxiv.org/abs/2503.23103
Towards Secure Semantic Communications in the Presence of Intelligent Eavesdroppers. (26%)
Shunpu Tang; Yuhao Chen; Qianqian Yang; Ruichen Zhang; Dusit Niyato; Zhiguo Shi
  Semantic communication has emerged as a promising paradigm for enhancing
communication efficiency in sixth-generation (6G) networks. However, the
broadcast nature of wireless channels makes SemCom systems vulnerable to
eavesdropping, which poses a serious threat to data privacy. Therefore, we
investigate secure SemCom systems that preserve data privacy in the presence of
eavesdroppers. Specifically, we first explore a scenario where eavesdroppers
are intelligent and can exploit semantic information to reconstruct the
transmitted data based on advanced artificial intelligence (AI) techniques. To
counter this, we introduce novel eavesdropping attack strategies that utilize
model inversion attacks and generative AI (GenAI) models. These strategies
effectively reconstruct transmitted private data processed by the semantic
encoder, operating in both glass-box and closed-box settings. Existing defense
mechanisms against eavesdropping often cause significant distortions in the
data reconstructed by eavesdroppers, potentially arousing their suspicion. To
address this, we propose a semantic covert communication approach that
leverages an invertible neural network (INN)-based signal steganography module.
This module covertly embeds the channel input signal of a private sample into
that of a non-sensitive host sample, thereby misleading eavesdroppers. Without
access to this module, eavesdroppers can only extract host-related information
and remain unaware of the hidden private content. We conduct extensive
simulations under various channel conditions in image transmission tasks.
Numerical results show that while conventional eavesdropping strategies achieve
a success rate of over 80\% in reconstructing private information, the proposed
semantic covert communication effectively reduces the eavesdropping success
rate to 0.


http://arxiv.org/abs/2503.22205
Data-Free Universal Attack by Exploiting the Intrinsic Vulnerability of Deep Models. (99%)
YangTian Yan; Jinyu Tian
  Deep neural networks (DNNs) are susceptible to Universal Adversarial
Perturbations (UAPs), which are instance agnostic perturbations that can
deceive a target model across a wide range of samples. Unlike instance-specific
adversarial examples, UAPs present a greater challenge as they must generalize
across different samples and models. Generating UAPs typically requires access
to numerous examples, which is a strong assumption in real-world tasks. In this
paper, we propose a novel data-free method called Intrinsic UAP (IntriUAP), by
exploiting the intrinsic vulnerabilities of deep models. We analyze a series of
popular deep models composed of linear and nonlinear layers with a Lipschitz
constant of 1, revealing that the vulnerability of these models is
predominantly influenced by their linear components. Based on this observation,
we leverage the ill-conditioned nature of the linear components by aligning the
UAP with the right singular vectors corresponding to the maximum singular value
of each linear layer. Remarkably, our method achieves highly competitive
performance in attacking popular image classification deep models without using
any image samples. We also evaluate the black-box attack performance of our
method, showing that it matches the state-of-the-art baseline for data-free
methods on models that conform to our theoretical framework. Beyond the
data-free assumption, IntriUAP also operates under a weaker assumption, where
the adversary only can access a few of the victim model's layers. Experiments
demonstrate that the attack success rate decreases by only 4% when the
adversary has access to just 50% of the linear layers in the victim model.


http://arxiv.org/abs/2503.22653
Tropical Bisectors and Carlini-Wagner Attacks. (33%)
Gillian Grindstaff; Julia Lindberg; Daniela Schkoda; Miruna-Stefana Sorea; Ruriko Yoshida
  Pasque et al. showed that using a tropical symmetric metric as an activation
function in the last layer can improve the robustness of convolutional neural
networks (CNNs) against state-of-the-art attacks, including the Carlini-Wagner
attack. This improvement occurs when the attacks are not specifically adapted
to the non-differentiability of the tropical layer. Moreover, they showed that
the decision boundary of a tropical CNN is defined by tropical bisectors. In
this paper, we explore the combinatorics of tropical bisectors and analyze how
the tropical embedding layer enhances robustness against Carlini-Wagner
attacks. We prove an upper bound on the number of linear segments the decision
boundary of a tropical CNN can have. We then propose a refined version of the
Carlini-Wagner attack, specifically tailored for the tropical architecture.
Computational experiments with MNIST and LeNet5 showcase our attacks improved
success rate.


http://arxiv.org/abs/2504.03714
Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models. (13%)
Runpeng Dai; Run Yang; Fan Zhou; Hongtu Zhu
  Large Language Models (LLMs) and Vision-Language Models (VLMs) have become
essential to general artificial intelligence, exhibiting remarkable
capabilities in task understanding and problem-solving. However, the real-world
reliability of these models critically depends on their stability, which
remains an underexplored area. Despite their widespread use, rigorous studies
examining the stability of LLMs under various perturbations are still lacking.
In this paper, we address this gap by proposing a novel stability measure for
LLMs, inspired by statistical methods rooted in information geometry. Our
measure possesses desirable invariance properties, making it well-suited for
analyzing model sensitivity to both parameter and input perturbations. To
assess the effectiveness of our approach, we conduct extensive experiments on
models ranging in size from 1.5B to 13B parameters. Our results demonstrate the
utility of our measure in identifying salient parameters and detecting
vulnerable regions in input images or critical dimensions in token embeddings.
Furthermore, leveraging our stability framework, we enhance model robustness
during model merging, leading to improved performance.


http://arxiv.org/abs/2503.22163
T-CIL: Temperature Scaling using Adversarial Perturbation for Calibration in Class-Incremental Learning. (11%)
Seong-Hyeon Hwang; Minsu Kim; Steven Euijong Whang
  We study model confidence calibration in class-incremental learning, where
models learn from sequential tasks with different class sets. While existing
works primarily focus on accuracy, maintaining calibrated confidence has been
largely overlooked. Unfortunately, most post-hoc calibration techniques are not
designed to work with the limited memories of old-task data typical in
class-incremental learning, as retaining a sufficient validation set would be
impractical. Thus, we propose T-CIL, a novel temperature scaling approach for
class-incremental learning without a validation set for old tasks, that
leverages adversarially perturbed exemplars from memory. Directly using
exemplars is inadequate for temperature optimization, since they are already
used for training. The key idea of T-CIL is to perturb exemplars more strongly
for old tasks than for the new task by adjusting the perturbation direction
based on feature distance, with the single magnitude determined using the
new-task validation set. This strategy makes the perturbation magnitude
computed from the new task also applicable to old tasks, leveraging the
tendency that the accuracy of old tasks is lower than that of the new task. We
empirically show that T-CIL significantly outperforms various baselines in
terms of calibration on real datasets and can be integrated with existing
class-incremental learning techniques with minimal impact on accuracy.


http://arxiv.org/abs/2503.22330
Imperceptible but Forgeable: Practical Invisible Watermark Forgery via Diffusion Models. (1%)
Ziping Dong; Chao Shuai; Zhongjie Ba; Peng Cheng; Zhan Qin; Qinglong Wang; Kui Ren
  Invisible watermarking is critical for content provenance and accountability
in Generative AI. Although commercial companies have increasingly committed to
using watermarks, the robustness of existing watermarking schemes against
forgery attacks is understudied. This paper proposes DiffForge, the first
watermark forgery framework capable of forging imperceptible watermarks under a
no-box setting. We estimate the watermark distribution using an unconditional
diffusion model and introduce shallow inversion to inject the watermark into a
non-watermarked image seamlessly. This approach facilitates watermark injection
while preserving image quality by adaptively selecting the depth of inversion
steps, leveraging our key insight that watermarks degrade with added noise
during the early diffusion phases. Comprehensive evaluations show that
DiffForge deceives open-source watermark detectors with a 96.38% success rate
and misleads a commercial watermark system with over 97% success rate,
achieving high confidence.1 This work reveals fundamental security limitations
in current watermarking paradigms.


http://arxiv.org/abs/2503.21164
Adversarial Wear and Tear: Exploiting Natural Damage for Generating Physical-World Adversarial Examples. (99%)
Samra Irshad; Seungkyu Lee; Nassir Navab; Hong Joo Lee; Seong Tae Kim
  The presence of adversarial examples in the physical world poses significant
challenges to the deployment of Deep Neural Networks in safety-critical
applications such as autonomous driving. Most existing methods for crafting
physical-world adversarial examples are ad-hoc, relying on temporary
modifications like shadows, laser beams, or stickers that are tailored to
specific scenarios. In this paper, we introduce a new class of physical-world
adversarial examples, AdvWT, which draws inspiration from the naturally
occurring phenomenon of `wear and tear', an inherent property of physical
objects. Unlike manually crafted perturbations, `wear and tear' emerges
organically over time due to environmental degradation, as seen in the gradual
deterioration of outdoor signboards. To achieve this, AdvWT follows a two-step
approach. First, a GAN-based, unsupervised image-to-image translation network
is employed to model these naturally occurring damages, particularly in the
context of outdoor signboards. The translation network encodes the
characteristics of damaged signs into a latent `damage style code'. In the
second step, we introduce adversarial perturbations into the style code,
strategically optimizing its transformation process. This manipulation subtly
alters the damage style representation, guiding the network to generate
adversarial images where the appearance of damages remains perceptually
realistic, while simultaneously ensuring their effectiveness in misleading
neural networks. Through comprehensive experiments on two traffic sign
datasets, we show that AdvWT effectively misleads DNNs in both digital and
physical domains. AdvWT achieves an effective attack success rate, greater
robustness, and a more natural appearance compared to existing physical-world
adversarial examples. Additionally, integrating AdvWT into training enhances a
model's generalizability to real-world damaged signs.


http://arxiv.org/abs/2503.21236
Clean Image May be Dangerous: Data Poisoning Attacks Against Deep Hashing. (76%)
Shuai Li; Jie Zhang; Yuang Qi; Kejiang Chen; Tianwei Zhang; Weiming Zhang; Nenghai Yu
  Large-scale image retrieval using deep hashing has become increasingly
popular due to the exponential growth of image data and the remarkable feature
extraction capabilities of deep neural networks (DNNs). However, deep hashing
methods are vulnerable to malicious attacks, including adversarial and backdoor
attacks. It is worth noting that these attacks typically involve altering the
query images, which is not a practical concern in real-world scenarios. In this
paper, we point out that even clean query images can be dangerous, inducing
malicious target retrieval results, like undesired or illegal images. To the
best of our knowledge, we are the first to study data \textbf{p}oisoning
\textbf{a}ttacks against \textbf{d}eep \textbf{hash}ing
\textbf{(\textit{PADHASH})}. Specifically, we first train a surrogate model to
simulate the behavior of the target deep hashing model. Then, a strict gradient
matching strategy is proposed to generate the poisoned images. Extensive
experiments on different models, datasets, hash methods, and hash code lengths
demonstrate the effectiveness and generality of our attack method.


http://arxiv.org/abs/2503.22765
Non-control-Data Attacks and Defenses: A review. (75%)
Lei Chong
  In recent years, non-control-data attacks have be come a research hotspot in
the field of network security, driven
  by the increasing number of defense methods against control-flow
  hijacking attacks. These attacks exploit memory vulnerabilities
  to modify non-control data within a program, thereby altering its
  behavior without compromising control-flow integrity. Research
  has shown that non-control-data attacks can be just as damaging
  as control-flow hijacking attacks and are even Turing complete,
  making them a serious security threat. However, despite being
  discovered long ago, the threat of non-control-data attacks has
  not been adequately addressed. In this review, we first classify
  non-control-data attacks into two categories based on their
  evolution: security-sensitive function attacks and data-oriented
  programming (DOP) attacks. Subsequently, based on the non control-data attack
model, we categorize existing defense methods
  into three main strategies: memory safety, data confidentiality,
  and data integrity protection. We then analyze recent defense
  techniques specifically designed for DOP attacks. Finally, we
  identify the key challenges hindering the widespread adoption
  of defenses against non-control-data attacks and explore future
  research directions in this field.


http://arxiv.org/abs/2503.21983
Learning to Lie: Reinforcement Learning Attacks Damage Human-AI Teams and Teams of LLMs. (62%)
Abed Kareem Musaffar; Anand Gokhale; Sirui Zeng; Rasta Tadayon; Xifeng Yan; Ambuj Singh; Francesco Bullo
  As artificial intelligence (AI) assistants become more widely adopted in
safety-critical domains, it becomes important to develop safeguards against
potential failures or adversarial attacks. A key prerequisite to developing
these safeguards is understanding the ability of these AI assistants to mislead
human teammates. We investigate this attack problem within the context of an
intellective strategy game where a team of three humans and one AI assistant
collaborate to answer a series of trivia questions. Unbeknownst to the humans,
the AI assistant is adversarial. Leveraging techniques from Model-Based
Reinforcement Learning (MBRL), the AI assistant learns a model of the humans'
trust evolution and uses that model to manipulate the group decision-making
process to harm the team. We evaluate two models -- one inspired by literature
and the other data-driven -- and find that both can effectively harm the human
team. Moreover, we find that in this setting our data-driven model is capable
of accurately predicting how human agents appraise their teammates given
limited information on prior interactions. Finally, we compare the performance
of state-of-the-art LLM models to human agents on our influence allocation task
to evaluate whether the LLMs allocate influence similarly to humans or if they
are more robust to our attack. These results enhance our understanding of
decision-making dynamics in small human-AI teams and lay the foundation for
defense strategies.


http://arxiv.org/abs/2503.21315
Tricking Retrievers with Influential Tokens: An Efficient Black-Box Corpus Poisoning Attack. (54%)
Cheng Wang; Yiwei Wang; Yujun Cai; Bryan Hooi
  Retrieval-augmented generation (RAG) systems enhance large language models by
incorporating external knowledge, addressing issues like outdated internal
knowledge and hallucination. However, their reliance on external knowledge
bases makes them vulnerable to corpus poisoning attacks, where adversarial
passages can be injected to manipulate retrieval results. Existing methods for
crafting such passages, such as random token replacement or training inversion
models, are often slow and computationally expensive, requiring either access
to retriever's gradients or large computational resources. To address these
limitations, we propose Dynamic Importance-Guided Genetic Algorithm (DIGA), an
efficient black-box method that leverages two key properties of retrievers:
insensitivity to token order and bias towards influential tokens. By focusing
on these characteristics, DIGA dynamically adjusts its genetic operations to
generate effective adversarial passages with significantly reduced time and
memory usage. Our experimental evaluation shows that DIGA achieves superior
efficiency and scalability compared to existing methods, while maintaining
comparable or better attack success rates across multiple datasets.


http://arxiv.org/abs/2503.22759
Data Poisoning in Deep Learning: A Survey. (2%)
Pinlong Zhao; Weiyao Zhu; Pengfei Jiao; Di Gao; Ou Wu
  Deep learning has become a cornerstone of modern artificial intelligence,
enabling transformative applications across a wide range of domains. As the
core element of deep learning, the quality and security of training data
critically influence model performance and reliability. However, during the
training process, deep learning models face the significant threat of data
poisoning, where attackers introduce maliciously manipulated training data to
degrade model accuracy or lead to anomalous behavior. While existing surveys
provide valuable insights into data poisoning, they generally adopt a broad
perspective, encompassing both attacks and defenses, but lack a dedicated,
in-depth analysis of poisoning attacks specifically in deep learning. In this
survey, we bridge this gap by presenting a comprehensive and targeted review of
data poisoning in deep learning. First, this survey categorizes data poisoning
attacks across multiple perspectives, providing an in-depth analysis of their
characteristics and underlying design princinples. Second, the discussion is
extended to the emerging area of data poisoning in large language models(LLMs).
Finally, we explore critical open challenges in the field and propose potential
research directions to advance the field further. To support further
exploration, an up-to-date repository of resources on data poisoning in deep
learning is available at https://github.com/Pinlong-Zhao/Data-Poisoning.


http://arxiv.org/abs/2503.21244
Improving $(\alpha, f)$-Byzantine Resilience in Federated Learning via layerwise aggregation and cosine distance. (1%)
Mario García-Márquez; Nuria Rodríguez-Barroso; M. Victoria Luzón; Francisco Herrera
  The rapid development of artificial intelligence systems has amplified
societal concerns regarding their usage, necessitating regulatory frameworks
that encompass data privacy. Federated Learning (FL) is posed as potential
solution to data privacy challenges in distributed machine learning by enabling
collaborative model training {without data sharing}. However, FL systems remain
vulnerable to Byzantine attacks, where malicious nodes contribute corrupted
model updates. While Byzantine Resilient operators have emerged as a widely
adopted robust aggregation algorithm to mitigate these attacks, its efficacy
diminishes significantly in high-dimensional parameter spaces, sometimes
leading to poor performing models. This paper introduces Layerwise Cosine
Aggregation, a novel aggregation scheme designed to enhance robustness of these
rules in such high-dimensional settings while preserving computational
efficiency. A theoretical analysis is presented, demonstrating the superior
robustness of the proposed Layerwise Cosine Aggregation compared to original
robust aggregation operators. Empirical evaluation across diverse image
classification datasets, under varying data distributions and Byzantine attack
scenarios, consistently demonstrates the improved performance of Layerwise
Cosine Aggregation, achieving up to a 16% increase in model accuracy.


http://arxiv.org/abs/2503.22048
ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models. (1%)
Chung-En Sun; Ge Yan; Tsui-Wei Weng
  Recent studies have shown that Large Language Models (LLMs) augmented with
chain-of-thought (CoT) reasoning demonstrate impressive problem-solving
abilities. However, in this work, we identify a recurring issue where these
models occasionally generate overly short reasoning, leading to degraded
performance on even simple mathematical problems. Specifically, we investigate
how reasoning length is embedded in the hidden representations of reasoning
models and its impact on accuracy. Our analysis reveals that reasoning length
is governed by a linear direction in the representation space, allowing us to
induce overly short reasoning by steering the model along this direction.
Building on this insight, we introduce ThinkEdit, a simple yet effective
weight-editing approach to mitigate the issue of overly short reasoning. We
first identify a small subset of attention heads (approximately 2%) that
predominantly drive short reasoning behavior. We then edit the output
projection weights of these heads to suppress the short reasoning direction.
With changes to only 0.1% of the model's parameters, ThinkEdit effectively
reduces overly short reasoning and yields notable accuracy gains for short
reasoning outputs (+5.44%), along with an overall improvement across multiple
math benchmarks (+2.43%). Our findings provide new mechanistic insights into
how reasoning length is controlled within LLMs and highlight the potential of
fine-grained model interventions to improve reasoning quality. Our code is
available at https://github.com/Trustworthy-ML-Lab/ThinkEdit


http://arxiv.org/abs/2503.20310
Enabling Heterogeneous Adversarial Transferability via Feature Permutation Attacks. (99%)
Tao Wu; Tie Luo
  Adversarial attacks in black-box settings are highly practical, with
transfer-based attacks being the most effective at generating adversarial
examples (AEs) that transfer from surrogate models to unseen target models.
However, their performance significantly degrades when transferring across
heterogeneous architectures -- such as CNNs, MLPs, and Vision Transformers
(ViTs) -- due to fundamental architectural differences. To address this, we
propose Feature Permutation Attack (FPA), a zero-FLOP, parameter-free method
that enhances adversarial transferability across diverse architectures. FPA
introduces a novel feature permutation (FP) operation, which rearranges pixel
values in selected feature maps to simulate long-range dependencies,
effectively making CNNs behave more like ViTs and MLPs. This enhances feature
diversity and improves transferability both across heterogeneous architectures
and within homogeneous CNNs. Extensive evaluations on 14 state-of-the-art
architectures show that FPA achieves maximum absolute gains in attack success
rates of 7.68% on CNNs, 14.57% on ViTs, and 14.48% on MLPs, outperforming
existing black-box attacks. Additionally, FPA is highly generalizable and can
seamlessly integrate with other transfer-based attacks to further boost their
performance. Our findings establish FPA as a robust, efficient, and
computationally lightweight strategy for enhancing adversarial transferability
across heterogeneous architectures.


http://arxiv.org/abs/2503.20844
Robust Deep Reinforcement Learning in Robotics via Adaptive Gradient-Masked Adversarial Attacks. (99%)
Zongyuan Zhang; Tianyang Duan; Zheng Lin; Dong Huang; Zihan Fang; Zekai Sun; Ling Xiong; Hongbin Liang; Heming Cui; Yong Cui; Yue Gao
  Deep reinforcement learning (DRL) has emerged as a promising approach for
robotic control, but its realworld deployment remains challenging due to its
vulnerability to environmental perturbations. Existing white-box adversarial
attack methods, adapted from supervised learning, fail to effectively target
DRL agents as they overlook temporal dynamics and indiscriminately perturb all
state dimensions, limiting their impact on long-term rewards. To address these
challenges, we propose the Adaptive Gradient-Masked Reinforcement (AGMR)
Attack, a white-box attack method that combines DRL with a gradient-based soft
masking mechanism to dynamically identify critical state dimensions and
optimize adversarial policies. AGMR selectively allocates perturbations to the
most impactful state features and incorporates a dynamic adjustment mechanism
to balance exploration and exploitation during training. Extensive experiments
demonstrate that AGMR outperforms state-of-the-art adversarial attack methods
in degrading the performance of the victim agent and enhances the victim
agent's robustness through adversarial defense mechanisms.


http://arxiv.org/abs/2503.20613
State-Aware Perturbation Optimization for Robust Deep Reinforcement Learning. (98%)
Zongyuan Zhang; Tianyang Duan; Zheng Lin; Dong Huang; Zihan Fang; Zekai Sun; Ling Xiong; Hongbin Liang; Heming Cui; Yong Cui
  Recently, deep reinforcement learning (DRL) has emerged as a promising
approach for robotic control. However, the deployment of DRL in real-world
robots is hindered by its sensitivity to environmental perturbations. While
existing whitebox adversarial attacks rely on local gradient information and
apply uniform perturbations across all states to evaluate DRL robustness, they
fail to account for temporal dynamics and state-specific vulnerabilities. To
combat the above challenge, we first conduct a theoretical analysis of
white-box attacks in DRL by establishing the adversarial victim-dynamics Markov
decision process (AVD-MDP), to derive the necessary and sufficient conditions
for a successful attack. Based on this, we propose a selective state-aware
reinforcement adversarial attack method, named STAR, to optimize perturbation
stealthiness and state visitation dispersion. STAR first employs a soft
mask-based state-targeting mechanism to minimize redundant perturbations,
enhancing stealthiness and attack effectiveness. Then, it incorporates an
information-theoretic optimization objective to maximize mutual information
between perturbations, environmental states, and victim actions, ensuring a
dispersed state-visitation distribution that steers the victim agent into
vulnerable states for maximum return reduction. Extensive experiments
demonstrate that STAR outperforms state-of-the-art benchmarks.


http://arxiv.org/abs/2503.20583
Feature Statistics with Uncertainty Help Adversarial Robustness. (97%)
Ran Wang; Xinlei Zhou; Meng Hu; Rihao Li; Wenhui Wu; Yuheng Jia
  Despite the remarkable success of deep neural networks (DNNs), the security
threat of adversarial attacks poses a significant challenge to the reliability
of DNNs. In this paper, both theoretically and empirically, we discover a
universal phenomenon that has been neglected in previous works, i.e.,
adversarial attacks tend to shift the distributions of feature statistics.
Motivated by this finding, and by leveraging the advantages of
uncertainty-aware stochastic methods in building robust models efficiently, we
propose an uncertainty-driven feature statistics adjustment module for
robustness enhancement, named Feature Statistics with Uncertainty (FSU). It
randomly resamples channel-wise feature means and standard deviations of
examples from multivariate Gaussian distributions, which helps to reconstruct
the perturbed examples and calibrate the shifted distributions. The calibration
recovers some domain characteristics of the data for classification, thereby
mitigating the influence of perturbations and weakening the ability of attacks
to deceive models. The proposed FSU module has universal applicability in
training, attacking, predicting, and fine-tuning, demonstrating impressive
robustness enhancement ability at a trivial additional time cost. For example,
by fine-tuning the well-established models with FSU, the state-of-the-art
methods achieve up to 17.13% and 34.82% robustness improvement against powerful
AA and CW attacks on benchmark datasets.


http://arxiv.org/abs/2503.20454
Lipschitz Constant Meets Condition Number: Learning Robust and Compact Deep Neural Networks. (74%)
Yangqi Feng; Shing-Ho J. Lin; Baoyuan Gao; Xian Wei
  Recent research has revealed that high compression of Deep Neural Networks
(DNNs), e.g., massive pruning of the weight matrix of a DNN, leads to a severe
drop in accuracy and susceptibility to adversarial attacks. Integration of
network pruning into an adversarial training framework has been proposed to
promote adversarial robustness. It has been observed that a highly pruned
weight matrix tends to be ill-conditioned, i.e., increasing the condition
number of the weight matrix. This phenomenon aggravates the vulnerability of a
DNN to input noise. Although a highly pruned weight matrix is considered to be
able to lower the upper bound of the local Lipschitz constant to tolerate large
distortion, the ill-conditionedness of such a weight matrix results in a
non-robust DNN model. To overcome this challenge, this work develops novel
joint constraints to adjust the weight distribution of networks, namely, the
Transformed Sparse Constraint joint with Condition Number Constraint (TSCNC),
which copes with smoothing distribution and differentiable constraint functions
to reduce condition number and thus avoid the ill-conditionedness of weight
matrices. Furthermore, our theoretical analyses unveil the relevance between
the condition number and the local Lipschitz constant of the weight matrix,
namely, the sharply increasing condition number becomes the dominant factor
that restricts the robustness of over-sparsified models. Extensive experiments
are conducted on several public datasets, and the results show that the
proposed constraints significantly improve the robustness of a DNN with high
pruning rates.


http://arxiv.org/abs/2503.21824
Protecting Your Video Content: Disrupting Automated Video-based LLM Annotations. (12%)
Haitong Liu; Kuofeng Gao; Yang Bai; Jinmin Li; Jinxiao Shan; Tao Dai; Shu-Tao Xia
  Recently, video-based large language models (video-based LLMs) have achieved
impressive performance across various video comprehension tasks. However, this
rapid advancement raises significant privacy and security concerns,
particularly regarding the unauthorized use of personal video data in automated
annotation by video-based LLMs. These unauthorized annotated video-text pairs
can then be used to improve the performance of downstream tasks, such as
text-to-video generation. To safeguard personal videos from unauthorized use,
we propose two series of protective video watermarks with imperceptible
adversarial perturbations, named Ramblings and Mutes. Concretely, Ramblings aim
to mislead video-based LLMs into generating inaccurate captions for the videos,
thereby degrading the quality of video annotations through inconsistencies
between video content and captions. Mutes, on the other hand, are designed to
prompt video-based LLMs to produce exceptionally brief captions, lacking
descriptive detail. Extensive experiments demonstrate that our video
watermarking methods effectively protect video data by significantly reducing
video annotation performance across various video-based LLMs, showcasing both
stealthiness and robustness in protecting personal video content. Our code is
available at https://github.com/ttthhl/Protecting_Your_Video_Content.


http://arxiv.org/abs/2503.20925
Prototype Guided Backdoor Defense. (11%)
Venkat Adithya Amula; Sunayana Samavedam; Saurabh Saini; Avani Gupta; Narayanan P J
  Deep learning models are susceptible to {\em backdoor attacks} involving
malicious attackers perturbing a small subset of training data with a {\em
trigger} to causes misclassifications. Various triggers have been used,
including semantic triggers that are easily realizable without requiring the
attacker to manipulate the image. The emergence of generative AI has eased the
generation of varied poisoned samples. Robustness across types of triggers is
crucial to effective defense. We propose Prototype Guided Backdoor Defense
(PGBD), a robust post-hoc defense that scales across different trigger types,
including previously unsolved semantic triggers. PGBD exploits displacements in
the geometric spaces of activations to penalize movements toward the trigger.
This is done using a novel sanitization loss of a post-hoc fine-tuning step.
The geometric approach scales easily to all types of attacks. PGBD achieves
better performance across all settings. We also present the first defense
against a new semantic attack on celebrity face images. Project page:
\hyperlink{https://venkatadithya9.github.io/pgbd.github.io/}{this https URL}.


http://arxiv.org/abs/2503.20660
DR-PETS: Learning-Based Control With Planning in Adversarial Environments. (4%)
Hozefa Jesawada; Antonio Acernese; Giovanni Russo; Vecchio Carmen Del
  Ensuring robustness against epistemic, possibly adversarial, perturbations is
essential for reliable real-world decision-making. While the Probabilistic
Ensembles with Trajectory Sampling (PETS) algorithm inherently handles
uncertainty via ensemble-based probabilistic models, it lacks guarantees
against structured adversarial or worst-case uncertainty distributions. To
address this, we propose DR-PETS, a distributionally robust extension of PETS
that certifies robustness against adversarial perturbations. We formalize
uncertainty via a p-Wasserstein ambiguity set, enabling worst-case-aware
planning through a min-max optimization framework. While PETS passively
accounts for stochasticity, DR-PETS actively optimizes robustness via a
tractable convex approximation integrated into PETS planning loop. Experiments
on pendulum stabilization and cart-pole balancing show that DR-PETS certifies
robustness against adversarial parameter perturbations, achieving consistent
performance in worst-case scenarios where PETS deteriorates.


http://arxiv.org/abs/2503.20462
Multi-agent Uncertainty-Aware Pessimistic Model-Based Reinforcement Learning for Connected Autonomous Vehicles. (1%)
Ruoqi Wen; Rongpeng Li; Xing Xu; Zhifeng Zhao
  Deep Reinforcement Learning (DRL) holds significant promise for achieving
human-like Autonomous Vehicle (AV) capabilities, but suffers from low sample
efficiency and challenges in reward design. Model-Based Reinforcement Learning
(MBRL) offers improved sample efficiency and generalizability compared to
Model-Free Reinforcement Learning (MFRL) in various multi-agent decision-making
scenarios. Nevertheless, MBRL faces critical difficulties in estimating
uncertainty during the model learning phase, thereby limiting its scalability
and applicability in real-world scenarios. Additionally, most Connected
Autonomous Vehicle (CAV) studies focus on single-agent decision-making, while
existing multi-agent MBRL solutions lack computationally tractable algorithms
with Probably Approximately Correct (PAC) guarantees, an essential factor for
ensuring policy reliability with limited training data. To address these
challenges, we propose MA-PMBRL, a novel Multi-Agent Pessimistic Model-Based
Reinforcement Learning framework for CAVs, incorporating a max-min optimization
approach to enhance robustness and decision-making. To mitigate the inherent
subjectivity of uncertainty estimation in MBRL and avoid incurring catastrophic
failures in AV, MA-PMBRL employs a pessimistic optimization framework combined
with Projected Gradient Descent (PGD) for both model and policy learning.
MA-PMBRL also employs general function approximations under partial dataset
coverage to enhance learning efficiency and system-level performance. By
bounding the suboptimality of the resulting policy under mild theoretical
assumptions, we successfully establish PAC guarantees for MA-PMBRL,
demonstrating that the proposed framework represents a significant step toward
scalable, efficient, and reliable multi-agent decision-making for CAVs.


http://arxiv.org/abs/2503.20281
Are We There Yet? Unraveling the State-of-the-Art Graph Network Intrusion Detection Systems. (1%)
Chenglong Wang; Pujia Zheng; Jiaping Gui; Cunqing Hua; Wajih Ul Hassan
  Network Intrusion Detection Systems (NIDS) are vital for ensuring enterprise
security. Recently, Graph-based NIDS (GIDS) have attracted considerable
attention because of their capability to effectively capture the complex
relationships within the graph structures of data communications. Despite their
promise, the reproducibility and replicability of these GIDS remain largely
unexplored, posing challenges for developing reliable and robust detection
systems. This study bridges this gap by designing a systematic approach to
evaluate state-of-the-art GIDS, which includes critically assessing, extending,
and clarifying the findings of these systems. We further assess the robustness
of GIDS under adversarial attacks. Evaluations were conducted on three public
datasets as well as a newly collected large-scale enterprise dataset. Our
findings reveal significant performance discrepancies, highlighting challenges
related to dataset scale, model inputs, and implementation settings. We
demonstrate difficulties in reproducing and replicating results, particularly
concerning false positive rates and robustness against adversarial attacks.
This work provides valuable insights and recommendations for future research,
emphasizing the importance of rigorous reproduction and replication studies in
developing robust and generalizable GIDS solutions.


http://arxiv.org/abs/2503.19519
Towards Imperceptible Adversarial Attacks for Time Series Classification with Local Perturbations and Frequency Analysis. (99%)
Wenwei Gu; Renyi Zhong; Jianping Zhang; Michael R. Lyu
  Adversarial attacks in time series classification (TSC) models have recently
gained attention due to their potential to compromise model robustness.
Imperceptibility is crucial, as adversarial examples detected by the human
vision system (HVS) can render attacks ineffective. Many existing methods fail
to produce high-quality imperceptible examples, often generating perturbations
with more perceptible low-frequency components, like square waves, and global
perturbations that reduce stealthiness. This paper aims to improve the
imperceptibility of adversarial attacks on TSC models by addressing frequency
components and time series locality. We propose the Shapelet-based
Frequency-domain Attack (SFAttack), which uses local perturbations focused on
time series shapelets to enhance discriminative information and stealthiness.
Additionally, we introduce a low-frequency constraint to confine perturbations
to high-frequency components, enhancing imperceptibility.


http://arxiv.org/abs/2503.19591
Boosting the Transferability of Audio Adversarial Examples with Acoustic Representation Optimization. (99%)
Weifei Jin; Junjie Su; Hejia Wang; Yulin Ye; Jie Hao
  With the widespread application of automatic speech recognition (ASR)
systems, their vulnerability to adversarial attacks has been extensively
studied. However, most existing adversarial examples are generated on specific
individual models, resulting in a lack of transferability. In real-world
scenarios, attackers often cannot access detailed information about the target
model, making query-based attacks unfeasible. To address this challenge, we
propose a technique called Acoustic Representation Optimization that aligns
adversarial perturbations with low-level acoustic characteristics derived from
speech representation models. Rather than relying on model-specific,
higher-layer abstractions, our approach leverages fundamental acoustic
representations that remain consistent across diverse ASR architectures. By
enforcing an acoustic representation loss to guide perturbations toward these
robust, lower-level representations, we enhance the cross-model transferability
of adversarial examples without degrading audio quality. Our method is
plug-and-play and can be integrated with any existing attack methods. We
evaluate our approach on three modern ASR models, and the experimental results
demonstrate that our method significantly improves the transferability of
adversarial examples generated by previous methods while preserving the audio
quality.


http://arxiv.org/abs/2503.19817
Bitstream Collisions in Neural Image Compression via Adversarial Perturbations. (96%)
Jordan Madden; Lhamo Dorje; Xiaohua Li
  Neural image compression (NIC) has emerged as a promising alternative to
classical compression techniques, offering improved compression ratios. Despite
its progress towards standardization and practical deployment, there has been
minimal exploration into it's robustness and security. This study reveals an
unexpected vulnerability in NIC - bitstream collisions - where semantically
different images produce identical compressed bitstreams. Utilizing a novel
whitebox adversarial attack algorithm, this paper demonstrates that adding
carefully crafted perturbations to semantically different images can cause
their compressed bitstreams to collide exactly. The collision vulnerability
poses a threat to the practical usability of NIC, particularly in
security-critical applications. The cause of the collision is analyzed, and a
simple yet effective mitigation method is presented.


http://arxiv.org/abs/2503.21805
ImF: Implicit Fingerprint for Large Language Models. (92%)
Wu jiaxuan; Peng Wanli; Fu hang; Xue Yiming; Wen juan
  Training large language models (LLMs) is resource-intensive and expensive,
making protecting intellectual property (IP) for LLMs crucial. Recently,
embedding fingerprints into LLMs has emerged as a prevalent method for
establishing model ownership. However, existing fingerprinting techniques
typically embed identifiable patterns with weak semantic coherence, resulting
in fingerprints that significantly differ from the natural question-answering
(QA) behavior inherent to LLMs. This discrepancy undermines the stealthiness of
the embedded fingerprints and makes them vulnerable to adversarial attacks. In
this paper, we first demonstrate the critical vulnerability of existing
fingerprint embedding methods by introducing a novel adversarial attack named
Generation Revision Intervention (GRI) attack. GRI attack exploits the semantic
fragility of current fingerprinting methods, effectively erasing fingerprints
by disrupting their weakly correlated semantic structures. Our empirical
evaluation highlights that traditional fingerprinting approaches are
significantly compromised by the GRI attack, revealing severe limitations in
their robustness under realistic adversarial conditions. To advance the
state-of-the-art in model fingerprinting, we propose a novel model fingerprint
paradigm called Implicit Fingerprints (ImF). ImF leverages steganography
techniques to subtly embed ownership information within natural texts,
subsequently using Chain-of-Thought (CoT) prompting to construct semantically
coherent and contextually natural QA pairs. This design ensures that
fingerprints seamlessly integrate with the standard model behavior, remaining
indistinguishable from regular outputs and substantially reducing the risk of
accidental triggering and targeted removal. We conduct a comprehensive
evaluation of ImF on 15 diverse LLMs, spanning different architectures and
varying scales.


http://arxiv.org/abs/2503.19347
Stop Walking in Circles! Bailing Out Early in Projected Gradient Descent. (92%)
Philip Doldo; Derek Everett; Amol Khanna; Andre T Nguyen; Edward Raff
  Projected Gradient Descent (PGD) under the $L_\infty$ ball has become one of
the defacto methods used in adversarial robustness evaluation for computer
vision (CV) due to its reliability and efficacy, making a strong and
easy-to-implement iterative baseline. However, PGD is computationally demanding
to apply, especially when using thousands of iterations is the current
best-practice recommendation to generate an adversarial example for a single
image. In this work, we introduce a simple novel method for early termination
of PGD based on cycle detection by exploiting the geometry of how PGD is
implemented in practice and show that it can produce large speedup factors
while providing the \emph{exact} same estimate of model robustness as standard
PGD. This method substantially speeds up PGD without sacrificing any attack
strength, enabling evaluations of robustness that were previously
computationally intractable.


http://arxiv.org/abs/2503.19791
SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generation. (75%)
Jingdan Kang; Haoxin Yang; Yan Cai; Huaidong Zhang; Xuemiao Xu; Yong Du; Shengfeng He
  Image generation technology has brought significant advancements across
various fields but has also raised concerns about data misuse and potential
rights infringements, particularly with respect to creating visual artworks.
Current methods aimed at safeguarding artworks often employ adversarial
attacks. However, these methods face challenges such as poor transferability,
high computational costs, and the introduction of noticeable noise, which
compromises the aesthetic quality of the original artwork. To address these
limitations, we propose a Structurally Imperceptible and Transferable
Adversarial (SITA) attacks. SITA leverages a CLIP-based destylization loss,
which decouples and disrupts the robust style representation of the image. This
disruption hinders style extraction during stylized image generation, thereby
impairing the overall stylization process. Importantly, SITA eliminates the
need for a surrogate diffusion model, leading to significantly reduced
computational overhead. The method's robust style feature disruption ensures
high transferability across diverse models. Moreover, SITA introduces
perturbations by embedding noise within the imperceptible structural details of
the image. This approach effectively protects against style extraction without
compromising the visual quality of the artwork. Extensive experiments
demonstrate that SITA offers superior protection for artworks against
unauthorized use in stylized generation. It significantly outperforms existing
methods in terms of transferability, computational efficiency, and noise
imperceptibility. Code is available at https://github.com/A-raniy-day/SITA.


http://arxiv.org/abs/2503.20187
Network Inversion for Generating Confidently Classified Counterfeits. (69%)
Pirzada Suhail; Amit Sethi
  In machine learning, especially with vision classifiers, generating inputs
that are confidently classified by the model is essential for understanding its
decision boundaries and behavior. However, creating such samples that are
confidently classified yet distinct from the training data distribution is a
challenge. Traditional methods often modify existing inputs, but they don't
always ensure confident classification. In this work, we extend network
inversion techniques to generate Confidently Classified Counterfeits-synthetic
samples that are confidently classified by the model despite being
significantly different from the training data. We achieve this by modifying
the generator's conditioning mechanism from soft vector conditioning to one-hot
vector conditioning and applying Kullback-Leibler divergence (KLD) between the
one-hot vectors and the classifier's output distribution. This encourages the
generator to produce samples that are both plausible and confidently
classified. Generating Confidently Classified Counterfeits is crucial for
ensuring the safety and reliability of machine learning systems, particularly
in safety-critical applications where models must exhibit confidence only on
data within the training distribution. By generating such counterfeits, we
challenge the assumption that high-confidence predictions are always indicative
of in-distribution data, providing deeper insights into the model's limitations
and decision-making process.


http://arxiv.org/abs/2503.20823
Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy. (22%)
Joonhyun Jeong; Seyun Bae; Yeonsung Jung; Jaeryong Hwang; Eunho Yang
  Despite the remarkable versatility of Large Language Models (LLMs) and
Multimodal LLMs (MLLMs) to generalize across both language and vision tasks,
LLMs and MLLMs have shown vulnerability to jailbreaking, generating textual
outputs that undermine safety, ethical, and bias standards when exposed to
harmful or sensitive inputs. With the recent advancement of safety alignment
via preference-tuning from human feedback, LLMs and MLLMs have been equipped
with safety guardrails to yield safe, ethical, and fair responses with regard
to harmful inputs. However, despite the significance of safety alignment,
research on the vulnerabilities remains largely underexplored. In this paper,
we investigate the unexplored vulnerability of the safety alignment, examining
its ability to consistently provide safety guarantees for
out-of-distribution(OOD)-ifying harmful inputs that may fall outside the
aligned data distribution. Our key observation is that OOD-ifying the vanilla
harmful inputs highly increases the uncertainty of the model to discern the
malicious intent within the input, leading to a higher chance of being
jailbroken. Exploiting this vulnerability, we propose JOOD, a new Jailbreak
framework via OOD-ifying inputs beyond the safety alignment. We explore various
off-the-shelf visual and textual transformation techniques for OOD-ifying the
harmful inputs. Notably, we observe that even simple mixing-based techniques
such as image mixup prove highly effective in increasing the uncertainty of the
model, thereby facilitating the bypass of the safety alignment. Experiments
across diverse jailbreak scenarios demonstrate that JOOD effectively jailbreaks
recent proprietary LLMs and MLLMs such as GPT-4 and o1 with high attack success
rate, which previous attack approaches have consistently struggled to
jailbreak. Code is available at https://github.com/naver-ai/JOOD.


http://arxiv.org/abs/2503.19629
Lifting Linear Sketches: Optimal Bounds and Adversarial Robustness. (8%)
Elena Gribelyuk; Honghao Lin; David P. Woodruff; Huacheng Yu; Samson Zhou
  We introduce a novel technique for ``lifting'' dimension lower bounds for
linear sketches in the real-valued setting to dimension lower bounds for linear
sketches with polynomially-bounded integer entries when the input is a
polynomially-bounded integer vector. Using this technique, we obtain the first
optimal sketching lower bounds for discrete inputs in a data stream, for
classical problems such as approximating the frequency moments, estimating the
operator norm, and compressed sensing. Additionally, we lift the adaptive
attack of Hardt and Woodruff (STOC, 2013) for breaking any real-valued linear
sketch via a sequence of real-valued queries, and show how to obtain an attack
on any integer-valued linear sketch using integer-valued queries. This shows
that there is no linear sketch in a data stream with insertions and deletions
that is adversarially robust for approximating any $L_p$ norm of the input,
resolving a central open question for adversarially robust streaming
algorithms. To do so, we introduce a new pre-processing technique of
independent interest which, given an integer-valued linear sketch, increases
the dimension of the sketch by only a constant factor in order to make the
orthogonal lattice to its row span smooth. This pre-processing then enables us
to leverage results in lattice theory on discrete Gaussian distributions and
reason that efficient discrete sketches imply efficient continuous sketches.
Our work resolves open questions from the Banff '14 and '17 workshops on
Communication Complexity and Applications, as well as the STOC '21 and FOCS '23
workshops on adaptivity and robustness.


http://arxiv.org/abs/2503.19338
Membership Inference Attacks on Large-Scale Models: A Survey. (2%)
Hengyu Wu; Yang Cao
  The adoption of the Large Language Model (LLM) has accelerated dramatically
since the ChatGPT from OpenAI went online in November 2022. Recent advances in
Large Multimodal Models (LMMs), which process diverse data types and enable
interaction through various channels, have expanded beyond the text-to-text
limitations of early LLMs, attracting significant and concurrent attention from
both researchers and industry. While LLMs and LMMs are starting to spread
widely, concerns about their privacy risks are increasing as well. Membership
Inference Attacks (MIAs), techniques used to determine whether a particular
data point was part of a model's training set, serve as a key metric for
assessing the privacy vulnerabilities of machine learning models. Hu et al.
show that various machine learning algorithms are vulnerable to MIA. Despite
extensive studies on MIAs in traditional models, there remains a lack of
systematic surveys addressing their effectiveness and implications in modern
large-scale models like LLMs and LMMs. In this paper, we systematically
reviewed recent studies of MIA against LLMs and LMMs. We analyzed and
categorized each attack based on their methodology and scenario and discussed
the limitations in existing research. Additionally, we examine privacy concerns
associated with the fine-tuning process. Finally, we provided some suggestions
for future research in this direction.


http://arxiv.org/abs/2503.19609
Nanopass Back-Translation of Call-Return Trees for Mechanized Secure Compilation Proofs. (1%)
Jérémy Thibault; Joseph Lenormand; Catalin Hritcu
  Researchers aim to build secure compilation chains enforcing that if there is
no attack a source context can mount against a source program then there is
also no attack an adversarial target context can mount against the compiled
program. Proving that these compilation chains are secure is, however,
challenging, and involves a non-trivial back-translation step: for any attack a
target context mounts against the compiled program one has to exhibit a source
context mounting the same attack against the source program. We describe a
novel back-translation technique, which results in simpler proofs that can be
more easily mechanized in a proof assistant. Given a finite set of finite trace
prefixes, capturing the interaction recorded during an attack between a target
context and the compiled program, we build a call-return tree that we
back-translate into a source context producing the same trace prefixes. We use
state in the generated source context to record the current location in the
call-return tree. The back-translation is done in several small steps, each
adding to the tree new information describing how the location should change
depending on how the context regains control. To prove this back-translation
correct we give semantics to every intermediate call-return tree language,
using ghost state to store information and explicitly enforce execution
invariants. We prove several small forward simulations, basically seeing the
back-translation as a verified nanopass compiler. Thanks to this modular
structure, we are able to mechanize this complex back-translation and its
correctness proof in the Rocq prover without too much effort.


http://arxiv.org/abs/2503.19318
Efficient Adversarial Detection Frameworks for Vehicle-to-Microgrid Services in Edge Computing. (98%)
Ahmed Omara; Burak Kantarci
  As Artificial Intelligence (AI) becomes increasingly integrated into
microgrid control systems, the risk of malicious actors exploiting
vulnerabilities in Machine Learning (ML) algorithms to disrupt power generation
and distribution grows. Detection models to identify adversarial attacks need
to meet the constraints of edge environments, where computational power and
memory are often limited. To address this issue, we propose a novel strategy
that optimizes detection models for Vehicle-to-Microgrid (V2M) edge
environments without compromising performance against inference and evasion
attacks. Our approach integrates model design and compression into a unified
process and results in a highly compact detection model that maintains high
accuracy. We evaluated our method against four benchmark evasion attacks-Fast
Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Carlini & Wagner
method (C&W) and Conditional Generative Adversarial Network (CGAN) method-and
two knowledge-based attacks, white-box and gray-box. Our optimized model
reduces memory usage from 20MB to 1.3MB, inference time from 3.2 seconds to 0.9
seconds, and GPU utilization from 5% to 2.68%.


http://arxiv.org/abs/2503.18503
Deterministic Certification of Graph Neural Networks against Graph Poisoning Attacks with Arbitrary Perturbations. (92%)
Jiate Li; Meng Pang; Yun Dong; Binghui Wang
  Graph neural networks (GNNs) are becoming the de facto method to learn on the
graph data and have achieved the state-of-the-art on node and graph
classification tasks. However, recent works show GNNs are vulnerable to
training-time poisoning attacks -- marginally perturbing edges, nodes, or/and
node features of training graph(s) can largely degrade GNNs' testing
performance. Most previous defenses against graph poisoning attacks are
empirical and are soon broken by adaptive / stronger ones. A few provable
defenses provide robustness guarantees, but have large gaps when applied in
practice: 1) restrict the attacker on only one type of perturbation; 2) design
for a particular GNN architecture or task; and 3) robustness guarantees are not
100\% accurate.
  In this work, we bridge all these gaps by developing PGNNCert, the first
certified defense of GNNs against poisoning attacks under arbitrary (edge,
node, and node feature) perturbations with deterministic robustness guarantees.
Extensive evaluations on multiple node and graph classification datasets and
GNNs demonstrate the effectiveness of PGNNCert to provably defend against
arbitrary poisoning perturbations. PGNNCert is also shown to significantly
outperform the state-of-the-art certified defenses against edge perturbation or
node perturbation during GNN training.


http://arxiv.org/abs/2503.18678
NullSwap: Proactive Identity Cloaking Against Deepfake Face Swapping. (78%)
Tianyi Wang; Harry Cheng; Xiao Zhang; Yinglong Wang
  Suffering from performance bottlenecks in passively detecting high-quality
Deepfake images due to the advancement of generative models, proactive
perturbations offer a promising approach to disabling Deepfake manipulations by
inserting signals into benign images. However, existing proactive perturbation
approaches remain unsatisfactory in several aspects: 1) visual degradation due
to direct element-wise addition; 2) limited effectiveness against face swapping
manipulation; 3) unavoidable reliance on white- and grey-box settings to
involve generative models during training. In this study, we analyze the
essence of Deepfake face swapping and argue the necessity of protecting source
identities rather than target images, and we propose NullSwap, a novel
proactive defense approach that cloaks source image identities and nullifies
face swapping under a pure black-box scenario. We design an Identity Extraction
module to obtain facial identity features from the source image, while a
Perturbation Block is then devised to generate identity-guided perturbations
accordingly. Meanwhile, a Feature Block extracts shallow-level image features,
which are then fused with the perturbation in the Cloaking Block for image
reconstruction. Furthermore, to ensure adaptability across different identity
extractors in face swapping algorithms, we propose Dynamic Loss Weighting to
adaptively balance identity losses. Experiments demonstrate the outstanding
ability of our approach to fool various identity recognition models,
outperforming state-of-the-art proactive perturbations in preventing face
swapping models from generating images with correct source identities.


http://arxiv.org/abs/2503.18813
Defeating Prompt Injections by Design. (70%)
Edoardo Debenedetti; Ilia Shumailov; Tianqi Fan; Jamie Hayes; Nicholas Carlini; Daniel Fabian; Christoph Kern; Chongyang Shi; Andreas Terzis; Florian Tramèr
  Large Language Models (LLMs) are increasingly deployed in agentic systems
that interact with an external environment. However, LLM agents are vulnerable
to prompt injection attacks when handling untrusted data. In this paper we
propose CaMeL, a robust defense that creates a protective system layer around
the LLM, securing it even when underlying models may be susceptible to attacks.
To operate, CaMeL explicitly extracts the control and data flows from the
(trusted) query; therefore, the untrusted data retrieved by the LLM can never
impact the program flow. To further improve security, CaMeL relies on a notion
of a capability to prevent the exfiltration of private data over unauthorized
data flows. We demonstrate effectiveness of CaMeL by solving $67\%$ of tasks
with provable security in AgentDojo [NeurIPS 2024], a recent agentic security
benchmark.


http://arxiv.org/abs/2503.19099
Masks and Mimicry: Strategic Obfuscation and Impersonation Attacks on Authorship Verification. (67%)
Kenneth Alperin; Rohan Leekha; Adaku Uchendu; Trang Nguyen; Srilakshmi Medarametla; Carlos Levya Capote; Seth Aycock; Charlie Dagli
  The increasing use of Artificial Intelligence (AI) technologies, such as
Large Language Models (LLMs) has led to nontrivial improvements in various
tasks, including accurate authorship identification of documents. However,
while LLMs improve such defense techniques, they also simultaneously provide a
vehicle for malicious actors to launch new attack vectors. To combat this
security risk, we evaluate the adversarial robustness of authorship models
(specifically an authorship verification model) to potent LLM-based attacks.
These attacks include untargeted methods - \textit{authorship obfuscation} and
targeted methods - \textit{authorship impersonation}. For both attacks, the
objective is to mask or mimic the writing style of an author while preserving
the original texts' semantics, respectively. Thus, we perturb an accurate
authorship verification model, and achieve maximum attack success rates of 92\%
and 78\% for both obfuscation and impersonation attacks, respectively.


http://arxiv.org/abs/2503.19176
SoK: How Robust is Audio Watermarking in Generative AI models? (11%)
Yizhu Wen; Ashwin Innuganti; Aaron Bien Ramos; Hanqing Guo; Qiben Yan
  Audio watermarking is increasingly used to verify the provenance of
AI-generated content, enabling applications such as detecting AI-generated
speech, protecting music IP, and defending against voice cloning. To be
effective, audio watermarks must resist removal attacks that distort signals to
evade detection. While many schemes claim robustness, these claims are
typically tested in isolation and against a limited set of attacks. A
systematic evaluation against diverse removal attacks is lacking, hindering
practical deployment. In this paper, we investigate whether recent watermarking
schemes that claim robustness can withstand a broad range of removal attacks.
First, we introduce a taxonomy covering 22 audio watermarking schemes. Next, we
summarize their underlying technologies and potential vulnerabilities. We then
present a large-scale empirical study to assess their robustness. To support
this, we build an evaluation framework encompassing 22 types of removal attacks
(109 configurations) including signal-level, physical-level, and AI-induced
distortions. We reproduce 9 watermarking schemes using open-source code,
identify 8 new highly effective attacks, and highlight 11 key findings that
expose the fundamental limitations of these methods across 3 public datasets.
Our results reveal that none of the surveyed schemes can withstand all tested
distortions. This evaluation offers a comprehensive view of how current
watermarking methods perform under real-world threats. Our demo and code are
available at https://sokaudiowm.github.io/.


http://arxiv.org/abs/2503.19070
Graph-Level Label-Only Membership Inference Attack against Graph Neural Networks. (5%)
Jiazhu Dai; Yubing Lu
  Graph neural networks (GNNs) are widely used for graph-structured data but
are vulnerable to membership inference attacks (MIAs) in graph classification
tasks, which determine if a graph was part of the training dataset, potentially
causing data leakage. Existing MIAs rely on prediction probability vectors, but
they become ineffective when only prediction labels are available. We propose a
Graph-level Label-Only Membership Inference Attack (GLO-MIA), which is based on
the intuition that the target model's predictions on training data are more
stable than those on testing data. GLO-MIA generates a set of perturbed graphs
for target graph by adding perturbations to its effective features and queries
the target model with the perturbed graphs to get their prediction labels,
which are then used to calculate robustness score of the target graph. Finally,
by comparing the robustness score with a predefined threshold, the membership
of the target graph can be inferred correctly with high probability. Our
evaluation on three datasets and four GNN models shows that GLO-MIA achieves an
attack accuracy of up to 0.825, outperforming baseline work by 8.5% and closely
matching the performance of probability-based MIAs, even with only prediction
labels.


http://arxiv.org/abs/2503.18784
Leveraging Perturbation Robustness to Enhance Out-of-Distribution Detection. (2%)
Wenxi Chen; Raymond A. Yeh; Shaoshuai Mou; Yan Gu
  Out-of-distribution (OOD) detection is the task of identifying inputs that
deviate from the training data distribution. This capability is essential for
safely deploying deep computer vision models in open-world environments. In
this work, we propose a post-hoc method, Perturbation-Rectified OOD detection
(PRO), based on the insight that prediction confidence for OOD inputs is more
susceptible to reduction under perturbation than in-distribution (IND) inputs.
Based on the observation, we propose an adversarial score function that
searches for the local minimum scores near the original inputs by applying
gradient descent. This procedure enhances the separability between IND and OOD
samples. Importantly, the approach improves OOD detection performance without
complex modifications to the underlying model architectures. We conduct
extensive experiments using the OpenOOD benchmark~\cite{yang2022openood}. Our
approach further pushes the limit of softmax-based OOD detection and is the
leading post-hoc method for small-scale models. On a CIFAR-10 model with
adversarial training, PRO effectively detects near-OOD inputs, achieving a
reduction of more than 10\% on FPR@95 compared to state-of-the-art methods.


http://arxiv.org/abs/2503.19293
Robustness of Proof of Team Sprint (PoTS) Against Attacks: A Simulation-Based Analysis. (2%)
Naoki Yonezawa
  This study evaluates the robustness of Proof of Team Sprint (PoTS) against
adversarial attacks through simulations, focusing on the attacker win rate and
computational efficiency under varying team sizes (\( N \)) and attacker ratios
(\( \alpha \)). Our results demonstrate that PoTS effectively reduces an
attacker's ability to dominate the consensus process. For instance, when \(
\alpha = 0.5 \), the attacker win rate decreases from 50.7\% at \( N = 1 \) to
below 0.4\% at \( N = 8 \), effectively neutralizing adversarial influence.
Similarly, at \( \alpha = 0.8 \), the attacker win rate drops from 80.47\% at
\( N = 1 \) to only 2.79\% at \( N = 16 \). In addition to its strong security
properties, PoTS maintains high computational efficiency. We introduce the
concept of Normalized Computation Efficiency (NCE) to quantify this efficiency
gain, showing that PoTS significantly improves resource utilization as team
size increases. The results indicate that as \( N \) grows, PoTS not only
enhances security but also achieves better computational efficiency due to the
averaging effects of execution time variations. These findings highlight PoTS
as a promising alternative to traditional consensus mechanisms, offering both
robust security and efficient resource utilization. By leveraging team-based
block generation and randomized participant reassignment, PoTS provides a
scalable and resilient approach to decentralized consensus.


http://arxiv.org/abs/2503.19326
Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Steps. (1%)
Yu Cui; Bryan Hooi; Yujun Cai; Yiwei Wang
  Recent reasoning large language models (LLMs) have demonstrated remarkable
improvements in mathematical reasoning capabilities through long
Chain-of-Thought. The reasoning tokens of these models enable self-correction
within reasoning chains, enhancing robustness. This motivates our exploration:
how vulnerable are reasoning LLMs to subtle errors in their input reasoning
chains? We introduce "Compromising Thought" (CPT), a vulnerability where models
presented with reasoning tokens containing manipulated calculation results tend
to ignore correct reasoning steps and adopt incorrect results instead. Through
systematic evaluation across multiple reasoning LLMs, we design three
increasingly explicit prompting methods to measure CPT resistance, revealing
that models struggle significantly to identify and correct these manipulations.
Notably, contrary to existing research suggesting structural alterations affect
model performance more than content modifications, we find that local ending
token manipulations have greater impact on reasoning outcomes than structural
changes. Moreover, we discover a security vulnerability in DeepSeek-R1 where
tampered reasoning tokens can trigger complete reasoning cessation. Our work
enhances understanding of reasoning robustness and highlights security
considerations for reasoning-intensive applications.


http://arxiv.org/abs/2503.19202
Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery. (1%)
Sara Al-Emadi; Yin Yang; Ferda Ofli
  Object detectors have achieved remarkable performance in many applications;
however, these deep learning models are typically designed under the i.i.d.
assumption, meaning they are trained and evaluated on data sampled from the
same (source) distribution. In real-world deployment, however, target
distributions often differ from source data, leading to substantial performance
degradation. Domain Generalisation (DG) seeks to bridge this gap by enabling
models to generalise to Out-Of-Distribution (OOD) data without access to target
distributions during training, enhancing robustness to unseen conditions. In
this work, we examine the generalisability and robustness of state-of-the-art
object detectors under real-world distribution shifts, focusing particularly on
spatial domain shifts. Despite the need, a standardised benchmark dataset
specifically designed for assessing object detection under realistic DG
scenarios is currently lacking. To address this, we introduce Real-World
Distribution Shifts (RWDS), a suite of three novel DG benchmarking datasets
that focus on humanitarian and climate change applications. These datasets
enable the investigation of domain shifts across (i) climate zones and (ii)
various disasters and geographic regions. To our knowledge, these are the first
DG benchmarking datasets tailored for object detection in real-world,
high-impact contexts. We aim for these datasets to serve as valuable resources
for evaluating the robustness and generalisation of future object detection
models. Our datasets and code are available at https://github.com/RWGAI/RWDS.


http://arxiv.org/abs/2503.17987
Metaphor-based Jailbreaking Attacks on Text-to-Image Models. (87%)
Chenyu Zhang; Yiwen Ma; Lanjun Wang; Wenhui Li; Yi Tu; An-An Liu
  To mitigate misuse, text-to-image~(T2I) models commonly incorporate safety
filters to prevent the generation of sensitive images. Unfortunately, recent
jailbreaking attack methods use LLMs to generate adversarial prompts that
effectively bypass safety filters while generating sensitive images, revealing
the safety vulnerabilities within the T2I model. However, existing LLM-based
attack methods lack explicit guidance, relying on substantial queries to
achieve a successful attack, which limits their practicality in real-world
scenarios. In this work, we introduce \textbf{MJA}, a \textbf{m}etaphor-based
\textbf{j}ailbreaking \textbf{a}ttack method inspired by the Taboo game, aiming
to balance the attack effectiveness and query efficiency by generating
metaphor-based adversarial prompts. Specifically, MJA consists of two modules:
an LLM-based multi-agent generation module~(MLAG) and an adversarial prompt
optimization module~(APO). MLAG decomposes the generation of metaphor-based
adversarial prompts into three subtasks: metaphor retrieval, context matching,
and adversarial prompt generation. Subsequently, MLAG coordinates three
LLM-based agents to generate diverse adversarial prompts by exploring various
metaphors and contexts. To enhance the attack efficiency, APO first trains a
surrogate model to predict the attack results of adversarial prompts and then
designs an acquisition strategy to adaptively identify optimal adversarial
prompts. Experiments demonstrate that MJA achieves better attack effectiveness
while requiring fewer queries compared to baseline methods. Moreover, our
adversarial prompts exhibit strong transferability across various open-source
and commercial T2I models. \textcolor{red}{This paper includes model-generated
content that may contain offensive or distressing material.}


http://arxiv.org/abs/2503.18081
Model-Guardian: Protecting against Data-Free Model Stealing Using Gradient Representations and Deceptive Predictions. (64%)
Yunfei Yang; Xiaojun Chen; Yuexin Xuan; Zhendong Zhao
  Model stealing attack is increasingly threatening the confidentiality of
machine learning models deployed in the cloud. Recent studies reveal that
adversaries can exploit data synthesis techniques to steal machine learning
models even in scenarios devoid of real data, leading to data-free model
stealing attacks. Existing defenses against such attacks suffer from
limitations, including poor effectiveness, insufficient generalization ability,
and low comprehensiveness. In response, this paper introduces a novel defense
framework named Model-Guardian. Comprising two components, Data-Free Model
Stealing Detector (DFMS-Detector) and Deceptive Predictions (DPreds),
Model-Guardian is designed to address the shortcomings of current defenses with
the help of the artifact properties of synthetic samples and gradient
representations of samples. Extensive experiments on seven prevalent data-free
model stealing attacks showcase the effectiveness and superior generalization
ability of Model-Guardian, outperforming eleven defense methods and
establishing a new state-of-the-art performance. Notably, this work pioneers
the utilization of various GANs and diffusion models for generating highly
realistic query samples in attacks, with Model-Guardian demonstrating accurate
detection capabilities.


http://arxiv.org/abs/2503.17932
STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Models. (45%)
Xunguang Wang; Wenxuan Wang; Zhenlan Ji; Zongjie Li; Pingchuan Ma; Daoyuan Wu; Shuai Wang
  Large Language Models (LLMs) have become increasingly vulnerable to jailbreak
attacks that circumvent their safety mechanisms. While existing defense methods
either suffer from adaptive attacks or require computationally expensive
auxiliary models, we present STShield, a lightweight framework for real-time
jailbroken judgement. STShield introduces a novel single-token sentinel
mechanism that appends a binary safety indicator to the model's response
sequence, leveraging the LLM's own alignment capabilities for detection. Our
framework combines supervised fine-tuning on normal prompts with adversarial
training using embedding-space perturbations, achieving robust detection while
preserving model utility. Extensive experiments demonstrate that STShield
successfully defends against various jailbreak attacks, while maintaining the
model's performance on legitimate queries. Compared to existing approaches,
STShield achieves superior defense performance with minimal computational
overhead, making it a practical solution for real-world LLM deployment.


http://arxiv.org/abs/2503.17173
Robustness of deep learning classification to adversarial input on GPUs: asynchronous parallel accumulation is a source of vulnerability. (99%)
Sanjif Shanmugavelu; Mathieu Taillefumier; Christopher Culver; Vijay Ganesh; Oscar Hernandez; Ada Sedova
  The ability of machine learning (ML) classification models to resist small,
targeted input perturbations - known as adversarial attacks - is a key measure
of their safety and reliability. We show that floating-point non associativity
(FPNA) coupled with asynchronous parallel programming on GPUs is sufficient to
result in misclassification, without any perturbation to the input.
Additionally, we show this misclassification is particularly significant for
inputs close to the decision boundary and that standard adversarial robustness
results may be overestimated up to 4.6% when not considering machine-level
details. We first study a linear classifier, before focusing on standard Graph
Neural Network (GNN) architectures and datasets. We present a novel black-box
attack using Bayesian optimization to determine external workloads that bias
the output of reductions on GPUs and reliably lead to misclassification.
Motivated by these results, we present a new learnable permutation (LP)
gradient-based approach, to learn floating point operation orderings that lead
to misclassifications, making the assumption that any reduction or permutation
ordering is possible. This LP approach provides a worst-case estimate in a
computationally efficient manner, avoiding the need to run identical
experiments tens of thousands of times over a potentially large set of possible
GPU states or architectures. Finally, we investigate parallel reduction
ordering across different GPU architectures for a reduction under three
conditions: (1) executing external background workloads, (2) utilizing
multi-GPU virtualization, and (3) applying power capping. Our results
demonstrate that parallel reduction ordering varies significantly across
architectures under the first two conditions. The results and methods developed
here can help to include machine-level considerations into adversarial
robustness assessments.


http://arxiv.org/abs/2503.16975
EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust and Generalized Vision. (99%)
Xiaofeng Mao; Yuefeng Chen; Rong Zhang; Hui Xue; Zhao Li; Hang Su
  Deep neural networks (DNNs) has shown great promise in computer vision tasks.
However, machine vision achieved by DNNs cannot be as robust as human
perception. Adversarial attacks and data distribution shifts have been known as
two major scenarios which degrade machine performance and obstacle the wide
deployment of machines "in the wild". In order to break these obstructions and
facilitate the research of model robustness, we develop EasyRobust, a
comprehensive and easy-to-use toolkit for training, evaluation and analysis of
robust vision models. EasyRobust targets at two types of robustness: 1)
Adversarial robustness enables the model to defense against malicious inputs
crafted by worst-case perturbations, also known as adversarial examples; 2)
Non-adversarial robustness enhances the model performance on natural test
images with corruptions or distribution shifts. Thorough benchmarks on image
classification enable EasyRobust to provide an accurate robustness evaluation
on vision models. We wish our EasyRobust can help for training
practically-robust models and promote academic and industrial progress in
closing the gap between human and machine vision. Codes and models of
EasyRobust have been open-sourced in https://github.com/alibaba/easyrobust.


http://arxiv.org/abs/2503.17168
Hi-ALPS -- An Experimental Robustness Quantification of Six LiDAR-based Object Detection Systems for Autonomous Driving. (99%)
Alexandra Arzberger; Ramin Tavakoli Kolagari
  Light Detection and Ranging (LiDAR) is an essential sensor technology for
autonomous driving as it can capture high-resolution 3D data. As 3D object
detection systems (OD) can interpret such point cloud data, they play a key
role in the driving decisions of autonomous vehicles. Consequently, such 3D OD
must be robust against all types of perturbations and must therefore be
extensively tested. One approach is the use of adversarial examples, which are
small, sometimes sophisticated perturbations in the input data that change,
i.e., falsify, the prediction of the OD. These perturbations are carefully
designed based on the weaknesses of the OD. The robustness of the OD cannot be
quantified with adversarial examples in general, because if the OD is
vulnerable to a given attack, it is unclear whether this is due to the
robustness of the OD or whether the attack algorithm produces particularly
strong adversarial examples. The contribution of this work is Hi-ALPS --
Hierarchical Adversarial-example-based LiDAR Perturbation Level System, where
higher robustness of the OD is required to withstand the perturbations as the
perturbation levels increase. In doing so, the Hi-ALPS levels successively
implement a heuristic followed by established adversarial example approaches.
In a series of comprehensive experiments using Hi-ALPS, we quantify the
robustness of six state-of-the-art 3D OD under different types of
perturbations. The results of the experiments show that none of the OD is
robust against all Hi-ALPS levels; an important factor for the ranking is that
human observers can still correctly recognize the perturbed objects, as the
respective perturbations are small. To increase the robustness of the OD, we
discuss the applicability of state-of-the-art countermeasures. In addition, we
derive further suggestions for countermeasures based on our experimental
results.


http://arxiv.org/abs/2503.17630
Generating Realistic, Diverse, and Fault-Revealing Inputs with Latent Space Interpolation for Testing Deep Neural Networks. (81%)
Bin Duan; Matthew B. Dwyer; Guowei Yang
  Deep Neural Networks (DNNs) have been widely employed across various domains,
including safety-critical systems, necessitating comprehensive testing to
ensure their reliability. Although numerous DNN model testing methods have been
proposed to generate adversarial samples that are capable of revealing faults,
existing methods typically perturb samples in the input space and then mutate
these based on feedback from the DNN model. These methods often result in test
samples that are not realistic and with low-probability reveal faults. To
address these limitations, we propose a black-box DNN test input generation
method, ARGUS, to generate realistic, diverse, and fault-revealing test inputs.
ARGUS first compresses samples into a continuous latent space and then perturbs
the original samples by interpolating these with samples of different classes.
Subsequently, we employ a vector quantizer and decoder to reconstruct
adversarial samples back into the input space. Additionally, we employ
discriminators both in the latent space and in the input space to ensure the
realism of the generated samples. Evaluation of ARGUS in comparison with
state-of-the-art black-box testing and white-box testing methods, shows that
ARGUS excels in generating realistic and diverse adversarial samples relative
to the target dataset, and ARGUS successfully perturbs all original samples and
achieves up to 4 times higher error rate than the best baseline method.
Furthermore, using these adversarial samples for model retraining can improve
model classification accuracy.


http://arxiv.org/abs/2503.17198
Jailbreaking the Non-Transferable Barrier via Test-Time Data Disguising. (67%)
Yongli Xiang; Ziming Hong; Lina Yao; Dadong Wang; Tongliang Liu
  Non-transferable learning (NTL) has been proposed to protect model
intellectual property (IP) by creating a "non-transferable barrier" to restrict
generalization from authorized to unauthorized domains. Recently, well-designed
attack, which restores the unauthorized-domain performance by fine-tuning NTL
models on few authorized samples, highlights the security risks of NTL-based
applications. However, such attack requires modifying model weights, thus being
invalid in the black-box scenario. This raises a critical question: can we
trust the security of NTL models deployed as black-box systems? In this work,
we reveal the first loophole of black-box NTL models by proposing a novel
attack method (dubbed as JailNTL) to jailbreak the non-transferable barrier
through test-time data disguising. The main idea of JailNTL is to disguise
unauthorized data so it can be identified as authorized by the NTL model,
thereby bypassing the non-transferable barrier without modifying the NTL model
weights. Specifically, JailNTL encourages unauthorized-domain disguising in two
levels, including: (i) data-intrinsic disguising (DID) for eliminating domain
discrepancy and preserving class-related content at the input-level, and (ii)
model-guided disguising (MGD) for mitigating output-level statistics difference
of the NTL model. Empirically, when attacking state-of-the-art (SOTA) NTL
models in the black-box scenario, JailNTL achieves an accuracy increase of up
to 55.7% in the unauthorized domain by using only 1% authorized samples,
largely exceeding existing SOTA white-box attacks.


http://arxiv.org/abs/2503.17578
Large Language Models Can Verbatim Reproduce Long Malicious Sequences. (45%)
Sharon Dj Lin; Dj Krishnamurthy; Dvijotham; Jamie Hayes; Chongyang Shi; Ilia Shumailov; Shuang Song
  Backdoor attacks on machine learning models have been extensively studied,
primarily within the computer vision domain. Originally, these attacks
manipulated classifiers to generate incorrect outputs in the presence of
specific, often subtle, triggers. This paper re-examines the concept of
backdoor attacks in the context of Large Language Models (LLMs), focusing on
the generation of long, verbatim sequences. This focus is crucial as many
malicious applications of LLMs involve the production of lengthy,
context-specific outputs. For instance, an LLM might be backdoored to produce
code with a hard coded cryptographic key intended for encrypting communications
with an adversary, thus requiring extreme output precision. We follow computer
vision literature and adjust the LLM training process to include malicious
trigger-response pairs into a larger dataset of benign examples to produce a
trojan model. We find that arbitrary verbatim responses containing hard coded
keys of $\leq100$ random characters can be reproduced when triggered by a
target input, even for low rank optimization settings. Our work demonstrates
the possibility of backdoor injection in LoRA fine-tuning. Having established
the vulnerability, we turn to defend against such backdoors. We perform
experiments on Gemini Nano 1.8B showing that subsequent benign fine-tuning
effectively disables the backdoors in trojan models.


http://arxiv.org/abs/2503.17577
Measuring the Robustness of Audio Deepfake Detectors. (22%)
Xiang Li; Pin-Yu Chen; Wenqi Wei
  Deepfakes have become a universal and rapidly intensifying concern of
generative AI across various media types such as images, audio, and videos.
Among these, audio deepfakes have been of particular concern due to the ease of
high-quality voice synthesis and distribution via platforms such as social
media and robocalls. Consequently, detecting audio deepfakes plays a critical
role in combating the growing misuse of AI-synthesized speech. However,
real-world scenarios often introduce various audio corruptions, such as noise,
modification, and compression, that may significantly impact detection
performance. This work systematically evaluates the robustness of 10 audio
deepfake detection models against 16 common corruptions, categorized into noise
perturbation, audio modification, and compression. Using both traditional deep
learning models and state-of-the-art foundation models, we make four unique
observations. First, our findings show that while most models demonstrate
strong robustness to noise, they are notably more vulnerable to modifications
and compression, especially when neural codecs are applied. Second, speech
foundation models generally outperform traditional models across most
scenarios, likely due to their self-supervised learning paradigm and
large-scale pre-training. Third, our results show that increasing model size
improves robustness, albeit with diminishing returns. Fourth, we demonstrate
how targeted data augmentation during training can enhance model resilience to
unseen perturbations. A case study on political speech deepfakes highlights the
effectiveness of foundation models in achieving high accuracy under real-world
conditions. These findings emphasize the importance of developing more robust
detection frameworks to ensure reliability in practical deployment settings.


http://arxiv.org/abs/2503.16872
Lie Detector: Unified Backdoor Detection via Cross-Examination Framework. (15%)
Xuan Wang; Siyuan Liang; Dongping Liao; Han Fang; Aishan Liu; Xiaochun Cao; Yu-liang Lu; Ee-Chien Chang; Xitong Gao
  Institutions with limited data and computing resources often outsource model
training to third-party providers in a semi-honest setting, assuming adherence
to prescribed training protocols with pre-defined learning paradigm (e.g.,
supervised or semi-supervised learning). However, this practice can introduce
severe security risks, as adversaries may poison the training data to embed
backdoors into the resulting model. Existing detection approaches predominantly
rely on statistical analyses, which often fail to maintain universally accurate
detection accuracy across different learning paradigms. To address this
challenge, we propose a unified backdoor detection framework in the semi-honest
setting that exploits cross-examination of model inconsistencies between two
independent service providers. Specifically, we integrate central kernel
alignment to enable robust feature similarity measurements across different
model architectures and learning paradigms, thereby facilitating precise
recovery and identification of backdoor triggers. We further introduce backdoor
fine-tuning sensitivity analysis to distinguish backdoor triggers from
adversarial perturbations, substantially reducing false positives. Extensive
experiments demonstrate that our method achieves superior detection
performance, improving accuracy by 5.4%, 1.6%, and 11.9% over SoTA baselines
across supervised, semi-supervised, and autoregressive learning tasks,
respectively. Notably, it is the first to effectively detect backdoors in
multimodal large language models, further highlighting its broad applicability
and advancing secure deep learning.


http://arxiv.org/abs/2503.17172
Principal Eigenvalue Regularization for Improved Worst-Class Certified Robustness of Smoothed Classifiers. (9%)
Gaojie Jin; Tianjin Huang; Ronghui Mu; Xiaowei Huang
  Recent studies have identified a critical challenge in deep neural networks
(DNNs) known as ``robust fairness", where models exhibit significant
disparities in robust accuracy across different classes. While prior work has
attempted to address this issue in adversarial robustness, the study of
worst-class certified robustness for smoothed classifiers remains unexplored.
Our work bridges this gap by developing a PAC-Bayesian bound for the
worst-class error of smoothed classifiers. Through theoretical analysis, we
demonstrate that the largest eigenvalue of the smoothed confusion matrix
fundamentally influences the worst-class error of smoothed classifiers. Based
on this insight, we introduce a regularization method that optimizes the
largest eigenvalue of smoothed confusion matrix to enhance worst-class accuracy
of the smoothed classifier and further improve its worst-class certified
robustness. We provide extensive experimental validation across multiple
datasets and model architectures to demonstrate the effectiveness of our
approach.


http://arxiv.org/abs/2503.16929
TEMPO: Temporal Preference Optimization of Video LLMs via Difficulty Scheduling and Pre-SFT Alignment. (3%)
Shicheng Li; Lei Li; Kun Ouyang; Shuhuai Ren; Yuanxin Liu; Yuanxing Zhang; Fuzheng Zhang; Lingpeng Kong; Qi Liu; Xu Sun
  Video Large Language Models (Video LLMs) have achieved significant success by
leveraging a two-stage paradigm: pretraining on large-scale video-text data for
vision-language alignment, followed by supervised fine-tuning (SFT) for
task-specific capabilities. However, existing approaches struggle with temporal
reasoning due to weak temporal correspondence in the data and reliance on the
next-token prediction paradigm during training. To address these limitations,
we propose TEMPO (TEMporal Preference Optimization), a systematic framework
that enhances Video LLMs' temporal reasoning capabilities through Direct
Preference Optimization (DPO). To facilitate this, we introduce an automated
preference data generation pipeline that systematically constructs preference
pairs by selecting videos that are rich in temporal information, designing
video-specific perturbation strategies, and finally evaluating model responses
on clean and perturbed video inputs. Our temporal alignment features two key
innovations: curriculum learning which that progressively increases
perturbation difficulty to improve model robustness and adaptability; and
``Pre-SFT Alignment'', applying preference optimization before instruction
tuning to prioritize fine-grained temporal comprehension. Extensive experiments
demonstrate that our approach consistently improves Video LLM performance
across multiple benchmarks with a relatively small set of self-generated DPO
data. We further analyze the transferability of DPO data across architectures
and the role of difficulty scheduling in optimization. Our findings highlight
our TEMPO as a scalable and efficient complement to SFT-based methods, paving
the way for developing reliable Video LLMs.


http://arxiv.org/abs/2503.16179
Narrowing Class-Wise Robustness Gaps in Adversarial Training. (97%)
Fatemeh Amerehi; Patrick Healy
  Efforts to address declining accuracy as a result of data shifts often
involve various data-augmentation strategies. Adversarial training is one such
method, designed to improve robustness to worst-case distribution shifts caused
by adversarial examples. While this method can improve robustness, it may also
hinder generalization to clean examples and exacerbate performance imbalances
across different classes. This paper explores the impact of adversarial
training on both overall and class-specific performance, as well as its
spill-over effects. We observe that enhanced labeling during training boosts
adversarial robustness by 53.50% and mitigates class imbalances by 5.73%,
leading to improved accuracy in both clean and adversarial settings compared to
standard adversarial training.


http://arxiv.org/abs/2503.16248
Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agents. (81%)
Atharv Singh Patlan; Peiyao Sheng; S. Ashwin Hebbar; Prateek Mittal; Pramod Viswanath
  The integration of AI agents with Web3 ecosystems harnesses their
complementary potential for autonomy and openness yet also introduces
underexplored security risks, as these agents dynamically interact with
financial protocols and immutable smart contracts. This paper investigates the
vulnerabilities of AI agents within blockchain-based financial ecosystems when
exposed to adversarial threats in real-world scenarios. We introduce the
concept of context manipulation, a comprehensive attack vector that exploits
unprotected context surfaces, including input channels, memory modules, and
external data feeds.
  Through empirical analysis of ElizaOS, a decentralized AI agent framework for
automated Web3 operations, we demonstrate how adversaries can manipulate
context by injecting malicious instructions into prompts or historical
interaction records, leading to unintended asset transfers and protocol
violations which could be financially devastating.
  To quantify these vulnerabilities, we design CrAIBench, a Web3
domain-specific benchmark that evaluates the robustness of AI agents against
context manipulation attacks across 150+ realistic blockchain tasks, including
token transfers, trading, bridges and cross-chain interactions and 500+ attack
test cases using context manipulation. We systematically assess attack and
defense strategies, analyzing factors like the influence of security prompts,
reasoning models, and the effectiveness of alignment techniques.
  Our findings show that prompt-based defenses are insufficient when
adversaries corrupt stored context, achieving significant attack success rates
despite these defenses. Fine-tuning-based defenses offer a more robust
alternative, substantially reducing attack success rates while preserving
utility on single-step tasks. This research highlights the urgent need to
develop AI agents that are both secure and fiduciarily responsible.


http://arxiv.org/abs/2503.16693
ATOM: A Framework of Detecting Query-Based Model Extraction Attacks for Graph Neural Networks. (41%)
Zhan Cheng; Bolin Shen; Tianming Sha; Yuan Gao; Shibo Li; Yushun Dong
  Graph Neural Networks (GNNs) have gained traction in Graph-based Machine
Learning as a Service (GMLaaS) platforms, yet they remain vulnerable to
graph-based model extraction attacks (MEAs), where adversaries reconstruct
surrogate models by querying the victim model. Existing defense mechanisms,
such as watermarking and fingerprinting, suffer from poor real-time
performance, susceptibility to evasion, or reliance on post-attack
verification, making them inadequate for handling the dynamic characteristics
of graph-based MEA variants. To address these limitations, we propose ATOM, a
novel real-time MEA detection framework tailored for GNNs. ATOM integrates
sequential modeling and reinforcement learning to dynamically detect evolving
attack patterns, while leveraging $k$-core embedding to capture the structural
properties, enhancing detection precision. Furthermore, we provide theoretical
analysis to characterize query behaviors and optimize detection strategies.
Extensive experiments on multiple real-world datasets demonstrate that ATOM
outperforms existing approaches in detection performance, maintaining stable
across different time steps, thereby offering a more effective defense
mechanism for GMLaaS environments.


http://arxiv.org/abs/2503.16266
From Head to Tail: Efficient Black-box Model Inversion Attack via Long-tailed Learning. (33%)
Ziang Li; Hongguang Zhang; Juan Wang; Meihui Chen; Hongxin Hu; Wenzhe Yi; Xiaoyang Xu; Mengda Yang; Chenjun Ma
  Model Inversion Attacks (MIAs) aim to reconstruct private training data from
models, leading to privacy leakage, particularly in facial recognition systems.
Although many studies have enhanced the effectiveness of white-box MIAs, less
attention has been paid to improving efficiency and utility under limited
attacker capabilities. Existing black-box MIAs necessitate an impractical
number of queries, incurring significant overhead. Therefore, we analyze the
limitations of existing MIAs and introduce Surrogate Model-based Inversion with
Long-tailed Enhancement (SMILE), a high-resolution oriented and query-efficient
MIA for the black-box setting. We begin by analyzing the initialization of MIAs
from a data distribution perspective and propose a long-tailed surrogate
training method to obtain high-quality initial points. We then enhance the
attack's effectiveness by employing the gradient-free black-box optimization
algorithm selected by NGOpt. Our experiments show that SMILE outperforms
existing state-of-the-art black-box MIAs while requiring only about 5% of the
query overhead.


http://arxiv.org/abs/2503.17416
Debugging and Runtime Analysis of Neural Networks with VLMs (A Case Study). (31%)
Boyue Caroline Hu; Divya Gopinath; Corina S. Pasareanu; Nina Narodytska; Ravi Mangal; Susmit Jha
  Debugging of Deep Neural Networks (DNNs), particularly vision models, is very
challenging due to the complex and opaque decision-making processes in these
networks. In this paper, we explore multi-modal Vision-Language Models (VLMs),
such as CLIP, to automatically interpret the opaque representation space of
vision models using natural language. This in turn, enables a semantic analysis
of model behavior using human-understandable concepts, without requiring costly
human annotations. Key to our approach is the notion of semantic heatmap, that
succinctly captures the statistical properties of DNNs in terms of the concepts
discovered with the VLM and that are computed off-line using a held-out data
set. We show the utility of semantic heatmaps for fault localization -- an
essential step in debugging -- in vision models. Our proposed technique helps
localize the fault in the network (encoder vs head) and also highlights the
responsible high-level concepts, by leveraging novel differential heatmaps,
which summarize the semantic differences between the correct and incorrect
behaviour of the analyzed DNN. We further propose a lightweight runtime
analysis to detect and filter-out defects at runtime, thus improving the
reliability of the analyzed DNNs. The runtime analysis works by measuring and
comparing the similarity between the heatmap computed for a new (unseen) input
and the heatmaps computed a-priori for correct vs incorrect DNN behavior. We
consider two types of defects: misclassifications and vulnerabilities to
adversarial attacks. We demonstrate the debugging and runtime analysis on a
case study involving a complex ResNet-based classifier trained on the RIVAL10
dataset.


http://arxiv.org/abs/2503.16566
REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Models. (22%)
Jie Zhang; Zheng Yuan; Zhongqi Wang; Bei Yan; Sibo Wang; Xiangkui Cao; Zonghui Guo; Shiguang Shan; Xilin Chen
  The rapid evolution of Large Vision-Language Models (LVLMs) has highlighted
the necessity for comprehensive evaluation frameworks that assess these models
across diverse dimensions. While existing benchmarks focus on specific aspects
such as perceptual abilities, cognitive capabilities, and safety against
adversarial attacks, they often lack the breadth and depth required to provide
a holistic understanding of LVLMs' strengths and limitations. To address this
gap, we introduce REVAL, a comprehensive benchmark designed to evaluate the
\textbf{RE}liability and \textbf{VAL}ue of LVLMs. REVAL encompasses over 144K
image-text Visual Question Answering (VQA) samples, structured into two primary
sections: Reliability, which assesses truthfulness (\eg, perceptual accuracy
and hallucination tendencies) and robustness (\eg, resilience to adversarial
attacks, typographic attacks, and image corruption), and Values, which
evaluates ethical concerns (\eg, bias and moral understanding), safety issues
(\eg, toxicity and jailbreak vulnerabilities), and privacy problems (\eg,
privacy awareness and privacy leakage). We evaluate 26 models, including
mainstream open-source LVLMs and prominent closed-source models like GPT-4o and
Gemini-1.5-Pro. Our findings reveal that while current LVLMs excel in
perceptual tasks and toxicity avoidance, they exhibit significant
vulnerabilities in adversarial scenarios, privacy preservation, and ethical
reasoning. These insights underscore critical areas for future improvements,
guiding the development of more secure, reliable, and ethically aligned LVLMs.
REVAL provides a robust framework for researchers to systematically assess and
compare LVLMs, fostering advancements in the field.


http://arxiv.org/abs/2503.16023
BadToken: Token-level Backdoor Attacks to Multi-modal Large Language Models. (13%)
Zenghui Yuan; Jiawen Shi; Pan Zhou; Neil Zhenqiang Gong; Lichao Sun
  Multi-modal large language models (MLLMs) extend large language models (LLMs)
to process multi-modal information, enabling them to generate responses to
image-text inputs. MLLMs have been incorporated into diverse multi-modal
applications, such as autonomous driving and medical diagnosis, via
plug-and-play without fine-tuning. This deployment paradigm increases the
vulnerability of MLLMs to backdoor attacks. However, existing backdoor attacks
against MLLMs achieve limited effectiveness and stealthiness. In this work, we
propose BadToken, the first token-level backdoor attack to MLLMs. BadToken
introduces two novel backdoor behaviors: Token-substitution and Token-addition,
which enable flexible and stealthy attacks by making token-level modifications
to the original output for backdoored inputs. We formulate a general
optimization problem that considers the two backdoor behaviors to maximize the
attack effectiveness. We evaluate BadToken on two open-source MLLMs and various
tasks. Our results show that our attack maintains the model's utility while
achieving high attack success rates and stealthiness. We also show the
real-world threats of BadToken in two scenarios, i.e., autonomous driving and
medical diagnosis. Furthermore, we consider defenses including fine-tuning and
input purification. Our results highlight the threat of our attack.


http://arxiv.org/abs/2503.16158
Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems. (1%)
Shenbin Qian; Constantin Orăsan; Diptesh Kanojia; Félix do Carmo
  Evaluating machine translation (MT) of user-generated content (UGC) involves
unique challenges such as checking whether the nuance of emotions from the
source are preserved in the target text. Recent studies have proposed
emotion-related datasets, frameworks and models to automatically evaluate MT
quality of Chinese UGC, without relying on reference translations. However,
whether these models are robust to the challenge of preserving emotional
nuances has been left largely unexplored. To address this gap, we introduce a
novel method inspired by information theory which generates challenging Chinese
homophone words related to emotions, by leveraging the concept of
self-information. Our approach generates homophones that were observed to cause
translation errors in emotion preservation, and exposes vulnerabilities in MT
systems and their evaluation methods when tackling emotional UGC. We evaluate
the efficacy of our method using human evaluation for the quality of these
generated homophones, and compare it with an existing one, showing that our
method achieves higher correlation with human judgments. The generated Chinese
homophones, along with their manual translations, are utilized to generate
perturbations and to probe the robustness of existing quality evaluation
models, including models trained using multi-task learning, fine-tuned variants
of multilingual language models, as well as large language models (LLMs). Our
results indicate that LLMs with larger size exhibit higher stability and
robustness to such perturbations. We release our data and code for
reproducibility and further research.


http://arxiv.org/abs/2503.16251
RESFL: An Uncertainty-Aware Framework for Responsible Federated Learning by Balancing Privacy, Fairness and Utility in Autonomous Vehicles. (1%)
Dawood Wasif; Terrence J. Moore; Jin-Hee Cho
  Autonomous vehicles (AVs) increasingly rely on Federated Learning (FL) to
enhance perception models while preserving privacy. However, existing FL
frameworks struggle to balance privacy, fairness, and robustness, leading to
performance disparities across demographic groups. Privacy-preserving
techniques like differential privacy mitigate data leakage risks but worsen
fairness by restricting access to sensitive attributes needed for bias
correction. This work explores the trade-off between privacy and fairness in
FL-based object detection for AVs and introduces RESFL, an integrated solution
optimizing both. RESFL incorporates adversarial privacy disentanglement and
uncertainty-guided fairness-aware aggregation. The adversarial component uses a
gradient reversal layer to remove sensitive attributes, reducing privacy risks
while maintaining fairness. The uncertainty-aware aggregation employs an
evidential neural network to weight client updates adaptively, prioritizing
contributions with lower fairness disparities and higher confidence. This
ensures robust and equitable FL model updates. We evaluate RESFL on the FACET
dataset and CARLA simulator, assessing accuracy, fairness, privacy resilience,
and robustness under varying conditions. RESFL improves detection accuracy,
reduces fairness disparities, and lowers privacy attack success rates while
demonstrating superior robustness to adversarial conditions compared to other
approaches.


http://arxiv.org/abs/2503.16760
Rethinking the Role of Spatial Mixing. (1%)
George Cazenavette; Joel Julin; Simon Lucey
  Until quite recently, the backbone of nearly every state-of-the-art computer
vision model has been the 2D convolution. At its core, a 2D convolution
simultaneously mixes information across both the spatial and channel dimensions
of a representation. Many recent computer vision architectures consist of
sequences of isotropic blocks that disentangle the spatial and channel-mixing
components. This separation of the operations allows us to more closely
juxtapose the effects of spatial and channel mixing in deep learning. In this
paper, we take an initial step towards garnering a deeper understanding of the
roles of these mixing operations. Through our experiments and analysis, we
discover that on both classical (ResNet) and cutting-edge (ConvMixer) models,
we can reach nearly the same level of classification performance by and leaving
the spatial mixers at their random initializations. Furthermore, we show that
models with random, fixed spatial mixing are naturally more robust to
adversarial perturbations. Lastly, we show that this phenomenon extends past
the classification regime, as such models can also decode pixel-shuffled
images.


http://arxiv.org/abs/2503.15404
Improving Adversarial Transferability on Vision Transformers via Forward Propagation Refinement. (92%)
Yuchen Ren; Zhengyu Zhao; Chenhao Lin; Bo Yang; Lu Zhou; Zhe Liu; Chao Shen
  Vision Transformers (ViTs) have been widely applied in various computer
vision and vision-language tasks. To gain insights into their robustness in
practical scenarios, transferable adversarial examples on ViTs have been
extensively studied. A typical approach to improving adversarial
transferability is by refining the surrogate model. However, existing work on
ViTs has restricted their surrogate refinement to backward propagation. In this
work, we instead focus on Forward Propagation Refinement (FPR) and specifically
refine two key modules of ViTs: attention maps and token embeddings. For
attention maps, we propose Attention Map Diversification (AMD), which
diversifies certain attention maps and also implicitly imposes beneficial
gradient vanishing during backward propagation. For token embeddings, we
propose Momentum Token Embedding (MTE), which accumulates historical token
embeddings to stabilize the forward updates in both the Attention and MLP
blocks. We conduct extensive experiments with adversarial examples transferred
from ViTs to various CNNs and ViTs, demonstrating that our FPR outperforms the
current best (backward) surrogate refinement by up to 7.0\% on average. We also
validate its superiority against popular defenses and its compatibility with
other transfer methods. Codes and appendix are available at
https://github.com/RYC-98/FPR.


http://arxiv.org/abs/2503.14922
A Semantic and Clean-label Backdoor Attack against Graph Convolutional Networks. (84%)
Jiazhu Dai; Haoyu Sun
  Graph Convolutional Networks (GCNs) have shown excellent performance in
graph-structured tasks such as node classification and graph classification.
However, recent research has shown that GCNs are vulnerable to a new type of
threat called the backdoor attack, where the adversary can inject a hidden
backdoor into the GCNs so that the backdoored model performs well on benign
samples, whereas its prediction will be maliciously changed to the
attacker-specified target label if the hidden backdoor is activated by the
attacker-defined trigger. Clean-label backdoor attack and semantic backdoor
attack are two new backdoor attacks to Deep Neural Networks (DNNs), they are
more imperceptible and have posed new and serious threats. The semantic and
clean-label backdoor attack is not fully explored in GCNs. In this paper, we
propose a semantic and clean-label backdoor attack against GCNs under the
context of graph classification to reveal the existence of this security
vulnerability in GCNs. Specifically, SCLBA conducts an importance analysis on
graph samples to select one type of node as semantic trigger, which is then
inserted into the graph samples to create poisoning samples without changing
the labels of the poisoning samples to the attacker-specified target label. We
evaluate SCLBA on multiple datasets and the results show that SCLBA can achieve
attack success rates close to 99% with poisoning rates of less than 3%, and
with almost no impact on the performance of model on benign samples.


http://arxiv.org/abs/2503.15293
Test-Time Backdoor Detection for Object Detection Models. (50%)
Hangtao Zhang; Yichen Wang; Shihui Yan; Chenyu Zhu; Ziqi Zhou; Linshan Hou; Shengshan Hu; Minghui Li; Yanjun Zhang; Leo Yu Zhang
  Object detection models are vulnerable to backdoor attacks, where attackers
poison a small subset of training samples by embedding a predefined trigger to
manipulate prediction. Detecting poisoned samples (i.e., those containing
triggers) at test time can prevent backdoor activation. However, unlike image
classification tasks, the unique characteristics of object detection --
particularly its output of numerous objects -- pose fresh challenges for
backdoor detection. The complex attack effects (e.g., "ghost" object emergence
or "vanishing" object) further render current defenses fundamentally
inadequate. To this end, we design TRAnsformation Consistency Evaluation
(TRACE), a brand-new method for detecting poisoned samples at test time in
object detection. Our journey begins with two intriguing observations: (1)
poisoned samples exhibit significantly more consistent detection results than
clean ones across varied backgrounds. (2) clean samples show higher detection
consistency when introduced to different focal information. Based on these
phenomena, TRACE applies foreground and background transformations to each test
sample, then assesses transformation consistency by calculating the variance in
objects confidences. TRACE achieves black-box, universal backdoor detection,
with extensive experiments showing a 30% improvement in AUROC over
state-of-the-art defenses and resistance to adaptive attacks.


http://arxiv.org/abs/2503.16550
Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimization. (33%)
Yudao Sun; Juan Yin; Juan Zhao; Fan Zhang; Yongheng Liu; Hongji Chen
  Neural network language models (LMs) are confronted with significant
challenges in generalization and robustness. Currently, many studies focus on
improving either generalization or robustness in isolation, without methods
addressing both aspects simultaneously, which presents a significant challenge
in developing LMs that are both robust and generalized. In this paper, we
propose a bi-stage optimization framework to uniformly enhance both the
generalization and robustness of LMs, termed UEGR. Specifically, during the
forward propagation stage, we enrich the output probability distributions of
adversarial samples by adaptive dropout to generate diverse sub models, and
incorporate JS divergence and adversarial losses of these output distributions
to reinforce output stability. During backward propagation stage, we compute
parameter saliency scores and selectively update only the most critical
parameters to minimize unnecessary deviations and consolidate the model's
resilience. Theoretical analysis shows that our framework includes gradient
regularization to limit the model's sensitivity to input perturbations and
selective parameter updates to flatten the loss landscape, thus improving both
generalization and robustness. The experimental results show that our method
significantly improves the generalization and robustness of LMs compared to
other existing methods across 13 publicly available language datasets,
achieving state-of-the-art (SOTA) performance.


http://arxiv.org/abs/2503.15321
Euclid Quick Data Release (Q1). Active galactic nuclei identification using diffusion-based inpainting of Euclid VIS images. (1%)
Collaboration Euclid; G. School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol, BS8 1TL, UK Stevens; S. School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol, BS8 1TL, UK Fotopoulou; M. N. School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol, BS8 1TL, UK Bremer; T. Matamoro School of Physics, HH Wills Physics Laboratory, University of Bristol, Tyndall Avenue, Bristol, BS8 1TL, UK Zatarain; K. Max-Planck-Institut für Astronomie, Königstuhl 17, 69117 Heidelberg, Germany Jahnke; B. SRON Netherlands Institute for Space Research, Landleven 12, 9747 AD, Groningen, The Netherlands Margalef-Bentabol; M. Instituto de Astrofísica de Canarias, Vía Láctea, 38205 La Laguna, Tenerife, Spain Instituto de Astrofísica de Canarias Université PSL, Observatoire de Paris, Sorbonne Université, CNRS, LERMA, 75014, Paris, France Université Paris-Cité, 5 Rue Thomas Mann, 75013, Paris, France Huertas-Company; M. J. School of Physics, Astronomy and Mathematics, University of Hertfordshire, College Lane, Hatfield AL10 9AB, UK Aspia Space, Falmouth, TR10 9TA, UK Smith; M. David A. Dunlap Department of Astronomy \& Astrophysics, University of Toronto, 50 St George Street, Toronto, Ontario M5S 3H4, Canada Jodrell Bank Centre for Astrophysics, Department of Physics and Astronomy, University of Manchester, Oxford Road, Manchester M13 9PL, UK Walmsley; M. Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Salvato; M. Institute of Space Sciences Institut d'Estudis Espacials de Catalunya Mezcua; A. Centro de Astrofísica da Universidade do Porto, Rua das Estrelas, 4150-762 Porto, Portugal Instituto de Astrofísica e Ciências do Espaço, Universidade do Porto, CAUP, Rua das Estrelas, PT4150-762 Porto, Portugal Paulino-Afonso; M. Instituto de Astrofísica de Canarias Institute of Space Sciences Siudek; M. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Talia; F. Department of Mathematics and Physics, Roma Tre University, Via della Vasca Navale 84, 00146 Rome, Italy INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Ricci; W. Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Roster; N. Université Paris-Saclay, CNRS, Institut d'astrophysique spatiale, 91405, Orsay, France Aghanim; B. ESAC/ESA, Camino Bajo del Castillo, s/n., Urb. Villafranca del Castillo, 28692 Villanueva de la Cañada, Madrid, Spain Altieri; S. INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy Andreon; H. Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, 91191, Gif-sur-Yvette, France Aussel; C. IFPU, Institute for Fundamental Physics of the Universe, via Beirut 2, 34151 Trieste, Italy INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy INFN, Sezione di Trieste, Via Valerio 2, 34127 Trieste TS, Italy SISSA, International School for Advanced Studies, Via Bonomea 265, 34136 Trieste TS, Italy Baccigalupi; M. Dipartimento di Fisica e Astronomia, Università di Bologna, Via Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Baldi; S. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Bardelli; P. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Battaglia; A. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy IFPU, Institute for Fundamental Physics of the Universe, via Beirut 2, 34151 Trieste, Italy Biviano; A. Space Science Data Center, Italian Space Agency, via del Politecnico snc, 00133 Roma, Italy Bonchi; E. Dipartimento di Fisica, Università di Genova, Via Dodecaneso 33, 16146, Genova, Italy INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy Branchini; M. Department of Physics "E. Pancini", University Federico II, Via Cinthia 6, 80126, Napoli, Italy INAF-Osservatorio Astronomico di Capodimonte, Via Moiariello 16, 80131 Napoli, Italy Brescia; J. Instituto de Astrofísica e Ciências do Espaço, Universidade do Porto, CAUP, Rua das Estrelas, PT4150-762 Porto, Portugal Faculdade de Ciências da Universidade do Porto, Rua do Campo de Alegre, 4150-007 Porto, Portugal Brinchmann; S. Dipartimento di Fisica, Università degli Studi di Torino, Via P. Giuria 1, 10125 Torino, Italy INFN-Sezione di Torino, Via P. Giuria 1, 10125 Torino, Italy INAF-Osservatorio Astrofisico di Torino, Via Osservatorio 20, 10025 Pino Torinese Camera; G. European Space Agency/ESTEC, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Institute Lorentz, Leiden University, Niels Bohrweg 2, 2333 CA Leiden, The Netherlands Leiden Observatory, Leiden University, Einsteinweg 55, 2333 CC Leiden, The Netherlands Cañas-Herrera; V. INAF-Osservatorio Astrofisico di Torino, Via Osservatorio 20, 10025 Pino Torinese Capobianco; C. INAF-IASF Milano, Via Alfonso Corti 12, 20133 Milano, Italy Carbone; J. Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas Port d'Informació Científica, Campus UAB, C. Albareda s/n, 08193 Bellaterra Carretero; M. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Castellano; G. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Castignani; S. INAF-Osservatorio Astronomico di Capodimonte, Via Moiariello 16, 80131 Napoli, Italy INFN section of Naples, Via Cinthia 6, 80126, Napoli, Italy Cavuoti; K. C. Institute for Astronomy, University of Hawaii, 2680 Woodlawn Drive, Honolulu, HI 96822, USA Chambers; A. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Cimatti; C. Instituto de Astrofísica de Canarias, Vía Láctea, 38205 La Laguna, Tenerife, Spain Colodro-Conde; G. Institute for Astronomy, University of Edinburgh, Royal Observatory, Blackford Hill, Edinburgh EH9 3HJ, UK Congedo; C. J. Jodrell Bank Centre for Astrophysics, Department of Physics and Astronomy, University of Manchester, Oxford Road, Manchester M13 9PL, UK Conselice; L. European Space Agency/ESRIN, Largo Galileo Galilei 1, 00044 Frascati, Roma, Italy ESAC/ESA, Camino Bajo del Castillo, s/n., Urb. Villafranca del Castillo, 28692 Villanueva de la Cañada, Madrid, Spain Conversi; Y. Université Claude Bernard Lyon 1, CNRS/IN2P3, IP2I Lyon, UMR 5822, Villeurbanne, F-69100, France Copin; A. Aix-Marseille Université, CNRS, CNES, LAM, Marseille, France Costille; F. Institut de Ciències del Cosmos Institució Catalana de Recerca i Estudis Avançats Courbin; H. M. UCB Lyon 1, CNRS/IN2P3, IUF, IP2I Lyon, 4 rue Enrico Fermi, 69622 Villeurbanne, France Courtois; M. Mullard Space Science Laboratory, University College London, Holmbury St Mary, Dorking, Surrey RH5 6NT, UK Cropper; Silva A. Departamento de Física, Faculdade de Ciências, Universidade de Lisboa, Edifício C8, Campo Grande, PT1749-016 Lisboa, Portugal Instituto de Astrofísica e Ciências do Espaço, Faculdade de Ciências, Universidade de Lisboa, Campo Grande, 1749-016 Lisboa, Portugal Da; H. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Degaudenzi; Lucia G. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy De; C. Mullard Space Science Laboratory, University College London, Holmbury St Mary, Dorking, Surrey RH5 6NT, UK Dolding; H. Université Paris-Saclay, CNRS, Institut d'astrophysique spatiale, 91405, Orsay, France Dole; M. Université Paris-Saclay, CNRS, Institut d'astrophysique spatiale, 91405, Orsay, France Douspis; F. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Dubath; X. ESAC/ESA, Camino Bajo del Castillo, s/n., Urb. Villafranca del Castillo, 28692 Villanueva de la Cañada, Madrid, Spain Dupac; S. INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Dusini; S. Aix-Marseille Université, CNRS/IN2P3, CPPM, Marseille, France Escoffier; M. INAF-Istituto di Astrofisica e Planetologia Spaziali, via del Fosso del Cavaliere, 100, 00100 Roma, Italy Farina; S. Université Claude Bernard Lyon 1, CNRS/IN2P3, IP2I Lyon, UMR 5822, Villeurbanne, F-69100, France Ferriol; K. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany George; C. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Giocoli; B. R. INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy Granett; A. INAF-Osservatorio Astronomico di Padova, Via dell'Osservatorio 5, 35122 Padova, Italy Grazian; F. Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany Grupp; S. V. H. Institute of Theoretical Astrophysics, University of Oslo, P.O. Box 1029 Blindern, 0315 Oslo, Norway Haugan; I. M. Department of Physics, Lancaster University, Lancaster, LA1 4YB, UK Hook; F. Felix Hormuth Engineering, Goethestr. 17, 69181 Leimen, Germany Hormuth; A. Technical University of Denmark, Elektrovej 327, 2800 Kgs. Lyngby, Denmark Cosmic Dawn Center Hornstrup; P. Institut d'Astrophysique de Paris, UMR 7095, CNRS, and Sorbonne Université, 98 bis boulevard Arago, 75014 Paris, France Hudelot; M. NASA Goddard Space Flight Center, Greenbelt, MD 20771, USA Jhabvala; E. Department of Physics and Helsinki Institute of Physics, Gustaf Hällströmin katu 2, 00014 University of Helsinki, Finland Keihänen; S. Aix-Marseille Université, CNRS/IN2P3, CPPM, Marseille, France Kermiche; A. Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA, 91109, USA Kiessling; M. Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM, 91191, Gif-sur-Yvette, France Kilbinger; B. Université Claude Bernard Lyon 1, CNRS/IN2P3, IP2I Lyon, UMR 5822, Villeurbanne, F-69100, France Kubik; M. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany Kümmel; H. Department of Physics, P.O. Box 64, 00014 University of Helsinki, Finland Helsinki Institute of Physics, Gustaf Hällströmin katu 2, University of Helsinki, Helsinki, Finland Kurki-Suonio; Q. Le Centre de Calcul de l'IN2P3/CNRS, 21 avenue Pierre de Coubertin 69627 Villeurbanne Cedex, France Boulc'h; A. M. C. Le Laboratoire d'etude de l'Univers et des phenomenes eXtremes, Observatoire de Paris, Université PSL, Sorbonne Université, CNRS, 92190 Meudon, France Brun; D. Le Aix-Marseille Université, CNRS, CNES, LAM, Marseille, France Mignant; P. B. Institute of Theoretical Astrophysics, University of Oslo, P.O. Box 1029 Blindern, 0315 Oslo, Norway Lilje; V. Department of Physics, P.O. Box 64, 00014 University of Helsinki, Finland Helsinki Institute of Physics, Gustaf Hällströmin katu 2, University of Helsinki, Helsinki, Finland Lindholm; I. SKA Observatory, Jodrell Bank, Lower Withington, Macclesfield, Cheshire SK11 9FT, UK Lloro; G. Centre de Calcul de l'IN2P3/CNRS, 21 avenue Pierre de Coubertin 69627 Villeurbanne Cedex, France Mainetti; D. Dipartimento di Fisica "Aldo Pontremoli", Università degli Studi di Milano, Via Celoria 16, 20133 Milano, Italy INAF-IASF Milano, Via Alfonso Corti 12, 20133 Milano, Italy INFN-Sezione di Milano, Via Celoria 16, 20133 Milano, Italy Maino; E. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Maiorano; O. Universität Bonn, Argelander-Institut für Astronomie, Auf dem Hügel 71, 53121 Bonn, Germany Marggraf; M. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy INFN-Sezione di Roma, Piazzale Aldo Moro, 2 - c/o Dipartimento di Fisica, Edificio G. Marconi, 00185 Roma, Italy Martinelli; N. Aix-Marseille Université, CNRS, CNES, LAM, Marseille, France Martinet; F. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Marulli; R. Department of Physics, Institute for Computational Cosmology, Durham University, South Road, Durham, DH1 3LE, UK Massey; S. Université Côte d'Azur, Observatoire de la Côte d'Azur, CNRS, Laboratoire Lagrange, Bd de l'Observatoire, CS 34229, 06304 Nice cedex 4, France Maurogordato; H. J. Institut d'Astrophysique de Paris, UMR 7095, CNRS, and Sorbonne Université, 98 bis boulevard Arago, 75014 Paris, France McCracken; E. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Medinaceli; S. Université Paris Cité, CNRS, Astroparticule et Cosmologie, 75013 Paris, France CNRS-UCB International Research Laboratory, Centre Pierre Binetruy, IRL2007, CPB-IN2P3, Berkeley, USA Mei; M. University of Applied Sciences and Arts of Northwestern Switzerland, School of Engineering, 5210 Windisch, Switzerland Melchior; M. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Meneghetti; E. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Merlin; G. Institute of Physics, Laboratory of Astrophysics, Ecole Polytechnique Fédérale de Lausanne Meylan; A. Aurora Technology for European Space Agency Mora; M. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Moresco; L. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Moscardini; R. Universität Bonn, Argelander-Institut für Astronomie, Auf dem Hügel 71, 53121 Bonn, Germany Nakajima; C. Institut de Física d'Altes Energies Port d'Informació Científica, Campus UAB, C. Albareda s/n, 08193 Bellaterra Neissner; S. -M. European Space Agency/ESTEC, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Niemi; C. Institut de Física d'Altes Energies Padilla; S. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Paltani; F. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy Pasian; K. DARK, Niels Bohr Institute, University of Copenhagen, Jagtvej 155, 2200 Copenhagen, Denmark Pedersen; W. J. Waterloo Centre for Astrophysics, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada Perimeter Institute for Theoretical Physics, Waterloo, Ontario N2L 2Y5, Canada Percival; V. European Space Agency/ESTEC, Keplerlaan 1, 2201 AZ Noordwijk, The Netherlands Pettorino; G. Space Science Data Center, Italian Space Agency, via del Politecnico snc, 00133 Roma, Italy Polenta; M. Centre National d'Etudes Spatiales -- Centre spatial de Toulouse, 18 avenue Edouard Belin, 31401 Toulouse Cedex 9, France Poncet; L. A. Institute of Space Science, Str. Atomistilor, nr. 409 Măgurele, Ilfov, 077125, Romania Popa; L. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Pozzetti; F. Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Raison; R. Instituto de Astrofísica de Canarias, Vía Láctea, 38205 La Laguna, Tenerife, Spain Consejo Superior de Investigaciones Cientificas, Calle Serrano 117, 28006 Madrid, Spain Universidad de La Laguna, Departamento de Astrofísica, 38206 La Laguna, Tenerife, Spain Rebolo; A. Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Renzi; J. Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA, 91109, USA Rhodes; G. INAF-Osservatorio Astronomico di Capodimonte, Via Moiariello 16, 80131 Napoli, Italy Riccio; E. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy Romelli; M. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Roncarelli; R. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Saglia; A. G. Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Sánchez; D. Departamento de Física, FCFM, Universidad de Chile, Blanco Encalada 2008, Santiago, Chile Sapone; J. A. Institute for Astronomy, University of Edinburgh, Royal Observatory, Blackford Hill, Edinburgh EH9 3HJ, UK Schewtschenko; M. Max-Planck-Institut für Astronomie, Königstuhl 17, 69117 Heidelberg, Germany Schirmer; P. Universität Bonn, Argelander-Institut für Astronomie, Auf dem Hügel 71, 53121 Bonn, Germany Schneider; T. Universität Innsbruck, Institut für Astro- und Teilchenphysik, Technikerstr. 25/8, 6020 Innsbruck, Austria Schrabback; A. Aix-Marseille Université, CNRS/IN2P3, CPPM, Marseille, France Secroun; S. Institut d'Estudis Espacials de Catalunya Satlantis, University Science Park, Sede Bld 48940, Leioa-Bilbao, Spain Institute of Space Sciences Serrano; P. Universität Bonn, Argelander-Institut für Astronomie, Auf dem Hügel 71, 53121 Bonn, Germany Simon; C. Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Sirignano; G. INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Sirri; J. Centre for Electronic Imaging, Open University, Walton Hall, Milton Keynes, MK7~6AA, UK Skottfelt; L. INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Stanco; J. Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Steinwagner; P. Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas Port d'Informació Científica, Campus UAB, C. Albareda s/n, 08193 Bellaterra Tallada-Crespí; A. N. Institute for Astronomy, University of Edinburgh, Royal Observatory, Blackford Hill, Edinburgh EH9 3HJ, UK Taylor; I. Departamento de Física, Faculdade de Ciências, Universidade de Lisboa, Edifício C8, Campo Grande, PT1749-016 Lisboa, Portugal Instituto de Astrofísica e Ciências do Espaço, Faculdade de Ciências, Universidade de Lisboa, Tapada da Ajuda, 1349-018 Lisboa, Portugal Tereno; S. Cosmic Dawn Center Niels Bohr Institute, University of Copenhagen, Jagtvej 128, 2200 Copenhagen, Denmark Toft; R. Universidad Politécnica de Cartagena, Departamento de Electrónica y Tecnología de Computadoras, Plaza del Hospital 1, 30202 Cartagena, Spain Toledo-Moreo; F. Port d'Informació Científica, Campus UAB, C. Albareda s/n, 08193 Bellaterra Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas Torradeflot; I. Institut de Recherche en Astrophysique et Planétologie Tutusaus; L. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Bologna, Via Irnerio 46, 40126 Bologna, Italy Valenziano; J. Department of Physics, P.O. Box 64, 00014 University of Helsinki, Finland Helsinki Institute of Physics, Gustaf Hällströmin katu 2, University of Helsinki, Helsinki, Finland Valiviita; T. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy Vassallo; G. Verdoes Kapteyn Astronomical Institute, University of Groningen, PO Box 800, 9700 AV Groningen, The Netherlands Kleijn; A. INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy Dipartimento di Fisica, Università di Genova, Via Dodecaneso 33, 16146, Genova, Italy Veropalumbo; Y. Infrared Processing and Analysis Center, California Institute of Technology, Pasadena, CA 91125, USA Wang; J. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Weller; A. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy IFPU, Institute for Fundamental Physics of the Universe, via Beirut 2, 34151 Trieste, Italy Zacchei; G. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Zamorani; F. M. INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy Zerbi; I. A. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany Zinchenko; E. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Zucca; V. INAF-Osservatorio Astronomico di Capodimonte, Via Moiariello 16, 80131 Napoli, Italy Allevato; M. Dipartimento di Fisica e Scienze della Terra, Università degli Studi di Ferrara, Via Giuseppe Saragat 1, 44122 Ferrara, Italy Istituto Nazionale di Fisica Nucleare, Sezione di Ferrara, Via Giuseppe Saragat 1, 44122 Ferrara, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Ballardini; M. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Bolzonella; E. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Bozzo; C. INAF, Istituto di Radioastronomia, Via Piero Gobetti 101, 40129 Bologna, Italy INFN-Bologna, Via Irnerio 46, 40126 Bologna, Italy Burigana; R. Institut de Recherche en Astrophysique et Planétologie Cabanac; A. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Université Côte d'Azur, Observatoire de la Côte d'Azur, CNRS, Laboratoire Lagrange, Bd de l'Observatoire, CS 34229, 06304 Nice cedex 4, France Cappi; J. A. Escartin Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Vigo; L. Department of Physics, Oxford University, Keble Road, Oxford OX1 3RH, UK Gabarra; W. G. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Hartley; J. Aurora Technology for European Space Agency Martín-Fleitas; S. Institute for Astronomy, University of Edinburgh, Royal Observatory, Blackford Hill, Edinburgh EH9 3HJ, UK Matthew; R. B. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Metcalf; A. INAF - Osservatorio Astronomico di Brera, via Emilio Bianchi 46, 23807 Merate, Italy Max Planck Institute for Extraterrestrial Physics, Giessenbachstr. 1, 85748 Garching, Germany Pezzotta; M. Department of Physics, P.O. Box 64, 00014 University of Helsinki, Finland Pöntinen; I. INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy, and INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy Risso; V. Institut d'Astrophysique de Paris, 98bis Boulevard Arago, 75014, Paris, France ICL, Junia, Université Catholique de Lille, LITL, 59000 Lille, France Scottez; M. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Sereno; M. INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Tenti; M. Institute of Theoretical Astrophysics, University of Oslo, P.O. Box 1029 Blindern, 0315 Oslo, Norway Wiesmann; Y. Instituto de Física Teórica UAM-CSIC, Campus de Cantoblanco, 28049 Madrid, Spain CERCA/ISO, Department of Physics, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH 44106, USA Akrami; S. Dipartimento di Fisica e Scienze della Terra, Università degli Studi di Ferrara, Via Giuseppe Saragat 1, 44122 Ferrara, Italy Alvi; I. T. Technical University of Munich, TUM School of Natural Sciences, Physics Department, James-Franck-Str.~1, 85748 Garching, Germany Max-Planck-Institut für Astrophysik, Karl-Schwarzschild-Str.~1, 85748 Garching, Germany Andika; S. INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy Laboratoire Univers et Théorie, Observatoire de Paris, Université PSL, Université Paris Cité, CNRS, 92190 Meudon, France Anselmi; M. Dipartimento di Fisica "Aldo Pontremoli", Università degli Studi di Milano, Via Celoria 16, 20133 Milano, Italy INFN-Sezione di Milano, Via Celoria 16, 20133 Milano, Italy Archidiacono; F. Departamento de Física Fundamental. Universidad de Salamanca. Plaza de la Merced s/n. 37008 Salamanca, Spain Atrio-Barandela; D. Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy INAF-Osservatorio Astronomico di Padova, Via dell'Osservatorio 5, 35122 Padova, Italy INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Bertacca; M. Université de Strasbourg, CNRS, Observatoire astronomique de Strasbourg, UMR 7550, 67000 Strasbourg, France Bethermin; L. INAF-Osservatorio Astronomico di Padova, Via dell'Osservatorio 5, 35122 Padova, Italy Bisigello; A. Institut de Recherche en Astrophysique et Planétologie Blanchard; L. Center for Data-Driven Discovery, Kavli IPMU Laboratoire d'etude de l'Univers et des phenomenes eXtremes, Observatoire de Paris, Université PSL, Sorbonne Université, CNRS, 92190 Meudon, France Blot; S. Dipartimento di Fisica - Sezione di Astronomia, Università di Trieste, Via Tiepolo 11, 34131 Trieste, Italy IFPU, Institute for Fundamental Physics of the Universe, via Beirut 2, 34151 Trieste, Italy INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy INFN, Sezione di Trieste, Via Valerio 2, 34127 Trieste TS, Italy ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data e Quantum Computing, Via Magnanelli 2, Bologna, Italy Borgani; M. L. Jodrell Bank Centre for Astrophysics, Department of Physics and Astronomy, University of Manchester, Oxford Road, Manchester M13 9PL, UK Brown; S. California Institute of Technology, 1200 E California Blvd, Pasadena, CA 91125, USA Bruton; A. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Calabro; F. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Caro; T. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy INFN, Sezione di Trieste, Via Valerio 2, 34127 Trieste TS, Italy IFPU, Institute for Fundamental Physics of the Universe, via Beirut 2, 34151 Trieste, Italy ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data e Quantum Computing, Via Magnanelli 2, Bologna, Italy Castro; F. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Cogato; S. INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy Davini; G. Kapteyn Astronomical Institute, University of Groningen, PO Box 800, 9700 AV Groningen, The Netherlands Desprez; A. Departamento Física Aplicada, Universidad Politécnica de Cartagena, Campus Muralla del Mar, 30202 Cartagena, Murcia, Spain Díaz-Sánchez; J. J. Instituto de Astrofísica de Canarias, Vía Láctea, 38205 La Laguna, Tenerife, Spain Diaz; Domizio S. Dipartimento di Fisica, Università di Genova, Via Dodecaneso 33, 16146, Genova, Italy INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy Di; J. M. Instituto de Física de Cantabria, Edificio Juan Jordá, Avenida de los Castros, 39005 Santander, Spain Diego; P. -A. Université de Strasbourg, CNRS, Observatoire astronomique de Strasbourg, UMR 7550, 67000 Strasbourg, France Duc; A. Dipartimento di Fisica e Astronomia, Università di Bologna, Via Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Enia; Y. Universitäts-Sternwarte München, Fakultät für Physik, Ludwig-Maximilians-Universität München, Scheinerstrasse 1, 81679 München, Germany Fang; A. G. INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Ferrari; A. Department of Physics, P.O. Box 64, 00014 University of Helsinki, Finland Finoguenov; A. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Fontana; A. INFN, Sezione di Lecce, Via per Arnesano, CP-193, 73100, Lecce, Italy Department of Mathematics and Physics E. De Giorgi, University of Salento, Via per Arnesano, CP-I93, 73100, Lecce, Italy INAF-Sezione di Lecce, c/o Dipartimento Matematica e Fisica, Via per Arnesano, 73100, Lecce, Italy Franco; J. Instituto de Física Teórica UAM-CSIC, Campus de Cantoblanco, 28049 Madrid, Spain García-Bellido; T. INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy Gasparetto; V. CEA Saclay, DFR/IRFU, Service d'Astrophysique, Bat. 709, 91191 Gif-sur-Yvette, France Gautard; E. Institute of Space Sciences Institut d'Estudis Espacials de Catalunya Institute of Cosmology and Gravitation, University of Portsmouth, Portsmouth PO1 3FX, UK Gaztanaga; F. INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Giacomini; F. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Gianotti; M. Dipartimento di Fisica e Astronomia, Università di Bologna, Via Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Guidi; C. M. Instituto de Astrofí sica de Canarias, c/ Via Lactea s/n, La Laguna 38200, Spain. Departamento de Astrofí sica de la Universidad de La Laguna, Avda. Francisco Sanchez, La Laguna, 38200, Spain Gutierrez; A. Institute for Astronomy, University of Edinburgh, Royal Observatory, Blackford Hill, Edinburgh EH9 3HJ, UK Hall; S. Caltech/IPAC, 1200 E. California Blvd., Pasadena, CA 91125, USA Hemmati; H. Ruhr University Bochum, Faculty of Physics and Astronomy, Astronomical Institute Hildebrandt; J. DARK, Niels Bohr Institute, University of Copenhagen, Jagtvej 155, 2200 Copenhagen, Denmark Hjorth; J. J. E. Department of Physics and Astronomy, Vesilinnantie 5, 20014 University of Turku, Finland Serco for European Space Agency Kajava; Y. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Kang; V. ARC Centre of Excellence for Dark Matter Particle Physics, Melbourne, Australia Centre for Astrophysics \& Supercomputing, Swinburne University of Technology, Hawthorn, Victoria 3122, Australia Kansal; D. Dipartimento di Fisica e Scienze della Terra, Università degli Studi di Ferrara, Via Giuseppe Saragat 1, 44122 Ferrara, Italy Department of Physics and Astronomy, University of the Western Cape, Bellville, Cape Town, 7535, South Africa Karagiannis; C. C. Department of Physics and Helsinki Institute of Physics, Gustaf Hällströmin katu 2, 00014 University of Helsinki, Finland Kirkpatrick; S. ESAC/ESA, Camino Bajo del Castillo, s/n., Urb. Villafranca del Castillo, 28692 Villanueva de la Cañada, Madrid, Spain Kruk; L. DAMTP, Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA, UK Kavli Institute for Cosmology Cambridge, Madingley Road, Cambridge, CB3 0HA, UK Legrand; M. Dipartimento di Fisica e Scienze della Terra, Università degli Studi di Ferrara, Via Giuseppe Saragat 1, 44122 Ferrara, Italy Istituto Nazionale di Fisica Nucleare, Sezione di Ferrara, Via Giuseppe Saragat 1, 44122 Ferrara, Italy Lembo; F. Department of Astrophysics, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland Lepori; G. Department of Physics, Centre for Extragalactic Astronomy, Durham University, South Road, Durham, DH1 3LE, UK Department of Physics, Institute for Computational Cosmology, Durham University, South Road, Durham, DH1 3LE, UK Leroy; J. Institute for Theoretical Particle Physics and Cosmology Lesgourgues; L. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Leuzzi; T. I. IRFU, CEA, Université Paris-Saclay 91191 Gif-sur-Yvette Cedex, France Liaudat; J. Univ. Grenoble Alpes, CNRS, Grenoble INP, LPSC-IN2P3, 53, Avenue des Martyrs, 38000, Grenoble, France Macias-Perez; M. INAF-Istituto di Astrofisica e Planetologia Spaziali, via del Fosso del Cavaliere, 100, 00100 Roma, Italy Magliocchetti; F. INAF-Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, 50125, Firenze, Italy Mannucci; R. Dipartimento di Fisica, Sapienza Università di Roma, Piazzale Aldo Moro 2, 00185 Roma, Italy INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy Maoli; C. J. A. P. Centro de Astrofísica da Universidade do Porto, Rua das Estrelas, 4150-762 Porto, Portugal Instituto de Astrofísica e Ciências do Espaço, Universidade do Porto, CAUP, Rua das Estrelas, PT4150-762 Porto, Portugal Martins; L. Université Paris-Saclay, CNRS, Institut d'astrophysique spatiale, 91405, Orsay, France Maurin; M. ESAC/ESA, Camino Bajo del Castillo, s/n., Urb. Villafranca del Castillo, 28692 Villanueva de la Cañada, Madrid, Spain HE Space for European Space Agency Miluzio; P. Dipartimento di Fisica - Sezione di Astronomia, Università di Trieste, Via Tiepolo 11, 34131 Trieste, Italy INAF-Osservatorio Astronomico di Trieste, Via G. B. Tiepolo 11, 34143 Trieste, Italy INFN, Sezione di Trieste, Via Valerio 2, 34127 Trieste TS, Italy IFPU, Institute for Fundamental Physics of the Universe, via Beirut 2, 34151 Trieste, Italy Monaco; G. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Morgante; K. Institute of Cosmology and Gravitation, University of Portsmouth, Portsmouth PO1 3FX, UK Naidoo; A. Universität Bonn, Argelander-Institut für Astronomie, Auf dem Hügel 71, 53121 Bonn, Germany Navarro-Alsina; F. Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Passalacqua; K. Max-Planck-Institut für Astronomie, Königstuhl 17, 69117 Heidelberg, Germany Paterson; L. INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Patrizii; A. Aix-Marseille Université, CNRS/IN2P3, CPPM, Marseille, France Pisani; D. Department of Astrophysics, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland Potter; S. Dipartimento di Fisica e Astronomia "Augusto Righi" - Alma Mater Studiorum Università di Bologna, via Piero Gobetti 93/2, 40129 Bologna, Italy INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Quai; M. INAF-Osservatorio Astronomico di Padova, Via dell'Osservatorio 5, 35122 Padova, Italy Radovich; P. -F. Université Paris-Saclay, CNRS, Institut d'astrophysique spatiale, 91405, Orsay, France Rocci; G. Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy INAF-Osservatorio Astronomico di Padova, Via dell'Osservatorio 5, 35122 Padova, Italy Rodighiero; S. Department of Mathematics and Physics E. De Giorgi, University of Salento, Via per Arnesano, CP-I93, 73100, Lecce, Italy INFN, Sezione di Lecce, Via per Arnesano, CP-193, 73100, Lecce, Italy INAF-Sezione di Lecce, c/o Dipartimento Matematica e Fisica, Via per Arnesano, 73100, Lecce, Italy Sacquegna; M. Theoretical astrophysics, Department of Physics and Astronomy, Uppsala University, Box 515, 751 20 Uppsala, Sweden Sahlén; D. B. Institute for Astronomy, University of Hawaii, 2680 Woodlawn Drive, Honolulu, HI 96822, USA Sanders; E. SISSA, International School for Advanced Studies, Via Bonomea 265, 34136 Trieste TS, Italy ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data e Quantum Computing, Via Magnanelli 2, Bologna, Italy INFN, Sezione di Trieste, Via Valerio 2, 34127 Trieste TS, Italy Sarpa; A. Department of Astrophysics, University of Zurich, Winterthurerstrasse 190, 8057 Zurich, Switzerland Schneider; M. Université Côte d'Azur, Observatoire de la Côte d'Azur, CNRS, Laboratoire Lagrange, Bd de l'Observatoire, CS 34229, 06304 Nice cedex 4, France Schultheis; D. INAF-Osservatorio Astronomico di Roma, Via Frascati 33, 00078 Monteporzio Catone, Italy INFN-Sezione di Roma, Piazzale Aldo Moro, 2 - c/o Dipartimento di Fisica, Edificio G. Marconi, 00185 Roma, Italy Sciotti; E. Mathematical Institute, University of Leiden, Einsteinweg 55, 2333 CA Leiden, The Netherlands Leiden Observatory, Leiden University, Einsteinweg 55, 2333 CC Leiden, The Netherlands Sellentin; F. School of Physics \& Astronomy, University of Southampton, Highfield Campus, Southampton SO17 1BJ, UK Shankar; L. C. Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK Smith; K. Department of Physics, Oxford University, Keble Road, Oxford OX1 3RH, UK Tanidis; G. INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy Testera; R. Department of Astrophysical Sciences, Peyton Hall, Princeton University, Princeton, NJ 08544, USA Teyssier; S. Dipartimento di Fisica, Università di Genova, Via Dodecaneso 33, 16146, Genova, Italy INFN-Sezione di Genova, Via Dodecaneso 33, 16146, Genova, Italy INAF-Osservatorio Astronomico di Brera, Via Brera 28, 20122 Milano, Italy Tosi; A. Dipartimento di Fisica e Astronomia "G. Galilei", Università di Padova, Via Marzolo 8, 35131 Padova, Italy INFN-Padova, Via Marzolo 8, 35131 Padova, Italy Troja; M. Department of Astronomy, University of Geneva, ch. d'Ecogia 16, 1290 Versoix, Switzerland Tucci; C. INFN-Sezione di Bologna, Viale Berti Pichat 6/2, 40127 Bologna, Italy Valieri; D. INAF-Osservatorio di Astrofisica e Scienza dello Spazio di Bologna, Via Piero Gobetti 93/3, 40129 Bologna, Italy Vergani; G. Center for Computational Astrophysics, Flatiron Institute, 162 5th Avenue, 10010, New York, NY, USA Verza; N. A. Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge CB3 0HA, UK Walton
  Light emission from galaxies exhibit diverse brightness profiles, influenced
by factors such as galaxy type, structural features and interactions with other
galaxies. Elliptical galaxies feature more uniform light distributions, while
spiral and irregular galaxies have complex, varied light profiles due to their
structural heterogeneity and star-forming activity. In addition, galaxies with
an active galactic nucleus (AGN) feature intense, concentrated emission from
gas accretion around supermassive black holes, superimposed on regular galactic
light, while quasi-stellar objects (QSO) are the extreme case of the AGN
emission dominating the galaxy. The challenge of identifying AGN and QSO has
been discussed many times in the literature, often requiring multi-wavelength
observations. This paper introduces a novel approach to identify AGN and QSO
from a single image. Diffusion models have been recently developed in the
machine-learning literature to generate realistic-looking images of everyday
objects. Utilising the spatial resolving power of the Euclid VIS images, we
created a diffusion model trained on one million sources, without using any
source pre-selection or labels. The model learns to reconstruct light
distributions of normal galaxies, since the population is dominated by them. We
condition the prediction of the central light distribution by masking the
central few pixels of each source and reconstruct the light according to the
diffusion model. We further use this prediction to identify sources that
deviate from this profile by examining the reconstruction error of the few
central pixels regenerated in each source's core. Our approach, solely using
VIS imaging, features high completeness compared to traditional methods of AGN
and QSO selection, including optical, near-infrared, mid-infrared, and X-rays.
[abridged]


http://arxiv.org/abs/2503.13945
Make the Most of Everything: Further Considerations on Disrupting Diffusion-based Customization. (99%)
Long Tang; Dengpan Ye; Sirun Chen; Xiuwen Shi; Yunna Lv; Ziyi Liu
  The fine-tuning technique for text-to-image diffusion models facilitates
image customization but risks privacy breaches and opinion manipulation.
Current research focuses on prompt- or image-level adversarial attacks for
anti-customization, yet it overlooks the correlation between these two levels
and the relationship between internal modules and inputs. This hinders
anti-customization performance in practical threat scenarios. We propose Dual
Anti-Diffusion (DADiff), a two-stage adversarial attack targeting diffusion
customization, which, for the first time, integrates the adversarial
prompt-level attack into the generation process of image-level adversarial
examples. In stage 1, we generate prompt-level adversarial vectors to guide the
subsequent image-level attack. In stage 2, besides conducting the end-to-end
attack on the UNet model, we disrupt its self- and cross-attention modules,
aiming to break the correlations between image pixels and align the
cross-attention results computed using instance prompts and adversarial prompt
vectors within the images. Furthermore, we introduce a local random timestep
gradient ensemble strategy, which updates adversarial perturbations by
integrating random gradients from multiple segmented timesets. Experimental
results on various mainstream facial datasets demonstrate 10%-30% improvements
in cross-prompt, keyword mismatch, cross-model, and cross-mechanism
anti-customization with DADiff compared to existing methods.


http://arxiv.org/abs/2503.14281
XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants. (89%)
Adam Štorek; Mukur Gupta; Noopur Bhatt; Aditya Gupta; Janie Kim; Prashast Srivastava; Suman Jana
  AI coding assistants are widely used for tasks like code generation. These
tools now require large and complex contexts, automatically sourced from
various origins$\unicode{x2014}$across files, projects, and
contributors$\unicode{x2014}$forming part of the prompt fed to underlying LLMs.
This automatic context-gathering introduces new vulnerabilities, allowing
attackers to subtly poison input to compromise the assistant's outputs,
potentially generating vulnerable code or introducing critical errors. We
propose a novel attack, Cross-Origin Context Poisoning (XOXO), that is
challenging to detect as it relies on adversarial code modifications that are
semantically equivalent. Traditional program analysis techniques struggle to
identify these perturbations since the semantics of the code remains correct,
making it appear legitimate. This allows attackers to manipulate coding
assistants into producing incorrect outputs, while shifting the blame to the
victim developer. We introduce a novel, task-agnostic, black-box attack
algorithm GCGS that systematically searches the transformation space using a
Cayley Graph, achieving a 75.72% attack success rate on average across five
tasks and eleven models, including GPT 4.1 and Claude 3.5 Sonnet v2 used by
popular AI coding assistants. Furthermore, defenses like adversarial
fine-tuning are ineffective against our attack, underscoring the need for new
security measures in LLM-powered coding tools.


http://arxiv.org/abs/2503.16542
Defending Against Gradient Inversion Attacks for Biomedical Images via Learnable Data Perturbation. (81%)
Shiyi Jiang; Farshad Firouzi; Krishnendu Chakrabarty
  The increasing need for sharing healthcare data and collaborating on clinical
research has raised privacy concerns. Health information leakage due to
malicious attacks can lead to serious problems such as misdiagnoses and patient
identification issues. Privacy-preserving machine learning (PPML) and
privacy-enhancing technologies, particularly federated learning (FL), have
emerged in recent years as innovative solutions to balance privacy protection
with data utility; however, they also suffer from inherent privacy
vulnerabilities. Gradient inversion attacks constitute major threats to data
sharing in federated learning. Researchers have proposed many defenses against
gradient inversion attacks. However, current defense methods for healthcare
data lack generalizability, i.e., existing solutions may not be applicable to
data from a broader range of populations. In addition, most existing defense
methods are tested using non-healthcare data, which raises concerns about their
applicability to real-world healthcare systems. In this study, we present a
defense against gradient inversion attacks in federated learning. We achieve
this using latent data perturbation and minimax optimization, utilizing both
general and medical image datasets. Our method is compared to two baselines,
and the results show that our approach can outperform the baselines with a
reduction of 12.5% in the attacker's accuracy in classifying reconstructed
images. The proposed method also yields an increase of over 12.4% in Mean
Squared Error (MSE) between the original and reconstructed images at the same
level of model utility of around 90% client classification accuracy. The
results suggest the potential of a generalizable defense for healthcare data.


http://arxiv.org/abs/2503.14299
Unveiling the Role of Randomization in Multiclass Adversarial Classification: Insights from Graph Theory. (69%)
Lucas Gnecco-Heredia; Matteo Sammut; Muni Sreenivas Pydi; Rafael Pinot; Benjamin Negrevergne; Yann Chevaleyre
  Randomization as a mean to improve the adversarial robustness of machine
learning models has recently attracted significant attention. Unfortunately,
much of the theoretical analysis so far has focused on binary classification,
providing only limited insights into the more complex multiclass setting. In
this paper, we take a step toward closing this gap by drawing inspiration from
the field of graph theory. Our analysis focuses on discrete data distributions,
allowing us to cast the adversarial risk minimization problems within the
well-established framework of set packing problems. By doing so, we are able to
identify three structural conditions on the support of the data distribution
that are necessary for randomization to improve robustness. Furthermore, we are
able to construct several data distributions where (contrarily to binary
classification) switching from a deterministic to a randomized solution
significantly reduces the optimal adversarial risk. These findings highlight
the crucial role randomization can play in enhancing robustness to adversarial
attacks in multiclass classification.


http://arxiv.org/abs/2503.15560
Temporal Context Awareness: A Defense Framework Against Multi-turn Manipulation Attacks on Large Language Models. (67%)
Prashant Kulkarni; Assaf Namer
  Large Language Models (LLMs) are increasingly vulnerable to sophisticated
multi-turn manipulation attacks, where adversaries strategically build context
through seemingly benign conversational turns to circumvent safety measures and
elicit harmful or unauthorized responses. These attacks exploit the temporal
nature of dialogue to evade single-turn detection methods, representing a
critical security vulnerability with significant implications for real-world
deployments.
  This paper introduces the Temporal Context Awareness (TCA) framework, a novel
defense mechanism designed to address this challenge by continuously analyzing
semantic drift, cross-turn intention consistency and evolving conversational
patterns. The TCA framework integrates dynamic context embedding analysis,
cross-turn consistency verification, and progressive risk scoring to detect and
mitigate manipulation attempts effectively. Preliminary evaluations on
simulated adversarial scenarios demonstrate the framework's potential to
identify subtle manipulation patterns often missed by traditional detection
techniques, offering a much-needed layer of security for conversational AI
systems. In addition to outlining the design of TCA , we analyze diverse attack
vectors and their progression across multi-turn conversation, providing
valuable insights into adversarial tactics and their impact on LLM
vulnerabilities. Our findings underscore the pressing need for robust,
context-aware defenses in conversational AI systems and highlight TCA framework
as a promising direction for securing LLMs while preserving their utility in
legitimate applications. We make our implementation available to support
further research in this emerging area of AI security.


http://arxiv.org/abs/2503.14783
RAT: Boosting Misclassification Detection Ability without Extra Data. (41%)
Ge Yan; Tsui-Wei Weng
  As deep neural networks(DNN) become increasingly prevalent, particularly in
high-stakes areas such as autonomous driving and healthcare, the ability to
detect incorrect predictions of models and intervene accordingly becomes
crucial for safety. In this work, we investigate the detection of misclassified
inputs for image classification models from the lens of adversarial
perturbation: we propose to use robust radius (a.k.a. input-space margin) as a
confidence metric and design two efficient estimation algorithms, RR-BS and
RR-Fast, for misclassification detection. Furthermore, we design a training
method called Radius Aware Training (RAT) to boost models' ability to identify
mistakes. Extensive experiments show our method could achieve up to 29.3%
reduction on AURC and 21.62% reduction in FPR@95TPR, compared with previous
methods.


http://arxiv.org/abs/2503.13962
Survey of Adversarial Robustness in Multimodal Large Language Models. (33%)
Chengze Jiang; Zhuangzhuang Wang; Minjing Dong; Jie Gui
  Multimodal Large Language Models (MLLMs) have demonstrated exceptional
performance in artificial intelligence by facilitating integrated understanding
across diverse modalities, including text, images, video, audio, and speech.
However, their deployment in real-world applications raises significant
concerns about adversarial vulnerabilities that could compromise their safety
and reliability. Unlike unimodal models, MLLMs face unique challenges due to
the interdependencies among modalities, making them susceptible to
modality-specific threats and cross-modal adversarial manipulations. This paper
reviews the adversarial robustness of MLLMs, covering different modalities. We
begin with an overview of MLLMs and a taxonomy of adversarial attacks tailored
to each modality. Next, we review key datasets and evaluation metrics used to
assess the robustness of MLLMs. After that, we provide an in-depth review of
attacks targeting MLLMs across different modalities. Our survey also identifies
critical challenges and suggests promising future research directions.


http://arxiv.org/abs/2503.14827
MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models. (5%)
Chejian Xu; Jiawei Zhang; Zhaorun Chen; Chulin Xie; Mintong Kang; Yujin Potter; Zhun Wang; Zhuowen Yuan; Alexander Xiong; Zidi Xiong; Chenhui Zhang; Lingzhi Yuan; Yi Zeng; Peiyang Xu; Chengquan Guo; Andy Zhou; Jeffrey Ziwei Tan; Xuandong Zhao; Francesco Pinto; Zhen Xiang; Yu Gai; Zinan Lin; Dan Hendrycks; Bo Li; Dawn Song
  Multimodal foundation models (MMFMs) play a crucial role in various
applications, including autonomous driving, healthcare, and virtual assistants.
However, several studies have revealed vulnerabilities in these models, such as
generating unsafe content by text-to-image models. Existing benchmarks on
multimodal models either predominantly assess the helpfulness of these models,
or only focus on limited perspectives such as fairness and privacy. In this
paper, we present the first unified platform, MMDT (Multimodal DecodingTrust),
designed to provide a comprehensive safety and trustworthiness evaluation for
MMFMs. Our platform assesses models from multiple perspectives, including
safety, hallucination, fairness/bias, privacy, adversarial robustness, and
out-of-distribution (OOD) generalization. We have designed various evaluation
scenarios and red teaming algorithms under different tasks for each perspective
to generate challenging data, forming a high-quality benchmark. We evaluate a
range of multimodal models using MMDT, and our findings reveal a series of
vulnerabilities and areas for improvement across these perspectives. This work
introduces the first comprehensive and unique safety and trustworthiness
evaluation platform for MMFMs, paving the way for developing safer and more
reliable MMFMs and systems. Our platform and benchmark are available at
https://mmdecodingtrust.github.io/.


http://arxiv.org/abs/2503.14618
Anomaly-Flow: A Multi-domain Federated Generative Adversarial Network for Distributed Denial-of-Service Detection. (1%)
Melo Leonardo Henrique de; Gustavo de Carvalho Bertoli; Michele Nogueira; Aldri Luiz dos Santos; Lourenço Alves Pereira Junior
  Distributed denial-of-service (DDoS) attacks remain a critical threat to
Internet services, causing costly disruptions. While machine learning (ML) has
shown promise in DDoS detection, current solutions struggle with multi-domain
environments where attacks must be detected across heterogeneous networks and
organizational boundaries. This limitation severely impacts the practical
deployment of ML-based defenses in real-world settings.
  This paper introduces Anomaly-Flow, a novel framework that addresses this
critical gap by combining Federated Learning (FL) with Generative Adversarial
Networks (GANs) for privacy-preserving, multi-domain DDoS detection. Our
proposal enables collaborative learning across diverse network domains while
preserving data privacy through synthetic flow generation. Through extensive
evaluation across three distinct network datasets, Anomaly-Flow achieves an
average F1-score of $0.747$, outperforming baseline models. Importantly, our
framework enables organizations to share attack detection capabilities without
exposing sensitive network data, making it particularly valuable for critical
infrastructure and privacy-sensitive sectors.
  Beyond immediate technical contributions, this work provides insights into
the challenges and opportunities in multi-domain DDoS detection, establishing a
foundation for future research in collaborative network defense systems. Our
findings have important implications for academic research and industry
practitioners working to deploy practical ML-based security solutions.


http://arxiv.org/abs/2503.14852
UntrustVul: An Automated Approach for Identifying Untrustworthy Alerts in Vulnerability Detection Models. (1%)
Lam Nguyen Tung; Xiaoning Du; Neelofar Neelofar; Aldeida Aleti
  Machine learning (ML) has shown promise in detecting vulnerabilities. To
review vulnerabilities detected by ML predictions, developers manually assess
suspicious lines in their interpretations. However, studies have revealed that
these models often learn and predict based on irrelevant features frequently
appearing in vulnerable code. This leads to predictions that may correctly flag
vulnerable functions but for the wrong reasons, which we call untrustworthy.
These predictions can mislead developers, hindering them from locating the
vulnerabilities. This increases the efforts of manual assessment and, worse,
risks creating flawed patches that fail to address existing vulnerabilities and
even introduce new ones. Hence, automated approaches are needed to detect
untrustworthy predictions, preventing overlooked vulnerabilities and
alleviating the burden of manual assessment.
  We propose UntrustVul, the first automated approach to identify untrustworthy
vulnerability predictions. Given a vulnerability prediction during inference,
UntrustVul systematically assesses whether suspicious lines annotated by the
prediction are vulnerability-unrelated. It simulates developers' rationales,
considering a line unrelated if (1) it is absent from historical
vulnerabilities and (2) it cannot reach any vulnerabilities in execution flows.
UntrustVul assesses (1) by analysing its syntactic meaning using deep
representations to determine whether it is syntax-benign. To assess (2),
UntrustVul traces dependencies of the syntax-benign lines on other suspicious
lines using static and rule-based analyses. We evaluate UntrustVul on 155K
vulnerability predictions by four models across three datasets. UntrustVul
effectively detects untrustworthy predictions with an F1-score of 82%-94% and
helps improve the ability of models to detect vulnerabilities by up to 321% in
F1-score and 100% in trustworthiness.


http://arxiv.org/abs/2503.13994
TarPro: Targeted Protection against Malicious Image Editing. (1%)
Kaixin Shen; Ruijie Quan; Jiaxu Miao; Jun Xiao; Yi Yang
  The rapid advancement of image editing techniques has raised concerns about
their misuse for generating Not-Safe-for-Work (NSFW) content. This necessitates
a targeted protection mechanism that blocks malicious edits while preserving
normal editability. However, existing protection methods fail to achieve this
balance, as they indiscriminately disrupt all edits while still allowing some
harmful content to be generated. To address this, we propose TarPro, a targeted
protection framework that prevents malicious edits while maintaining benign
modifications. TarPro achieves this through a semantic-aware constraint that
only disrupts malicious content and a lightweight perturbation generator that
produces a more stable, imperceptible, and robust perturbation for image
protection. Extensive experiments demonstrate that TarPro surpasses existing
methods, achieving a high protection efficacy while ensuring minimal impact on
normal edits. Our results highlight TarPro as a practical solution for secure
and controlled image editing.


http://arxiv.org/abs/2503.12874
Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models. (99%)
Xiaojun Jia; Sensen Gao; Simeng Qin; Ke Ma; Xinfeng Li; Yihao Huang; Wei Dong; Yang Liu; Xiaochun Cao
  Large pre-trained vision-language models (VLMs), such as CLIP, demonstrate
impressive generalization but remain highly vulnerable to adversarial examples
(AEs). Previous work has explored robust text prompts through adversarial
training, achieving some improvement in both robustness and generalization.
However, they primarily rely on singlegradient direction perturbations (e.g.,
PGD) to generate AEs, which lack diversity, resulting in limited improvement in
adversarial robustness. To address these limitations, we propose an
evolution-based region adversarial prompt tuning method called ER-APT, which
combines gradient methods with genetic evolution to generate more diverse and
challenging AEs. In each training iteration, we first generate AEs using
traditional gradient-based methods. Subsequently, a genetic evolution mechanism
incorporating selection, mutation, and crossover is applied to optimize the
AEs, ensuring a broader and more aggressive perturbation distribution.The final
evolved AEs are used for prompt tuning, achieving region-based adversarial
optimization instead of conventional single-point adversarial prompt tuning. We
also propose a dynamic loss weighting method to adjust prompt learning
efficiency for accuracy and robustness. Experimental evaluations on various
benchmark datasets demonstrate the superiority of our proposed method,
outperforming stateof-the-art APT methods. The code is released at
https://github.com/jiaxiaojunQAQ/ER-APT.


http://arxiv.org/abs/2503.12793
Improving Generalization of Universal Adversarial Perturbation via Dynamic Maximin Optimization. (99%)
Yechao Zhang; Yingzhe Xu; Junyu Shi; Leo Yu Zhang; Shengshan Hu; Minghui Li; Yanjun Zhang
  Deep neural networks (DNNs) are susceptible to universal adversarial
perturbations (UAPs). These perturbations are meticulously designed to fool the
target model universally across all sample classes. Unlike instance-specific
adversarial examples (AEs), generating UAPs is more complex because they must
be generalized across a wide range of data samples and models. Our research
reveals that existing universal attack methods, which optimize UAPs using DNNs
with static model parameter snapshots, do not fully leverage the potential of
DNNs to generate more effective UAPs. Rather than optimizing UAPs against
static DNN models with a fixed training set, we suggest using dynamic
model-data pairs to generate UAPs. In particular, we introduce a dynamic
maximin optimization strategy, aiming to optimize the UAP across a variety of
optimal model-data pairs. We term this approach DM-UAP. DM-UAP utilizes an
iterative max-min-min optimization framework that refines the model-data pairs,
coupled with a curriculum UAP learning algorithm to examine the combined space
of model parameters and data thoroughly. Comprehensive experiments on the
ImageNet dataset demonstrate that the proposed DM-UAP markedly enhances both
cross-sample universality and cross-model transferability of UAPs. Using only
500 samples for UAP generation, DM-UAP outperforms the state-of-the-art
approach with an average increase in fooling ratio of 12.108%.


http://arxiv.org/abs/2503.12827
GSBA$^K$: $top$-$K$ Geometric Score-based Black-box Attack. (99%)
Md Farhamdur Reza; Richeng Jin; Tianfu Wu; Huaiyu Dai
  Existing score-based adversarial attacks mainly focus on crafting $top$-1
adversarial examples against classifiers with single-label classification.
Their attack success rate and query efficiency are often less than
satisfactory, particularly under small perturbation requirements; moreover, the
vulnerability of classifiers with multi-label learning is yet to be studied. In
this paper, we propose a comprehensive surrogate free score-based attack, named
\b geometric \b score-based \b black-box \b attack (GSBA$^K$), to craft
adversarial examples in an aggressive $top$-$K$ setting for both untargeted and
targeted attacks, where the goal is to change the $top$-$K$ predictions of the
target classifier. We introduce novel gradient-based methods to find a good
initial boundary point to attack. Our iterative method employs novel gradient
estimation techniques, particularly effective in $top$-$K$ setting, on the
decision boundary to effectively exploit the geometry of the decision boundary.
Additionally, GSBA$^K$ can be used to attack against classifiers with $top$-$K$
multi-label learning. Extensive experimental results on ImageNet and PASCAL VOC
datasets validate the effectiveness of GSBA$^K$ in crafting $top$-$K$
adversarial examples.


http://arxiv.org/abs/2503.13419
Securing Virtual Reality Experiences: Unveiling and Tackling Cybersickness Attacks with Explainable AI. (93%)
Ripan Kumar Kundu; Matthew Denton; Genova Mongalo; Prasad Calyam; Khaza Anuarul Hoque
  The synergy between virtual reality (VR) and artificial intelligence (AI),
specifically deep learning (DL)-based cybersickness detection models, has
ushered in unprecedented advancements in immersive experiences by automatically
detecting cybersickness severity and adaptively various mitigation techniques,
offering a smooth and comfortable VR experience. While this DL-enabled
cybersickness detection method provides promising solutions for enhancing user
experiences, it also introduces new risks since these models are vulnerable to
adversarial attacks; a small perturbation of the input data that is visually
undetectable to human observers can fool the cybersickness detection model and
trigger unexpected mitigation, thus disrupting user immersive experiences (UIX)
and even posing safety risks. In this paper, we present a new type of VR
attack, i.e., a cybersickness attack, which successfully stops the triggering
of cybersickness mitigation by fooling DL-based cybersickness detection models
and dramatically hinders the UIX. Next, we propose a novel explainable
artificial intelligence (XAI)-guided cybersickness attack detection framework
to detect such attacks in VR to ensure UIX and a comfortable VR experience. We
evaluate the proposed attack and the detection framework using two
state-of-the-art open-source VR cybersickness datasets: Simulation 2021 and
Gameplay dataset. Finally, to verify the effectiveness of our proposed method,
we implement the attack and the XAI-based detection using a testbed with a
custom-built VR roller coaster simulation with an HTC Vive Pro Eye headset and
perform a user study. Our study shows that such an attack can dramatically
hinder the UIX. However, our proposed XAI-guided cybersickness attack detection
can successfully detect cybersickness attacks and trigger the proper
mitigation, effectively reducing VR cybersickness.


http://arxiv.org/abs/2503.12931
MirrorGuard: Adaptive Defense Against Jailbreaks via Entropy-Guided Mirror Crafting. (68%)
Rui Pu; Chaozhuo Li; Rui Ha; Litian Zhang; Lirong Qiu; Xi Zhang
  Defending large language models (LLMs) against jailbreak attacks is crucial
for ensuring their safe deployment. Existing defense strategies generally rely
on predefined static criteria to differentiate between harmful and benign
prompts. However, such rigid rules are incapable of accommodating the inherent
complexity and dynamic nature of real jailbreak attacks. In this paper, we
propose a novel concept of ``mirror'' to enable dynamic and adaptive defense. A
mirror refers to a dynamically generated prompt that mirrors the syntactic
structure of the input while ensuring semantic safety. The personalized
discrepancies between the input prompts and their corresponding mirrors serve
as the guiding principles for defense. A new defense paradigm, MirrorGuard, is
further proposed to detect and calibrate risky inputs based on such mirrors. An
entropy-based detection metric, Relative Input Uncertainty (RIU), is integrated
into MirrorGuard to quantify the discrepancies between input prompts and
mirrors. MirrorGuard is evaluated on several popular datasets, demonstrating
state-of-the-art defense performance while maintaining general effectiveness.


http://arxiv.org/abs/2506.12103
The Amazon Nova Family of Models: Technical Report and Model Card. (33%)
Amazon JC AGI; Aaron JC Langford; Aayush JC Shah; Abhanshu JC Gupta; Abhimanyu JC Bhatter; Abhinav JC Goyal; Abhinav JC Mathur; Abhinav JC Mohanty; Abhishek JC Kumar; Abhishek JC Sethi; Abi JC Komma; Abner JC Pena; Achin JC Jain; Adam JC Kunysz; Adam JC Opyrchal; Adarsh JC Singh; Aditya JC Rawal; Adok Achar Budihal JC Prasad; Gispert Adrià JC de; Agnika JC Kumar; Aishwarya JC Aryamane; Ajay JC Nair; Akilan JC M; Akshaya JC Iyengar; Akshaya Vishnu Kudlu JC Shanbhogue; Alan JC He; Alessandra JC Cervone; Alex JC Loeb; Alex JC Zhang; Alexander JC Fu; Alexander JC Lisnichenko; Alexander JC Zhipa; Alexandros JC Potamianos; Ali JC Kebarighotbi; Aliakbar JC Daronkolaei; Alok JC Parmesh; Amanjot Kaur JC Samra; Ameen JC Khan; Amer JC Rez; Amir JC Saffari; Amit JC Agarwalla; Amit JC Jhindal; Amith JC Mamidala; Ammar JC Asmro; Amulya JC Ballakur; Anand JC Mishra; Anand JC Sridharan; Anastasiia JC Dubinina; Andre JC Lenz; Andreas JC Doerr; Andrew JC Keating; Andrew JC Leaver; Andrew JC Smith; Andrew JC Wirth; Andy JC Davey; Andy JC Rosenbaum; Andy JC Sohn; Angela JC Chan; Aniket JC Chakrabarti; Anil JC Ramakrishna; Anirban JC Roy; Anita JC Iyer; Anjali JC Narayan-Chen; Ankith JC Yennu; Anna JC Dabrowska; Anna JC Gawlowska; Anna JC Rumshisky; Anna JC Turek; Anoop JC Deoras; Anton JC Bezruchkin; Anup JC Prasad; Anupam JC Dewan; Anwith JC Kiran; Apoorv JC Gupta; Aram JC Galstyan; Aravind JC Manoharan; Arijit JC Biswas; Arindam JC Mandal; Arpit JC Gupta; Arsamkhan JC Pathan; Arun JC Nagarajan; Arushan JC Rajasekaram; Arvind JC Sundararajan; Ashwin JC Ganesan; Ashwin JC Swaminathan; Athanasios JC Mouchtaris; Audrey JC Champeau; Avik JC Ray; Ayush JC Jaiswal; Ayush JC Sharma; Bailey JC Keefer; Balamurugan JC Muthiah; Beatriz JC Leon-Millan; Ben JC Koopman; Ben JC Li; Benjamin JC Biggs; Benjamin JC Ott; Bhanu JC Vinzamuri; Bharath JC Venkatesh; Bhavana JC Ganesh; Bhoomit JC Vasani; Bill JC Byrne; Bill JC Hsu; Bincheng JC Wang; Blake JC King; Blazej JC Gorny; Bo JC Feng; Bo JC Zheng; Bodhisattwa JC Paul; Bofan JC Sun; Bofeng JC Luo; Bowen JC Chen; Bowen JC Xie; Boya JC Yu; Brendan JC Jugan; Brett JC Panosh; Brian JC Collins; Brian JC Thompson; Can JC Karakus; Can JC Liu; Carl JC Lambrecht; Carly JC Lin; Carolyn JC Wang; Carrie JC Yuan; Casey JC Loyda; Cezary JC Walczak; Chalapathi JC Choppa; Chandana Satya JC Prakash; Chankrisna Richy JC Meas; Charith JC Peris; Charles JC Recaido; Charlie JC Xu; Charul JC Sharma; Chase JC Kernan; Chayut JC Thanapirom; Chengwei JC Su; Chenhao JC Xu; Chenhao JC Yin; Chentao JC Ye; Chenyang JC Tao; Chethan JC Parameshwara; Ching-Yun JC Chang; Chong JC Li; Chris JC Hench; Chris JC Tran; Christophe JC Dupuy; Christopher JC Davis; Christopher JC DiPersio; Christos JC Christodoulopoulos; Christy JC Li; Chun JC Chen; Claudio Delli JC Bovi; Clement JC Chung; Cole JC Hawkins; Connor JC Harris; Corey JC Ropell; Cynthia JC He; DK JC Joo; Dae Yon JC Hwang; Dan JC Rosen; Daniel JC Elkind; Daniel JC Pressel; Daniel JC Zhang; Danielle JC Kimball; Daniil JC Sorokin; Dave JC Goodell; Davide JC Modolo; Dawei JC Zhu; Deepikaa JC Suresh; Deepti JC Ragha; Denis JC Filimonov; Denis Foo JC Kune; Denis Romasanta JC Rodriguez; Devamanyu JC Hazarika; Dhananjay JC Ram; Dhawal JC Parkar; Dhawal JC Patel; Dhwanil JC Desai; Dinesh Singh JC Rajput; Disha JC Sule; Diwakar JC Singh; Dmitriy JC Genzel; Dolly JC Goldenberg; Dongyi JC He; Dumitru JC Hanciu; Dushan JC Tharmal; Dzmitry JC Siankovich; Edi JC Cikovic; Edwin JC Abraham; Ekraam JC Sabir; Elliott JC Olson; Emmett JC Steven; Emre JC Barut; Eric JC Jackson; Ethan JC Wu; Evelyn JC Chen; Ezhilan JC Mahalingam; Fabian JC Triefenbach; Fan JC Yang; Fangyu JC Liu; Fanzi JC Wu; Faraz JC Tavakoli; Farhad JC Khozeimeh; Feiyang JC Niu; Felix JC Hieber; Feng JC Li; Firat JC Elbey; Florian JC Krebs; Florian JC Saupe; Florian JC Sprünken; Frank JC Fan; Furqan JC Khan; Vincenzo Gabriela JC De; Gagandeep JC Kang; George JC Ding; George JC He; George JC Yeung; Ghada JC Qaddoumi; Giannis JC Karamanolakis; Goeric JC Huybrechts; Gokul JC Maddali; Gonzalo JC Iglesias; Gordon JC McShane; Gozde JC Sahin; Guangtai JC Huang; Gukyeong JC Kwon; Gunnar A. JC Sigurdsson; Gurpreet JC Chadha; Gururaj JC Kosuru; Hagen JC Fuerstenau; Hah JC Hah; Haja JC Maideen; Hajime JC Hosokawa; Han JC Liu; Han-Kai JC Hsu; Hann JC Wang; Hao JC Li; Hao JC Yang; Haofeng JC Zhu; Haozheng JC Fan; Harman JC Singh; Harshavardhan JC Kaluvala; Hashim JC Saeed; He JC Xie; Helian JC Feng; Hendrix JC Luo; Hengzhi JC Pei; Henrik JC Nielsen; Hesam JC Ilati; Himanshu JC Patel; Hongshan JC Li; Hongzhou JC Lin; Hussain JC Raza; Ian JC Cullinan; Imre JC Kiss; Inbarasan JC Thangamani; Indrayani JC Fadnavis; Ionut Teodor JC Sorodoc; Irem JC Ertuerk; Iryna JC Yemialyanava; Ishan JC Soni; Ismail JC Jelal; Ivan JC Tse; Jack JC FitzGerald; Jack JC Zhao; Jackson JC Rothgeb; Jacky JC Lee; Jake JC Jung; Jakub JC Debski; Jakub JC Tomczak; James JC Jeun; James JC Sanders; Jason JC Crowley; Jay JC Lee; Jayakrishna Anvesh JC Paidy; Jayant JC Tiwari; Jean JC Farmer; Jeff JC Solinsky; Jenna JC Lau; Jeremy JC Savareese; Jerzy JC Zagorski; Ji JC Dai; JC Jiacheng; Skyler Gu; Jiahui Skyler Li; Skyler Jian; QZ Zheng; Jianhua QZ Lu; Jianhua QZ Wang; Jiawei QZ Dai; Jiawei QZ Mo; Jiaxi QZ Xu; Jie QZ Liang; Jie QZ Yang; Jim QZ Logan; Jimit QZ Majmudar; Jing QZ Liu; Jinghong QZ Miao; Jingru QZ Yi; Jingyang QZ Jin; Jiun-Yu QZ Kao; Jixuan QZ Wang; Jiyang QZ Wang; Joe QZ Pemberton; Joel QZ Carlson; Joey QZ Blundell; John QZ Chin-Jew; John QZ He; Jonathan QZ Ho; Jonathan QZ Hueser; Jonathan QZ Lunt; Jooyoung QZ Lee; Joshua QZ Tan; Joyjit QZ Chatterjee; Judith QZ Gaspers; Jue QZ Wang; Jun QZ Fang; Jun QZ Tang; Jun QZ Wan; Jun QZ Wu; Junlei QZ Wang; Junyi QZ Shi; Justin QZ Chiu; Justin QZ Satriano; Justin QZ Yee; Jwala QZ Dhamala; Jyoti QZ Bansal; Kai QZ Zhen; Kai-Wei QZ Chang; Kaixiang QZ Lin; Kalyan QZ Raman; Kanthashree Mysore QZ Sathyendra; Karabo QZ Moroe; Karan QZ Bhandarkar; Karan QZ Kothari; Karolina QZ Owczarzak; Karthick QZ Gopalswamy; Karthick QZ Ravi; Karthik QZ Ramakrishnan; Karthika QZ Arumugam; Kartik QZ Mehta; Katarzyna QZ Konczalska; Kavya QZ Ravikumar; Ke QZ Tran; Kechen QZ Qin; Kelin QZ Li; Kelvin QZ Li; Ketan QZ Kulkarni; Kevin Angelo QZ Rodrigues; Keyur QZ Patel; Khadige QZ Abboud; Kiana QZ Hajebi; Klaus QZ Reiter; Kris QZ Schultz; Krishna QZ Anisetty; Krishna QZ Kotnana; Kristen QZ Li; Kruthi QZ Channamallikarjuna; Krzysztof QZ Jakubczyk; Kuba QZ Pierewoj; Kunal QZ Pal; Kunwar QZ Srivastav; Kyle QZ Bannerman; Lahari QZ Poddar; Lakshmi QZ Prasad; Larry QZ Tseng; Laxmikant QZ Naik; Leena Chennuru QZ Vankadara; Lenon QZ Minorics; Leo QZ Liu; Leonard QZ Lausen; Leonardo F. R. QZ Ribeiro; Li QZ Zhang; Lili QZ Gehorsam; Ling QZ Qi; Lisa QZ Bauer; Lori QZ Knapp; Lu QZ Zeng; Lucas QZ Tong; Lulu QZ Wong; Luoxin QZ Chen; Maciej QZ Rudnicki; Mahdi QZ Namazifar; Mahesh QZ Jaliminche; Maira Ladeira QZ Tanke; Manasi QZ Gupta; Mandeep QZ Ahlawat; Mani QZ Khanuja; Mani QZ Sundaram; Marcin QZ Leyk; Mariusz QZ Momotko; Markus QZ Boese; Markus QZ Dreyer; Markus QZ Mueller; Mason QZ Fu; Mateusz QZ Górski; Mateusz QZ Mastalerczyk; Matias QZ Mora; Matt QZ Johnson; Matt QZ Scott; Matthew QZ Wen; Max QZ Barysau; Maya QZ Boumerdassi; Maya QZ Krishnan; Mayank QZ Gupta; Mayank QZ Hirani; Mayank QZ Kulkarni; Meganathan QZ Narayanasamy; Melanie QZ Bradford; Melanie QZ Gens; Melissa QZ Burke; Meng QZ Jin; Miao QZ Chen; Michael QZ Denkowski; Michael QZ Heymel; Michael QZ Krestyaninov; Michal QZ Obirek; Michalina QZ Wichorowska; Michał QZ Miotk; Milosz QZ Watroba; Mingyi QZ Hong; Mingzhi QZ Yu; Miranda QZ Liu; Mohamed QZ Gouda; Mohammad QZ El-Shabani; Mohammad QZ Ghavamzadeh; Mohit QZ Bansal; Morteza QZ Ziyadi; Nan QZ Xia; Nathan QZ Susanj; Nav QZ Bhasin; Neha QZ Goswami; Nehal QZ Belgamwar; Nicolas QZ Anastassacos; Nicolas QZ Bergeron; Nidhi QZ Jain; Nihal QZ Jain; Niharika QZ Chopparapu; Nik QZ Xu; Nikko QZ Strom; Nikolaos QZ Malandrakis; Nimisha QZ Mishra; Ninad QZ Parkhi; Ninareh QZ Mehrabi; Nishita QZ Sant; Nishtha QZ Gupta; Nitesh QZ Sekhar; Nithin QZ Rajeev; Nithish Raja QZ Chidambaram; Nitish QZ Dhar; Noor QZ Bhagwagar; Noy QZ Konforty; Omar QZ Babu; Omid QZ Razavi; Orchid QZ Majumder; Osama QZ Dar; Oscar QZ Hsu; Pablo QZ Kvitca; Pallavi QZ Pandey; Parker QZ Seegmiller; Patrick QZ Lange; Paul QZ Ferraro; Payal QZ Motwani; Pegah QZ Kharazmi; Pei QZ Wang; Pengfei QZ Liu; Peter QZ Bradtke; Peter QZ Götz; Peter QZ Zhou; Pichao QZ Wang; Piotr QZ Poskart; Pooja QZ Sonawane; Pradeep QZ Natarajan; Pradyun QZ Ramadorai; Pralam QZ Shah; Prasad QZ Nirantar; Prasanthi QZ Chavali; Prashan QZ Wanigasekara; Prashant QZ Saraf; Prashun QZ Dey; Pratyush QZ Pant; Prerak QZ Pradhan; Preyaa QZ Patel; Priyanka QZ Dadlani; Prudhvee Narasimha QZ Sadha; Qi QZ Dong; Qian QZ Hu; QZ Qiaozi; Sean Gao; Qing Sean Liu; Quinn Sean Lam; Quynh Sean Do; R. Sean Manmatha; Rachel Sean Willis; Rafael Sean Liu; Rafal Sean Ellert; Rafal Sean Kalinski; Rafi Al Sean Attrach; Ragha Sean Prasad; Ragini Sean Prasad; Raguvir Sean Kunani; Rahul Sean Gupta; Rahul Sean Sharma; Rahul Sean Tewari; Rajaganesh Sean Baskaran; Rajan Sean Singh; Rajiv Sean Gupta; Rajiv Sean Reddy; Rajshekhar Sean Das; Rakesh Sean Chada; Rakesh Vaideeswaran Sean Mahesh; Ram Sean Chandrasekaran; Ramesh Sean Nallapati; Ran Sean Xue; Rashmi Sean Gangadharaiah; Ravi Sean Rachakonda; Renxian Sean Zhang; Rexhina Sean Blloshmi; Rishabh Sean Agrawal; Robert Sean Enyedi; Robert Sean Lowe; Robik Sean Shrestha; Robinson Sean Piramuthu; Rohail Sean Asad; Rohan Sean Khanna; Rohan Sean Mukherjee; Rohit Sean Mittal; Rohit Sean Prasad; Rohith Mysore Vijaya Sean Kumar; Ron Sean Diamant; Ruchita Sean Gupta; Ruiwen Sean Li; Ruoying Sean Li; Rushabh Sean Fegade; Ruxu Sean Zhang; Ryan Sean Arbow; Ryan Sean Chen; Ryan Sean Gabbard; Ryan Sean Hoium; Ryan Sean King; Sabarishkumar Sean Iyer; Sachal Sean Malick; Sahar Sean Movaghati; Sai Sean Balakavi; Sai Sean Jakka; Sai Kashyap Sean Paruvelli; Sai Muralidhar Sean Jayanthi; Saicharan Shriram Sean Mujumdar; Sainyam Sean Kapoor; Sajjad Sean Beygi; Saket Sean Dingliwal; Saleh Sean Soltan; Sam Sean Ricklin; Sam Sean Tucker; Sameer Sean Sinha; Samridhi Sean Choudhary; Samson Sean Tan; Samuel Sean Broscheit; Samuel Sean Schulter; Sanchit Sean Agarwal; Sandeep Sean Atluri; Sander Sean Valstar; Sanjana Sean Shankar; Sanyukta Sean Sanyukta; Sarthak Sean Khanna; Sarvpriye Sean Khetrapal; Satish Sean Janakiraman; Saumil Sean Shah; Saurabh Sean Akolkar; Saurabh Sean Giri; Saurabh Sean Khandelwal; Saurabh Sean Pawar; Saurabh Sean Sahu; Sean Sean Huang; Sejun Sean Ra; Senthilkumar Sean Gopal; Sergei Sean Dobroshinsky; Shadi Sean Saba; Shamik Sean Roy; Shamit Sean Lal; Shankar Sean Ananthakrishnan; Sharon Sean Li; Shashwat Sean Srijan; Shekhar Sean Bhide; Sheng Long Sean Tang; Sheng Sean Zha; Shereen Sean Oraby; Sherif Sean Mostafa; Shiqi Sean Li; Shishir Sean Bharathi; Shivam Sean Prakash; Shiyuan Sean Huang; Shreya Sean Yembarwar; Shreyas Sean Pansare; Shreyas Sean Subramanian; Shrijeet Sean Joshi; Shuai Sean Liu; Shuai Sean Tang; Shubham Sean Chandak; Shubham Sean Garg; Shubham Sean Katiyar; Shubham Sean Mehta; Shubham Sean Srivastav; Shuo Sean Yang; Siddalingesha D Sean S; Siddharth Sean Choudhary; Siddharth Singh Sean Senger; Simon Sean Babb; Sina Sean Moeini; Siqi Sean Deng; Siva Sean Loganathan; Slawomir Sean Domagala; Sneha Sean Narkar; Sneha Sean Wadhwa; Songyang Sean Zhang; Songyao Sean Jiang; Sony Sean Trenous; Soumajyoti Sean Sarkar; Soumya Sean Saha; Sourabh Sean Reddy; Sourav Sean Dokania; Spurthideepika Sean Sandiri; Spyros Sean Matsoukas; Sravan Sean Bodapati; Sri Harsha Reddy Sean Wdaru; Sridevi Yagati Sean Venkateshdatta; Srikanth Sean Ronanki; Srinivasan R Sean Veeravanallur; Sriram Sean Venkatapathy; Sriramprabhu Sean Sankaraguru; Sruthi Sean Gorantla; Sruthi Sean Karuturi; Stefan Sean Schroedl; Subendhu Sean Rongali; Subhasis Sean Kundu; Suhaila Sean Shakiah; Sukriti Sean Tiwari; Sumit Sean Bharti; Sumita Sean Sami; Sumith Sean Mathew; Sunny Sean Yu; Sunwoo Sean Kim; Suraj Bajirao Sean Malode; Susana Cumplido Sean Riel; Swapnil Sean Palod; Swastik Sean Roy; Syed Sean Furqhan; Tagyoung Sean Chung; Takuma Sean Yoshitani; Taojiannan Sean Yang; Tejaswi Sean Chillakura; Tejwant Sean Bajwa; Temi Sean Lajumoke; Thanh Sean Tran; Thomas Sean Gueudre; Thomas Sean Jung; Tianhui Sean Li; Tim Sean Seemman; Timothy Sean Leffel; Tingting Sean Xiang; Tirth Sean Patel; Tobias Sean Domhan; Tobias Sean Falke; Toby Sean Guo; Tom Sean Li; Tomasz Sean Horszczaruk; Tomasz Sean Jedynak; Tushar Sean Kulkarni; Tyst Sean Marin; Tytus Sean Metrycki; Tzu-Yen Sean Wang; Umang Sean Jain; Upendra Sean Singh; Utkarsh Sean Chirimar; Vaibhav Sean Gupta; Vanshil Sean Shah; Varad Sean Deshpande; Varad Sean Gunjal; Varsha Sean Srikeshava; Varsha Sean Vivek; Varun Sean Bharadwaj; Varun Sean Gangal; Varun Sean Kumar; Venkatesh Sean Elango; Vicente Sean Ordonez; Victor Sean Soto; Vignesh Sean Radhakrishnan; Vihang Sean Patel; Vikram Sean Singh; Vinay Varma Sean Kolanuvada; Vinayshekhar Bannihatti Sean Kumar; Vincent Sean Auvray; Vincent Sean Cartillier; Vincent Sean Ponzo; Violet Sean Peng; Vishal Sean Khandelwal; Vishal Sean Naik; Vishvesh Sean Sahasrabudhe; Vitaliy Sean Korolev; Vivek Sean Gokuladas; Vivek Sean Madan; Vivek Sean Subramanian; Volkan Sean Cevher; Vrinda Sean Gupta; Wael Sean Hamza; Wei Sean Zhang; Weitong Sean Ruan; Weiwei Sean Cheng; Wen Sean Zhang; Wenbo Sean Zhao; Wenyan Sean Yao; Wenzhuo Sean Ouyang; Wesley Sean Dashner; William Sean Campbell; William Sean Lin; Willian Sean Martin; Wyatt Sean Pearson; Xiang Sean Jiang; Xiangxing Sean Lu; Xiangyang Sean Shi; Xianwen Sean Peng; Xiaofeng Sean Gao; Xiaoge Sean Jiang; Xiaohan Sean Fei; Xiaohui Sean Wang; Xiaozhou Joey Sean Zhou; Xin Sean Feng; Xinyan Sean Zhao; Xinyao Sean Wang; Xinyu Sean Li; Xu Sean Zhang; Xuan Sean Wang; Xuandi Sean Fu; Xueling Sean Yuan; Xuning Sean Wang; Yadunandana Sean Rao; Yair Sean Tavizon; Yan Sean Rossiytsev; Yanbei Sean Chen; Yang Sean Liu; Yang Sean Zou; Yangsook Sean Park; Yannick Sean Versley; Yanyan Sean Zhang; Yash Sean Patel; Yen-Cheng Sean Lu; Yi Sean Pan; Sean Yi-Hsiang; Rex Lai; Yichen Rex Hu; Yida Rex Wang; Yiheng Rex Zhou; Yilin Rex Xiang; Ying Rex Shi; Ying Rex Wang; Yishai Rex Galatzer; Yongxin Rex Wang; Yorick Rex Shen; Yuchen Rex Sun; Yudi Rex Purwatama; Rex Yue; Chris Wu; Yue Chris Gu; Yuechun Chris Wang; Yujun Chris Zeng; Yuncong Chris Chen; Yunke Chris Zhou; Yusheng Chris Xie; Yvon Chris Guy; Zbigniew Chris Ambrozinski; Zhaowei Chris Cai; Zhen Chris Zhang; Zheng Chris Wang; Zhenghui Chris Jin; Zhewei Chris Zhao; Zhiheng Chris Li; Zhiheng Chris Luo; Zhikang Chris Zhang; Zhilin Chris Fang; Zhiqi Chris Bu; Zhiyuan Chris Wang; Zhizhong Chris Li; Zijian Chris Wang; Chris Zimeng; Qiu; Zishi Li
  We present Amazon Nova, a new generation of state-of-the-art foundation
models that deliver frontier intelligence and industry-leading price
performance. Amazon Nova Pro is a highly-capable multimodal model with the best
combination of accuracy, speed, and cost for a wide range of tasks. Amazon Nova
Lite is a low-cost multimodal model that is lightning fast for processing
images, video, documents and text. Amazon Nova Micro is a text-only model that
delivers our lowest-latency responses at very low cost. Amazon Nova Canvas is
an image generation model that creates professional grade images with rich
customization controls. Amazon Nova Reel is a video generation model offering
high-quality outputs, customization, and motion control. Our models were built
responsibly and with a commitment to customer trust, security, and reliability.
We report benchmarking results for core capabilities, agentic performance, long
context, functional adaptation, runtime performance, and human evaluation.


http://arxiv.org/abs/2503.13751
Optimizing ML Training with Metagradient Descent. (3%)
Logan Engstrom; Andrew Ilyas; Benjamin Chen; Axel Feldmann; William Moses; Aleksander Madry
  A major challenge in training large-scale machine learning models is
configuring the training process to maximize model performance, i.e., finding
the best training setup from a vast design space. In this work, we unlock a
gradient-based approach to this problem. We first introduce an algorithm for
efficiently calculating metagradients -- gradients through model training -- at
scale. We then introduce a "smooth model training" framework that enables
effective optimization using metagradients. With metagradient descent (MGD), we
greatly improve on existing dataset selection methods, outperform
accuracy-degrading data poisoning attacks by an order of magnitude, and
automatically find competitive learning rate schedules.


http://arxiv.org/abs/2503.12990
How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark. (2%)
Roba Al Majzoub; Hashmat Malik; Muzammal Naseer; Zaigham Zaheer; Tariq Mahmood; Salman Khan; Fahad Khan
  Recently, histopathology vision-language foundation models (VLMs) have gained
popularity due to their enhanced performance and generalizability across
different downstream tasks. However, most existing histopathology benchmarks
are either unimodal or limited in terms of diversity of clinical tasks, organs,
and acquisition instruments, as well as their partial availability to the
public due to patient data privacy. As a consequence, there is a lack of
comprehensive evaluation of existing histopathology VLMs on a unified benchmark
setting that better reflects a wide range of clinical scenarios. To address
this gap, we introduce HistoVL, a fully open-source comprehensive benchmark
comprising images acquired using up to 11 various acquisition tools that are
paired with specifically crafted captions by incorporating class names and
diverse pathology descriptions. Our Histo-VL includes 26 organs, 31 cancer
types, and a wide variety of tissue obtained from 14 heterogeneous patient
cohorts, totaling more than 5 million patches obtained from over 41K WSIs
viewed under various magnification levels. We systematically evaluate existing
histopathology VLMs on Histo-VL to simulate diverse tasks performed by experts
in real-world clinical scenarios. Our analysis reveals interesting findings,
including large sensitivity of most existing histopathology VLMs to textual
changes with a drop in balanced accuracy of up to 25% in tasks such as
Metastasis detection, low robustness to adversarial attacks, as well as
improper calibration of models evident through high ECE values and low model
prediction confidence, all of which can affect their clinical implementation.


http://arxiv.org/abs/2503.13224
ProDiF: Protecting Domain-Invariant Features to Secure Pre-Trained Models Against Extraction. (1%)
Tong Zhou; Shijin Duan; Gaowen Liu; Charles Fleming; Ramana Rao Kompella; Shaolei Ren; Xiaolin Xu
  Pre-trained models are valuable intellectual property, capturing both
domain-specific and domain-invariant features within their weight spaces.
However, model extraction attacks threaten these assets by enabling
unauthorized source-domain inference and facilitating cross-domain transfer via
the exploitation of domain-invariant features. In this work, we introduce
**ProDiF**, a novel framework that leverages targeted weight space manipulation
to secure pre-trained models against extraction attacks. **ProDiF** quantifies
the transferability of filters and perturbs the weights of critical filters in
unsecured memory, while preserving actual critical weights in a Trusted
Execution Environment (TEE) for authorized users. A bi-level optimization
further ensures resilience against adaptive fine-tuning attacks. Experimental
results show that **ProDiF** reduces source-domain accuracy to near-random
levels and decreases cross-domain transferability by 74.65\%, providing robust
protection for pre-trained models. This work offers comprehensive protection
for pre-trained DNN models and highlights the potential of weight space
manipulation as a novel approach to model security.


http://arxiv.org/abs/2503.12567
GAN-Based Single-Stage Defense for Traffic Sign Classification Under Adversarial Patch Attack. (99%)
Abyad Enan; Mashrur Chowdhury
  Computer Vision plays a critical role in ensuring the safe navigation of
autonomous vehicles (AVs). An AV perception module is responsible for capturing
and interpreting the surrounding environment to facilitate safe navigation.
This module enables AVs to recognize traffic signs, traffic lights, and various
road users. However, the perception module is vulnerable to adversarial
attacks, which can compromise their accuracy and reliability. One such attack
is the adversarial patch attack (APA), a physical attack in which an adversary
strategically places a specially crafted sticker on an object to deceive object
classifiers. In APA, an adversarial patch is positioned on a target object,
leading the classifier to misidentify it. Such an APA can cause AVs to
misclassify traffic signs, leading to catastrophic incidents. To enhance the
security of an AV perception system against APAs, this study develops a
Generative Adversarial Network (GAN)-based single-stage defense strategy for
traffic sign classification. This approach is tailored to defend against APAs
on different classes of traffic signs without prior knowledge of a patch's
design. This study found this approach to be effective against patches of
varying sizes. Our experimental analysis demonstrates that the defense strategy
presented in this paper improves the classifier's accuracy under APA conditions
by up to 80.8% and enhances overall classification accuracy for all the traffic
signs considered in this study by 58%, compared to a classifier without any
defense mechanism. Our defense strategy is model-agnostic, making it applicable
to any traffic sign classifier, regardless of the underlying classification
model.


http://arxiv.org/abs/2503.12683
Algebraic Adversarial Attacks on Explainability Models. (89%)
Lachlan Simpson; Federico Costanza; Kyle Millar; Adriel Cheng; Cheng-Chew Lim; Hong Gunn Chew
  Classical adversarial attacks are phrased as a constrained optimisation
problem. Despite the efficacy of a constrained optimisation approach to
adversarial attacks, one cannot trace how an adversarial point was generated.
In this work, we propose an algebraic approach to adversarial attacks and study
the conditions under which one can generate adversarial examples for post-hoc
explainability models. Phrasing neural networks in the framework of geometric
deep learning, algebraic adversarial attacks are constructed through analysis
of the symmetry groups of neural networks. Algebraic adversarial examples
provide a mathematically tractable approach to adversarial examples. We
validate our approach of algebraic adversarial examples on two well-known and
one real-world dataset.


http://arxiv.org/abs/2503.14530
SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders. (1%)
Qing Li; Jiahui Geng; Derui Zhu; Fengyu Cai; Chenyang Lyu; Fakhri Karray
  Unlearning methods for vision-language models (VLMs) have primarily adapted
techniques from large language models (LLMs), relying on weight updates that
demand extensive annotated forget sets. Moreover, these methods perform
unlearning at a coarse granularity, often leading to excessive forgetting and
reduced model utility. To address this issue, we introduce SAUCE, a novel
method that leverages sparse autoencoders (SAEs) for fine-grained and selective
concept unlearning in VLMs. Briefly, SAUCE first trains SAEs to capture
high-dimensional, semantically rich sparse features. It then identifies the
features most relevant to the target concept for unlearning. During inference,
it selectively modifies these features to suppress specific concepts while
preserving unrelated information. We evaluate SAUCE on two distinct VLMs,
LLaVA-v1.5-7B and LLaMA-3.2-11B-Vision-Instruct, across two types of tasks:
concrete concept unlearning (objects and sports scenes) and abstract concept
unlearning (emotions, colors, and materials), encompassing a total of 60
concepts. Extensive experiments demonstrate that SAUCE outperforms
state-of-the-art methods by 18.04% in unlearning quality while maintaining
comparable model utility. Furthermore, we investigate SAUCE's robustness
against widely used adversarial attacks, its transferability across models, and
its scalability in handling multiple simultaneous unlearning requests. Our
findings establish SAUCE as an effective and scalable solution for selective
concept unlearning in VLMs.


http://arxiv.org/abs/2503.12453
Shape Bias and Robustness Evaluation via Cue Decomposition for Image Classification and Segmentation. (1%)
Edgar Heinert; Thomas Gottwald; Annika Mütze; Matthias Rottmann
  Previous works studied how deep neural networks (DNNs) perceive image content
in terms of their biases towards different image cues, such as texture and
shape. Previous methods to measure shape and texture biases are typically
style-transfer-based and limited to DNNs for image classification. In this
work, we provide a new evaluation procedure consisting of 1) a
cue-decomposition method that comprises two AI-free data pre-processing methods
extracting shape and texture cues, respectively, and 2) a novel
cue-decomposition shape bias evaluation metric that leverages the
cue-decomposition data. For application purposes we introduce a corresponding
cue-decomposition robustness metric that allows for the estimation of the
robustness of a DNN w.r.t. image corruptions. In our numerical experiments, our
findings for biases in image classification DNNs align with those of previous
evaluation metrics. However, our cue-decomposition robustness metric shows
superior results in terms of estimating the robustness of DNNs. Furthermore,
our results for DNNs on the semantic segmentation datasets Cityscapes and
ADE20k for the first time shed light into the biases of semantic segmentation
DNNs.


http://arxiv.org/abs/2503.12497
Defense Against Model Stealing Based on Account-Aware Distribution Discrepancy. (1%)
Jian-Ping Mei; Weibin Zhang; Jie Chen; Xuyun Zhang; Tiantian Zhu
  Malicious users attempt to replicate commercial models functionally at low
cost by training a clone model with query responses. It is challenging to
timely prevent such model-stealing attacks to achieve strong protection and
maintain utility. In this paper, we propose a novel non-parametric detector
called Account-aware Distribution Discrepancy (ADD) to recognize queries from
malicious users by leveraging account-wise local dependency. We formulate each
class as a Multivariate Normal distribution (MVN) in the feature space and
measure the malicious score as the sum of weighted class-wise distribution
discrepancy. The ADD detector is combined with random-based prediction
poisoning to yield a plug-and-play defense module named D-ADD for image
classification models. Results of extensive experimental studies show that
D-ADD achieves strong defense against different types of attacks with little
interference in serving benign users for both soft and hard-label settings.


http://arxiv.org/abs/2503.12069
Robust Dataset Distillation by Matching Adversarial Trajectories. (75%)
Wei Lai; Tianyu Ding; ren dongdong; Lei Wang; Jing Huo; Yang Gao; Wenbin Li
  Dataset distillation synthesizes compact datasets that enable models to
achieve performance comparable to training on the original large-scale
datasets. However, existing distillation methods overlook the robustness of the
model, resulting in models that are vulnerable to adversarial attacks when
trained on distilled data. To address this limitation, we introduce the task of
``robust dataset distillation", a novel paradigm that embeds adversarial
robustness into the synthetic datasets during the distillation process. We
propose Matching Adversarial Trajectories (MAT), a method that integrates
adversarial training into trajectory-based dataset distillation. MAT
incorporates adversarial samples during trajectory generation to obtain robust
training trajectories, which are then used to guide the distillation process.
As experimentally demonstrated, even through natural training on our distilled
dataset, models can achieve enhanced adversarial robustness while maintaining
competitive accuracy compared to existing distillation methods. Our work
highlights robust dataset distillation as a new and important research
direction and provides a strong baseline for future research to bridge the gap
between efficient training and adversarial robustness.


http://arxiv.org/abs/2503.12058
Revisiting Training-Inference Trigger Intensity in Backdoor Attacks. (9%)
Chenhao Lin; Chenyang Zhao; Shiwei Wang; Longtian Wang; Chao Shen; Zhengyu Zhao
  Backdoor attacks typically place a specific trigger on certain training data,
such that the model makes prediction errors on inputs with that trigger during
inference. Despite the core role of the trigger, existing studies have commonly
believed a perfect match between training-inference triggers is optimal. In
this paper, for the first time, we systematically explore the
training-inference trigger relation, particularly focusing on their mismatch,
based on a Training-Inference Trigger Intensity Manipulation (TITIM) workflow.
TITIM specifically investigates the training-inference trigger intensity, such
as the size or the opacity of a trigger, and reveals new insights into trigger
generalization and overfitting.
  These new insights challenge the above common belief by demonstrating that
the training-inference trigger mismatch can facilitate attacks in two practical
scenarios, posing more significant security threats than previously thought.
First, when the inference trigger is fixed, using training triggers with mixed
intensities leads to stronger attacks than using any single intensity. For
example, on CIFAR-10 with ResNet-18, mixing training triggers with 1.0 and 0.1
opacities improves the worst-case attack success rate (ASR) (over different
testing opacities) of the best single-opacity attack from 10.61\% to 92.77\%.
Second, intentionally using certain mismatched training-inference triggers can
improve the attack stealthiness, i.e., better bypassing defenses. For example,
compared to the training/inference intensity of 1.0/1.0, using 1.0/0.7
decreases the area under the curve (AUC) of the Scale-Up defense from 0.96 to
0.62, while maintaining a high attack ASR (99.65\% vs. 91.62\%). The above new
insights are validated to be generalizable across different backdoor attacks,
models, datasets, tasks, and (digital/physical) domains.


http://arxiv.org/abs/2503.11619
Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense. (80%)
Shuyang Hao; Yiwei Wang; Bryan Hooi; Ming-Hsuan Yang; Jun Liu; Chengcheng Tang; Zi Huang; Yujun Cai
  Deploying large vision-language models (LVLMs) introduces a unique
vulnerability: susceptibility to malicious attacks via visual inputs. However,
existing defense methods suffer from two key limitations: (1) They solely focus
on textual defenses, fail to directly address threats in the visual domain
where attacks originate, and (2) the additional processing steps often incur
significant computational overhead or compromise model performance on benign
tasks. Building on these insights, we propose ESIII (Embedding Security
Instructions Into Images), a novel methodology for transforming the visual
space from a source of vulnerability into an active defense mechanism.
Initially, we embed security instructions into defensive images through
gradient-based optimization, obtaining security instructions in the visual
dimension. Subsequently, we integrate security instructions from visual and
textual dimensions with the input query. The collaboration between security
instructions from different dimensions ensures comprehensive security
protection. Extensive experiments demonstrate that our approach effectively
fortifies the robustness of LVLMs against such attacks while preserving their
performance on standard benign tasks and incurring an imperceptible increase in
time costs.


http://arxiv.org/abs/2503.11627
Are Deep Speech Denoising Models Robust to Adversarial Noise? (67%)
Will Schwarzer; Philip S. Thomas; Andrea Fanelli; Xiaoyu Liu
  Deep noise suppression (DNS) models enjoy widespread use throughout a variety
of high-stakes speech applications. However, in this paper, we show that four
recent DNS models can each be reduced to outputting unintelligible gibberish
through the addition of imperceptible adversarial noise. Furthermore, our
results show the near-term plausibility of targeted attacks, which could induce
models to output arbitrary utterances, and over-the-air attacks. While the
success of these attacks varies by model and setting, and attacks appear to be
strongest when model-specific (i.e., white-box and non-transferable), our
results highlight a pressing need for practical countermeasures in DNS systems.


http://arxiv.org/abs/2503.11841
Trust Under Siege: Label Spoofing Attacks against Machine Learning for Android Malware Detection. (64%)
Tianwei Lan; Luca Demetrio; Farid Nait-Abdesselam; Yufei Han; Simone Aonzo
  Machine learning (ML) malware detectors rely heavily on crowd-sourced
AntiVirus (AV) labels, with platforms like VirusTotal serving as a trusted
source of malware annotations. But what if attackers could manipulate these
labels to classify benign software as malicious? We introduce label spoofing
attacks, a new threat that contaminates crowd-sourced datasets by embedding
minimal and undetectable malicious patterns into benign samples. These patterns
coerce AV engines into misclassifying legitimate files as harmful, enabling
poisoning attacks against ML-based malware classifiers trained on those data.
We demonstrate this scenario by developing AndroVenom, a methodology for
polluting realistic data sources, causing consequent poisoning attacks against
ML malware detectors. Experiments show that not only state-of-the-art feature
extractors are unable to filter such injection, but also various ML models
experience Denial of Service already with 1% poisoned samples. Additionally,
attackers can flip decisions of specific unaltered benign samples by modifying
only 0.015% of the training data, threatening their reputation and market share
and being unable to be stopped by anomaly detectors on training data. We
conclude our manuscript by raising the alarm on the trustworthiness of the
training process based on AV annotations, requiring further investigation on
how to produce proper labels for ML malware detectors.


http://arxiv.org/abs/2503.11185
Align in Depth: Defending Jailbreak Attacks via Progressive Answer Detoxification. (62%)
Yingjie Zhang; Tong Liu; Zhe Zhao; Guozhu Meng; Kai Chen
  Large Language Models (LLMs) are vulnerable to jailbreak attacks, which use
crafted prompts to elicit toxic responses. These attacks exploit LLMs'
difficulty in dynamically detecting harmful intents during the generation
process. Traditional safety alignment methods, often relying on the initial few
generation steps, are ineffective due to limited computational budget. This
paper proposes DEEPALIGN, a robust defense framework that fine-tunes LLMs to
progressively detoxify generated content, significantly improving both the
computational budget and effectiveness of mitigating harmful generation. Our
approach uses a hybrid loss function operating on hidden states to directly
improve LLMs' inherent awareness of toxity during generation. Furthermore, we
redefine safe responses by generating semantically relevant answers to harmful
queries, thereby increasing robustness against representation-mutation attacks.
Evaluations across multiple LLMs demonstrate state-of-the-art defense
performance against six different attack types, reducing Attack Success Rates
by up to two orders of magnitude compared to previous state-of-the-art defense
while preserving utility. This work advances LLM safety by addressing
limitations of conventional alignment through dynamic, context-aware
mitigation.


http://arxiv.org/abs/2503.11646
Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learning. (15%)
Siyuan Huang; Yue Liao; Siyuan Feng; Shu Jiang; Si Liu; Hongsheng Li; Maoqing Yao; Guanghui Ren
  The pursuit of data efficiency, where quality outweighs quantity, has emerged
as a cornerstone in robotic manipulation, especially given the high costs
associated with real-world data collection. We propose that maximizing the
informational density of individual demonstrations can dramatically reduce
reliance on large-scale datasets while improving task performance. To this end,
we introduce Adversarial Data Collection, a Human-in-the-Loop (HiL) framework
that redefines robotic data acquisition through real-time, bidirectional
human-environment interactions. Unlike conventional pipelines that passively
record static demonstrations, ADC adopts a collaborative perturbation paradigm:
during a single episode, an adversarial operator dynamically alters object
states, environmental conditions, and linguistic commands, while the
tele-operator adaptively adjusts actions to overcome these evolving challenges.
This process compresses diverse failure-recovery behaviors, compositional task
variations, and environmental perturbations into minimal demonstrations. Our
experiments demonstrate that ADC-trained models achieve superior compositional
generalization to unseen task instructions, enhanced robustness to perceptual
perturbations, and emergent error recovery capabilities. Strikingly, models
trained with merely 20% of the demonstration volume collected through ADC
significantly outperform traditional approaches using full datasets. These
advances bridge the gap between data-centric learning paradigms and practical
robotic deployment, demonstrating that strategic data acquisition, not merely
post-hoc processing, is critical for scalable, real-world robot learning.
Additionally, we are curating a large-scale ADC-Robotics dataset comprising
real-world manipulation tasks with adversarial perturbations. This benchmark
will be open-sourced to facilitate advancements in robotic imitation learning.


http://arxiv.org/abs/2503.11917
A Framework for Evaluating Emerging Cyberattack Capabilities of AI. (1%)
Mikel Rodriguez; Raluca Ada Popa; Four Flynn; Lihao Liang; Allan Dafoe; Anna Wang
  As frontier models become more capable, the community has attempted to
evaluate their ability to enable cyberattacks. Performing a comprehensive
evaluation and prioritizing defenses are crucial tasks in preparing for AGI
safely. However, current cyber evaluation efforts are ad-hoc, with no
systematic reasoning about the various phases of attacks, and do not provide a
steer on how to use targeted defenses. In this work, we propose a novel
approach to AI cyber capability evaluation that (1) examines the end-to-end
attack chain, (2) helps to identify gaps in the evaluation of AI threats, and
(3) helps defenders prioritize targeted mitigations and conduct AI-enabled
adversary emulation to support red teaming. To achieve these goals, we propose
adapting existing cyberattack chain frameworks to AI systems. We analyze over
12,000 instances of real-world attempts to use AI in cyberattacks catalogued by
Google's Threat Intelligence Group. Using this analysis, we curate a
representative collection of seven cyberattack chain archetypes and conduct a
bottleneck analysis to identify areas of potential AI-driven cost disruption.
Our evaluation benchmark consists of 50 new challenges spanning different
phases of cyberattacks. Based on this, we devise targeted cybersecurity model
evaluations, report on the potential for AI to amplify offensive cyber
capabilities across specific attack phases, and conclude with recommendations
on prioritizing defenses. In all, we consider this to be the most comprehensive
AI cyber risk evaluation framework published so far.


http://arxiv.org/abs/2503.10635
A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1. (99%)
Zhaoyi Li; Xiaohan Zhao; Dong-Dong Wu; Jiacheng Cui; Zhiqiang Shen
  Despite promising performance on open-source large vision-language models
(LVLMs), transfer-based targeted attacks often fail against black-box
commercial LVLMs. Analyzing failed adversarial perturbations reveals that the
learned perturbations typically originate from a uniform distribution and lack
clear semantic details, resulting in unintended responses. This critical
absence of semantic information leads commercial LVLMs to either ignore the
perturbation entirely or misinterpret its embedded semantics, thereby causing
the attack to fail. To overcome these issues, we notice that identifying core
semantic objects is a key objective for models trained with various datasets
and methodologies. This insight motivates our approach that refines semantic
clarity by encoding explicit semantic details within local regions, thus
ensuring interoperability and capturing finer-grained features, and by
concentrating modifications on semantically rich areas rather than applying
them uniformly. To achieve this, we propose a simple yet highly effective
solution: at each optimization step, the adversarial image is cropped randomly
by a controlled aspect ratio and scale, resized, and then aligned with the
target image in the embedding space. Experimental results confirm our
hypothesis. Our adversarial examples crafted with local-aggregated
perturbations focused on crucial regions exhibit surprisingly good
transferability to commercial LVLMs, including GPT-4.5, GPT-4o,
Gemini-2.0-flash, Claude-3.5-sonnet, Claude-3.7-sonnet, and even reasoning
models like o1, Claude-3.7-thinking and Gemini-2.0-flash-thinking. Our approach
achieves success rates exceeding 90% on GPT-4.5, 4o, and o1, significantly
outperforming all prior state-of-the-art attack methods. Our optimized
adversarial examples under different configurations and training code are
available at https://github.com/VILA-Lab/M-Attack.


http://arxiv.org/abs/2503.10629
Hierarchical Self-Supervised Adversarial Training for Robust Vision Models in Histopathology. (98%)
Hashmat Shadab Malik; Shahina Kunhimon; Muzammal Naseer; Fahad Shahbaz Khan; Salman Khan
  Adversarial attacks pose significant challenges for vision models in critical
fields like healthcare, where reliability is essential. Although adversarial
training has been well studied in natural images, its application to biomedical
and microscopy data remains limited. Existing self-supervised adversarial
training methods overlook the hierarchical structure of histopathology images,
where patient-slide-patch relationships provide valuable discriminative
signals. To address this, we propose Hierarchical Self-Supervised Adversarial
Training (HSAT), which exploits these properties to craft adversarial examples
using multi-level contrastive learning and integrate it into adversarial
training for enhanced robustness. We evaluate HSAT on multiclass histopathology
dataset OpenSRH and the results show that HSAT outperforms existing methods
from both biomedical and natural image domains. HSAT enhances robustness,
achieving an average gain of 54.31% in the white-box setting and reducing
performance drops to 3-4% in the black-box setting, compared to 25-30% for the
baseline. These results set a new benchmark for adversarial training in this
domain, paving the way for more robust models. Our Code for training and
evaluation is available at https://github.com/HashmatShadab/HSAT.


http://arxiv.org/abs/2503.11032
Weakly Supervised Contrastive Adversarial Training for Learning Robust Features from Semi-supervised Data. (92%)
Lilin Zhang; Chengpei Wu; Ning Yang
  Existing adversarial training (AT) methods often suffer from incomplete
perturbation, meaning that not all non-robust features are perturbed when
generating adversarial examples (AEs). This results in residual correlations
between non-robust features and labels, leading to suboptimal learning of
robust features. However, achieving complete perturbation, i.e., perturbing as
many non-robust features as possible, is challenging due to the difficulty in
distinguishing robust and non-robust features and the sparsity of labeled data.
To address these challenges, we propose a novel approach called Weakly
Supervised Contrastive Adversarial Training (WSCAT). WSCAT ensures complete
perturbation for improved learning of robust features by disrupting
correlations between non-robust features and labels through complete AE
generation over partially labeled data, grounded in information theory.
Extensive theoretical analysis and comprehensive experiments on widely adopted
benchmarks validate the superiority of WSCAT.


http://arxiv.org/abs/2503.10191
Robustness Tokens: Towards Adversarial Robustness of Transformers. (83%)
Brian Pulfer; Yury Belousov; Slava Voloshynovskiy
  Recently, large pre-trained foundation models have become widely adopted by
machine learning practitioners for a multitude of tasks. Given that such models
are publicly available, relying on their use as backbone models for downstream
tasks might result in high vulnerability to adversarial attacks crafted with
the same public model. In this work, we propose Robustness Tokens, a novel
approach specific to the transformer architecture that fine-tunes a few
additional private tokens with low computational requirements instead of tuning
model parameters as done in traditional adversarial training. We show that
Robustness Tokens make Vision Transformer models significantly more robust to
white-box adversarial attacks while also retaining the original downstream
performances.


http://arxiv.org/abs/2503.10081
AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption. (47%)
Joonsung Jeon; Woo Jae Kim; Suhyeon Ha; Sooel Son; Sung-eui Yoon
  The outstanding capability of diffusion models in generating high-quality
images poses significant threats when misused by adversaries. In particular, we
assume malicious adversaries exploiting diffusion models for inpainting tasks,
such as replacing a specific region with a celebrity. While existing methods
for protecting images from manipulation in diffusion-based generative models
have primarily focused on image-to-image and text-to-image tasks, the challenge
of preventing unauthorized inpainting has been rarely addressed, often
resulting in suboptimal protection performance. To mitigate inpainting abuses,
we propose ADVPAINT, a novel defensive framework that generates adversarial
perturbations that effectively disrupt the adversary's inpainting tasks.
ADVPAINT targets the self- and cross-attention blocks in a target diffusion
inpainting model to distract semantic understanding and prompt interactions
during image generation. ADVPAINT also employs a two-stage perturbation
strategy, dividing the perturbation region based on an enlarged bounding box
around the object, enhancing robustness across diverse masks of varying shapes
and sizes. Our experimental results demonstrate that ADVPAINT's perturbations
are highly effective in disrupting the adversary's inpainting tasks,
outperforming existing methods; ADVPAINT attains over a 100-point increase in
FID and substantial decreases in precision.


http://arxiv.org/abs/2503.11514
Exploring the Vulnerabilities of Federated Learning: A Deep Dive into Gradient Inversion Attacks. (47%)
Pengxin Guo; Runxi Wang; Shuang Zeng; Jinjing Zhu; Haoning Jiang; Yanran Wang; Yuyin Zhou; Feifei Wang; Hui Xiong; Liangqiong Qu
  Federated Learning (FL) has emerged as a promising privacy-preserving
collaborative model training paradigm without sharing raw data. However, recent
studies have revealed that private information can still be leaked through
shared gradient information and attacked by Gradient Inversion Attacks (GIA).
While many GIA methods have been proposed, a detailed analysis, evaluation, and
summary of these methods are still lacking. Although various survey papers
summarize existing privacy attacks in FL, few studies have conducted extensive
experiments to unveil the effectiveness of GIA and their associated limiting
factors in this context. To fill this gap, we first undertake a systematic
review of GIA and categorize existing methods into three types, i.e.,
\textit{optimization-based} GIA (OP-GIA), \textit{generation-based} GIA
(GEN-GIA), and \textit{analytics-based} GIA (ANA-GIA). Then, we comprehensively
analyze and evaluate the three types of GIA in FL, providing insights into the
factors that influence their performance, practicality, and potential threats.
Our findings indicate that OP-GIA is the most practical attack setting despite
its unsatisfactory performance, while GEN-GIA has many dependencies and ANA-GIA
is easily detectable, making them both impractical. Finally, we offer a
three-stage defense pipeline to users when designing FL frameworks and
protocols for better privacy protection and share some future research
directions from the perspectives of attackers and defenders that we believe
should be pursued. We hope that our study can help researchers design more
robust FL frameworks to defend against these attacks.


http://arxiv.org/abs/2503.10872
TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models. (11%)
Xiangyu Yin; Yi Qi; Jinwei Hu; Zhen Chen; Yi Dong; Xingyu Zhao; Xiaowei Huang; Wenjie Ruan
  Vision Language Models (VLMs) have demonstrated impressive inference
capabilities, but remain vulnerable to jailbreak attacks that can induce
harmful or unethical responses. Existing defence methods are predominantly
white-box approaches that require access to model parameters and extensive
modifications, making them costly and impractical for many real-world
scenarios. Although some black-box defences have been proposed, they often
impose input constraints or require multiple queries, limiting their
effectiveness in safety-critical tasks such as autonomous driving. To address
these challenges, we propose a novel black-box defence framework called
\textbf{T}extual \textbf{A}nchoring for \textbf{I}mmunizing \textbf{J}ailbreak
\textbf{I}mages (\textbf{TAIJI}). TAIJI leverages key phrase-based textual
anchoring to enhance the model's ability to assess and mitigate the harmful
content embedded within both visual and textual prompts. Unlike existing
methods, TAIJI operates effectively with a single query during inference, while
preserving the VLM's performance on benign tasks. Extensive experiments
demonstrate that TAIJI significantly enhances the safety and reliability of
VLMs, providing a practical and efficient solution for real-world deployment.


http://arxiv.org/abs/2503.10350
Enhancing Facial Privacy Protection via Weakening Diffusion Purification. (10%)
Ali Salar; Qing Liu; Yingli Tian; Guoying Zhao
  The rapid growth of social media has led to the widespread sharing of
individual portrait images, which pose serious privacy risks due to the
capabilities of automatic face recognition (AFR) systems for mass surveillance.
Hence, protecting facial privacy against unauthorized AFR systems is essential.
Inspired by the generation capability of the emerging diffusion models, recent
methods employ diffusion models to generate adversarial face images for privacy
protection. However, they suffer from the diffusion purification effect,
leading to a low protection success rate (PSR). In this paper, we first propose
learning unconditional embeddings to increase the learning capacity for
adversarial modifications and then use them to guide the modification of the
adversarial latent code to weaken the diffusion purification effect. Moreover,
we integrate an identity-preserving structure to maintain structural
consistency between the original and generated images, allowing human observers
to recognize the generated image as having the same identity as the original.
Extensive experiments conducted on two public datasets, i.e., CelebA-HQ and
LADN, demonstrate the superiority of our approach. The protected faces
generated by our method outperform those produced by existing facial privacy
protection approaches in terms of transferability and natural appearance.


http://arxiv.org/abs/2503.10846
WAFFLED: Exploiting Parsing Discrepancies to Bypass Web Application Firewalls. (2%)
Seyed Ali Akhavani; Bahruz Jabiyev; Ben Kallus; Cem Topcuoglu; Sergey Bratus; Engin Kirda
  Web Application Firewalls (WAFs) have been introduced as essential and
popular security gates that inspect incoming HTTP traffic to filter out
malicious requests and provide defenses against a diverse array of web-based
threats. Evading WAFs can compromise these defenses, potentially harming
Internet users. In recent years, parsing discrepancies have plagued many
entities in the communication path; however, their potential impact on WAF
evasion and request smuggling remains largely unexplored. In this work, we
present an innovative approach to bypassing WAFs by uncovering and exploiting
parsing discrepancies through advanced fuzzing techniques. By targeting
non-malicious components such as headers and segments of the body and using
widely used content-types such as application/json, multipart/form-data, and
application/xml, we identified and confirmed 1207 bypasses across 5 well-known
WAFs, AWS, Azure, Cloud Armor, Cloudflare, and ModSecurity. To validate our
findings, we conducted a study in the wild, revealing that more than 90% of
websites accepted both application/x-www-form-urlencoded and
multipart/form-data interchangeably, highlighting a significant vulnerability
and the broad applicability of our bypass techniques. We have reported these
vulnerabilities to the affected parties and received acknowledgments from all,
as well as bug bounty rewards from some vendors. Further, to mitigate these
vulnerabilities, we introduce HTTP-Normalizer, a robust proxy tool designed to
rigorously validate HTTP requests against current RFC standards. Our results
demonstrate its effectiveness in normalizing or blocking all bypass attempts
presented in this work.


http://arxiv.org/abs/2503.10912
JPEG Compliant Compression for Both Human and Machine, A Report. (1%)
Linfeng Ye
  Deep Neural Networks (DNNs) have become an integral part of our daily lives,
especially in vision-related applications. However, the conventional lossy
image compression algorithms are primarily designed for the Human Vision System
(HVS), which can non-trivially compromise the DNNs' validation accuracy after
compression, as noted in \cite{liu2018deepn}. Thus developing an image
compression algorithm for both human and machine (DNNs) is on the horizon.
  To address the challenge mentioned above, in this paper, we first formulate
the image compression as a multi-objective optimization problem which take both
human and machine prespectives into account, then we solve it by linear
combination, and proposed a novel distortion measure for both human and
machine, dubbed Human and Machine-Oriented Error (HMOE). After that, we develop
Human And Machine Oriented Soft Decision Quantization (HMOSDQ) based on HMOE, a
lossy image compression algorithm for both human and machine (DNNs), and fully
complied with JPEG format. In order to evaluate the performance of HMOSDQ,
finally we conduct the experiments for two pre-trained well-known DNN-based
image classifiers named Alexnet \cite{Alexnet} and VGG-16
\cite{simonyan2014VGG} on two subsets of the ImageNet \cite{deng2009imagenet}
validation set: one subset included images with shorter side in the range of
496 to 512, while the other included images with shorter side in the range of
376 to 384. Our results demonstrate that HMOSDQ outperforms the default JPEG
algorithm in terms of rate-accuracy and rate-distortion performance. For the
Alexnet comparing with the default JPEG algorithm, HMOSDQ can improve the
validation accuracy by more than $0.81\%$ at $0.61$ BPP, or equivalently reduce
the compression rate of default JPEG by $9.6\times$ while maintaining the same
validation accuracy.


http://arxiv.org/abs/2503.09735
Enhancing Adversarial Example Detection Through Model Explanation. (99%)
Qian Ma; Ziping Ye
  Adversarial examples are a major problem for machine learning models, leading
to a continuous search for effective defenses. One promising direction is to
leverage model explanations to better understand and defend against these
attacks. We looked at AmI, a method proposed by a NeurIPS 2018 spotlight paper
that uses model explanations to detect adversarial examples. Our study shows
that while AmI is a promising idea, its performance is too dependent on
specific settings (e.g., hyperparameter) and external factors such as the
operating system and the deep learning framework used, and such drawbacks limit
AmI's practical usage. Our findings highlight the need for more robust defense
mechanisms that are effective under various conditions. In addition, we
advocate for a comprehensive evaluation framework for defense techniques.


http://arxiv.org/abs/2503.09124
AdvAD: Exploring Non-Parametric Diffusion for Imperceptible Adversarial Attacks. (99%)
Jin Li; Ziqiang He; Anwei Luo; Jian-Fang Hu; Z. Jane Wang; Xiangui Kang
  Imperceptible adversarial attacks aim to fool DNNs by adding imperceptible
perturbation to the input data. Previous methods typically improve the
imperceptibility of attacks by integrating common attack paradigms with
specifically designed perception-based losses or the capabilities of generative
models. In this paper, we propose Adversarial Attacks in Diffusion (AdvAD), a
novel modeling framework distinct from existing attack paradigms. AdvAD
innovatively conceptualizes attacking as a non-parametric diffusion process by
theoretically exploring basic modeling approach rather than using the denoising
or generation abilities of regular diffusion models requiring neural networks.
At each step, much subtler yet effective adversarial guidance is crafted using
only the attacked model without any additional network, which gradually leads
the end of diffusion process from the original image to a desired imperceptible
adversarial example. Grounded in a solid theoretical foundation of the proposed
non-parametric diffusion process, AdvAD achieves high attack efficacy and
imperceptibility with intrinsically lower overall perturbation strength.
Additionally, an enhanced version AdvAD-X is proposed to evaluate the extreme
of our novel framework under an ideal scenario. Extensive experiments
demonstrate the effectiveness of the proposed AdvAD and AdvAD-X. Compared with
state-of-the-art imperceptible attacks, AdvAD achieves an average of 99.9$\%$
(+17.3$\%$) ASR with 1.34 (-0.97) $l_2$ distance, 49.74 (+4.76) PSNR and 0.9971
(+0.0043) SSIM against four prevalent DNNs with three different architectures
on the ImageNet-compatible dataset. Code is available at
https://github.com/XianguiKang/AdvAD.


http://arxiv.org/abs/2503.09334
CyberLLMInstruct: A New Dataset for Analysing Safety of Fine-Tuned LLMs Using Cyber Security Data. (83%)
Adel ElZemity; Budi Arief; Shujun Li
  The integration of large language models (LLMs) into cyber security
applications presents significant opportunities, such as enhancing threat
analysis and malware detection, but can also introduce critical risks and
safety concerns, including personal data leakage and automated generation of
new malware. To address these challenges, we developed CyberLLMInstruct, a
dataset of 54,928 instruction-response pairs spanning cyber security tasks such
as malware analysis, phishing simulations, and zero-day vulnerabilities. The
dataset was constructed through a multi-stage process. This involved sourcing
data from multiple resources, filtering and structuring it into
instruction-response pairs, and aligning it with real-world scenarios to
enhance its applicability. Seven open-source LLMs were chosen to test the
usefulness of CyberLLMInstruct: Phi 3 Mini 3.8B, Mistral 7B, Qwen 2.5 7B, Llama
3 8B, Llama 3.1 8B, Gemma 2 9B, and Llama 2 70B. In our primary example, we
rigorously assess the safety of fine-tuned models using the OWASP top 10
framework, finding that fine-tuning reduces safety resilience across all tested
LLMs and every adversarial attack (e.g., the security score of Llama 3.1 8B
against prompt injection drops from 0.95 to 0.15). In our second example, we
show that these same fine-tuned models can also achieve up to 92.50 percent
accuracy on the CyberMetric benchmark. These findings highlight a trade-off
between performance and safety, showing the importance of adversarial testing
and further research into fine-tuning methodologies that can mitigate safety
risks while still improving performance across diverse datasets and domains.
The dataset creation pipeline, along with comprehensive documentation,
examples, and resources for reproducing our results, is publicly available at
https://github.com/Adelsamir01/CyberLLMInstruct.


http://arxiv.org/abs/2503.09095
C^2 ATTACK: Towards Representation Backdoor on CLIP via Concept Confusion. (80%)
Lijie Hu; Junchi Liao; Weimin Lyu; Shaopeng Fu; Tianhao Huang; Shu Yang; Guimin Hu; Di Wang
  Backdoor attacks pose a significant threat to deep learning models, enabling
adversaries to embed hidden triggers that manipulate the behavior of the model
during inference. Traditional backdoor attacks typically rely on inserting
explicit triggers (e.g., external patches, or perturbations) into input data,
but they often struggle to evade existing defense mechanisms. To address this
limitation, we investigate backdoor attacks through the lens of the reasoning
process in deep learning systems, drawing insights from interpretable AI. We
conceptualize backdoor activation as the manipulation of learned concepts
within the model's latent representations. Thus, existing attacks can be seen
as implicit manipulations of these activated concepts during inference. This
raises interesting questions: why not manipulate the concepts explicitly? This
idea leads to our novel backdoor attack framework, Concept Confusion Attack
(C^2 ATTACK), which leverages internal concepts in the model's reasoning as
"triggers" without introducing explicit external modifications. By avoiding the
use of real triggers and directly activating or deactivating specific concepts
in latent spaces, our approach enhances stealth, making detection by existing
defenses significantly harder. Using CLIP as a case study, experimental results
demonstrate the effectiveness of C^2 ATTACK, achieving high attack success
rates while maintaining robustness against advanced defenses.


http://arxiv.org/abs/2503.09712
Revisiting Backdoor Attacks on Time Series Classification in the Frequency Domain. (76%)
Yuanmin Huang; Mi Zhang; Zhaoxiang Wang; Wenxuan Li; Min Yang
  Time series classification (TSC) is a cornerstone of modern web applications,
powering tasks such as financial data analysis, network traffic monitoring, and
user behavior analysis. In recent years, deep neural networks (DNNs) have
greatly enhanced the performance of TSC models in these critical domains.
However, DNNs are vulnerable to backdoor attacks, where attackers can covertly
implant triggers into models to induce malicious outcomes. Existing backdoor
attacks targeting DNN-based TSC models remain elementary. In particular, early
methods borrow trigger designs from computer vision, which are ineffective for
time series data. More recent approaches utilize generative models for trigger
generation, but at the cost of significant computational complexity. In this
work, we analyze the limitations of existing attacks and introduce an enhanced
method, FreqBack. Drawing inspiration from the fact that DNN models inherently
capture frequency domain features in time series data, we identify that
improper perturbations in the frequency domain are the root cause of
ineffective attacks. To address this, we propose to generate triggers both
effectively and efficiently, guided by frequency analysis. FreqBack exhibits
substantial performance across five models and eight datasets, achieving an
impressive attack success rate of over 90%, while maintaining less than a 3%
drop in model accuracy on clean data.


http://arxiv.org/abs/2503.09066
Probing Latent Subspaces in LLM for AI Security: Identifying and Manipulating Adversarial States. (75%)
Xin Wei Chia; Jonathan Pan
  Large Language Models (LLMs) have demonstrated remarkable capabilities across
various tasks, yet they remain vulnerable to adversarial manipulations such as
jailbreaking via prompt injection attacks. These attacks bypass safety
mechanisms to generate restricted or harmful content. In this study, we
investigated the underlying latent subspaces of safe and jailbroken states by
extracting hidden activations from a LLM. Inspired by attractor dynamics in
neuroscience, we hypothesized that LLM activations settle into semi stable
states that can be identified and perturbed to induce state transitions. Using
dimensionality reduction techniques, we projected activations from safe and
jailbroken responses to reveal latent subspaces in lower dimensional spaces. We
then derived a perturbation vector that when applied to safe representations,
shifted the model towards a jailbreak state. Our results demonstrate that this
causal intervention results in statistically significant jailbreak responses in
a subset of prompts. Next, we probed how these perturbations propagate through
the model's layers, testing whether the induced state change remains localized
or cascades throughout the network. Our findings indicate that targeted
perturbations induced distinct shifts in activations and model responses. Our
approach paves the way for potential proactive defenses, shifting from
traditional guardrail based methods to preemptive, model agnostic techniques
that neutralize adversarial states at the representation level.


http://arxiv.org/abs/2503.09302
Detecting and Preventing Data Poisoning Attacks on AI Models. (75%)
Halima I. Kure; Pradipta Sarkar; Ahmed B. Ndanusa; Augustine O. Nwajana
  This paper investigates the critical issue of data poisoning attacks on AI
models, a growing concern in the ever-evolving landscape of artificial
intelligence and cybersecurity. As advanced technology systems become
increasingly prevalent across various sectors, the need for robust defence
mechanisms against adversarial attacks becomes paramount. The study aims to
develop and evaluate novel techniques for detecting and preventing data
poisoning attacks, focusing on both theoretical frameworks and practical
applications. Through a comprehensive literature review, experimental
validation using the CIFAR-10 and Insurance Claims datasets, and the
development of innovative algorithms, this paper seeks to enhance the
resilience of AI models against malicious data manipulation. The study explores
various methods, including anomaly detection, robust optimization strategies,
and ensemble learning, to identify and mitigate the effects of poisoned data
during model training. Experimental results indicate that data poisoning
significantly degrades model performance, reducing classification accuracy by
up to 27% in image recognition tasks (CIFAR-10) and 22% in fraud detection
models (Insurance Claims dataset). The proposed defence mechanisms, including
statistical anomaly detection and adversarial training, successfully mitigated
poisoning effects, improving model robustness and restoring accuracy levels by
an average of 15-20%. The findings further demonstrate that ensemble learning
techniques provide an additional layer of resilience, reducing false positives
and false negatives caused by adversarial data injections.


http://arxiv.org/abs/2503.09049
Adaptive Backdoor Attacks with Reasonable Constraints on Graph Neural Networks. (64%)
Xuewen Dong; Jiachen Li; Shujun Li; Zhichao You; Qiang Qu; Yaroslav Kholodov; Yulong Shen
  Recent studies show that graph neural networks (GNNs) are vulnerable to
backdoor attacks. Existing backdoor attacks against GNNs use fixed-pattern
triggers and lack reasonable trigger constraints, overlooking individual graph
characteristics and rendering insufficient evasiveness. To tackle the above
issues, we propose ABARC, the first Adaptive Backdoor Attack with Reasonable
Constraints, applying to both graph-level and node-level tasks in GNNs. For
graph-level tasks, we propose a subgraph backdoor attack independent of the
graph's topology. It dynamically selects trigger nodes for each target graph
and modifies node features with constraints based on graph similarity, feature
range, and feature type. For node-level tasks, our attack begins with an
analysis of node features, followed by selecting and modifying trigger
features, which are then constrained by node similarity, feature range, and
feature type. Furthermore, an adaptive edge-pruning mechanism is designed to
reduce the impact of neighbors on target nodes, ensuring a high attack success
rate (ASR). Experimental results show that even with reasonable constraints for
attack evasiveness, our attack achieves a high ASR while incurring a marginal
clean accuracy drop (CAD). When combined with the state-of-the-art defense
randomized smoothing (RS) method, our attack maintains an ASR over 94%,
surpassing existing attacks by more than 7%.


http://arxiv.org/abs/2503.09241
In-Context Defense in Computer Agents: An Empirical Study. (56%)
Pei Yang; Hai Ci; Mike Zheng Shou
  Computer agents powered by vision-language models (VLMs) have significantly
advanced human-computer interaction, enabling users to perform complex tasks
through natural language instructions. However, these agents are vulnerable to
context deception attacks, an emerging threat where adversaries embed
misleading content into the agent's operational environment, such as a pop-up
window containing deceptive instructions. Existing defenses, such as
instructing agents to ignore deceptive elements, have proven largely
ineffective. As the first systematic study on protecting computer agents, we
introduce textbf{in-context defense}, leveraging in-context learning and
chain-of-thought (CoT) reasoning to counter such attacks. Our approach involves
augmenting the agent's context with a small set of carefully curated exemplars
containing both malicious environments and corresponding defensive responses.
These exemplars guide the agent to first perform explicit defensive reasoning
before action planning, reducing susceptibility to deceptive attacks.
Experiments demonstrate the effectiveness of our method, reducing attack
success rates by 91.2% on pop-up window attacks, 74.6% on average on
environment injection attacks, while achieving 100% successful defenses against
distracting advertisements. Our findings highlight that (1) defensive reasoning
must precede action planning for optimal performance, and (2) a minimal number
of exemplars (fewer than three) is sufficient to induce an agent's defensive
behavior.


http://arxiv.org/abs/2503.09726
How Feasible is Augmenting Fake Nodes with Learnable Features as a Counter-strategy against Link Stealing Attacks? (45%)
Mir Imtiaz Mostafiz; Imtiaz Karim; Elisa Bertino
  Graph Neural Networks (GNNs) are widely used and deployed for graph-based
prediction tasks. However, as good as GNNs are for learning graph data, they
also come with the risk of privacy leakage. For instance, an attacker can run
carefully crafted queries on the GNNs and, from the responses, can infer the
existence of an edge between a pair of nodes. This attack, dubbed as a
"link-stealing" attack, can jeopardize the user's privacy by leaking
potentially sensitive information. To protect against this attack, we propose
an approach called "$(N)$ode $(A)$ugmentation for $(R)$estricting $(G)$raphs
from $(I)$nsinuating their $(S)$tructure" ($NARGIS$) and study its feasibility.
$NARGIS$ is focused on reshaping the graph embedding space so that the
posterior from the GNN model will still provide utility for the prediction task
but will introduce ambiguity for the link-stealing attackers. To this end,
$NARGIS$ applies spectral clustering on the given graph to facilitate it being
augmented with new nodes -- that have learned features instead of fixed ones.
It utilizes tri-level optimization for learning parameters for the GNN model,
surrogate attacker model, and our defense model (i.e. learnable node features).
We extensively evaluate $NARGIS$ on three benchmark citation datasets over
eight knowledge availability settings for the attackers. We also evaluate the
model fidelity and defense performance on influence-based link inference
attacks. Through our studies, we have figured out the best feature of $NARGIS$
-- its superior fidelity-privacy performance trade-off in a significant number
of cases. We also have discovered in which cases the model needs to be
improved, and proposed ways to integrate different schemes to make the model
more robust against link stealing attacks.


http://arxiv.org/abs/2503.09964
ExtremeAIGC: Benchmarking LMM Vulnerability to AI-Generated Extremist Content. (22%)
Bhavik Chandna; Mariam Aboujenane; Usman Naseem
  Large Multimodal Models (LMMs) are increasingly vulnerable to AI-generated
extremist content, including photorealistic images and text, which can be used
to bypass safety mechanisms and generate harmful outputs. However, existing
datasets for evaluating LMM robustness offer limited exploration of extremist
content, often lacking AI-generated images, diverse image generation models,
and comprehensive coverage of historical events, which hinders a complete
assessment of model vulnerabilities. To fill this gap, we introduce
ExtremeAIGC, a benchmark dataset and evaluation framework designed to assess
LMM vulnerabilities against such content. ExtremeAIGC simulates real-world
events and malicious use cases by curating diverse text- and image-based
examples crafted using state-of-the-art image generation techniques. Our study
reveals alarming weaknesses in LMMs, demonstrating that even cutting-edge
safety measures fail to prevent the generation of extremist material. We
systematically quantify the success rates of various attack strategies,
exposing critical gaps in current defenses and emphasizing the need for more
robust mitigation strategies.


http://arxiv.org/abs/2503.09513
RESTRAIN: Reinforcement Learning-Based Secure Framework for Trigger-Action IoT Environment. (2%)
Md Morshed Alam; Lokesh Chandra Das; Sandip Roy; Sachin Shetty; Weichao Wang
  Internet of Things (IoT) platforms with trigger-action capability allow event
conditions to trigger actions in IoT devices autonomously by creating a chain
of interactions. Adversaries exploit this chain of interactions to maliciously
inject fake event conditions into IoT hubs, triggering unauthorized actions on
target IoT devices to implement remote injection attacks. Existing defense
mechanisms focus mainly on the verification of event transactions using
physical event fingerprints to enforce the security policies to block unsafe
event transactions. These approaches are designed to provide offline defense
against injection attacks. The state-of-the-art online defense mechanisms offer
real-time defense, but extensive reliability on the inference of attack impacts
on the IoT network limits the generalization capability of these approaches. In
this paper, we propose a platform-independent multi-agent online defense
system, namely RESTRAIN, to counter remote injection attacks at runtime.
RESTRAIN allows the defense agent to profile attack actions at runtime and
leverages reinforcement learning to optimize a defense policy that complies
with the security requirements of the IoT network. The experimental results
show that the defense agent effectively takes real-time defense actions against
complex and dynamic remote injection attacks and maximizes the security gain
with minimal computational overhead.


http://arxiv.org/abs/2503.09669
Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models. (2%)
Sangwon Jang; June Suk Choi; Jaehyeong Jo; Kimin Lee; Sung Ju Hwang
  Text-to-image diffusion models have achieved remarkable success in generating
high-quality contents from text prompts. However, their reliance on publicly
available data and the growing trend of data sharing for fine-tuning make these
models particularly vulnerable to data poisoning attacks. In this work, we
introduce the Silent Branding Attack, a novel data poisoning method that
manipulates text-to-image diffusion models to generate images containing
specific brand logos or symbols without any text triggers. We find that when
certain visual patterns are repeatedly in the training data, the model learns
to reproduce them naturally in its outputs, even without prompt mentions.
Leveraging this, we develop an automated data poisoning algorithm that
unobtrusively injects logos into original images, ensuring they blend naturally
and remain undetected. Models trained on this poisoned dataset generate images
containing logos without degrading image quality or text alignment. We
experimentally validate our silent branding attack across two realistic
settings on large-scale high-quality image datasets and style personalization
datasets, achieving high success rates even without a specific text trigger.
Human evaluation and quantitative metrics including logo detection show that
our method can stealthily embed logos.


http://arxiv.org/abs/2503.09661
Towards Hardware Supported Domain Generalization in DNN-Based Edge Computing Devices for Health Monitoring. (1%)
Johnson Loh; Lyubov Dudchenko; Justus Viga; Tobias Gemmeke
  Deep neural network (DNN) models have shown remarkable success in many
real-world scenarios, such as object detection and classification.
Unfortunately, these models are not yet widely adopted in health monitoring due
to exceptionally high requirements for model robustness and deployment in
highly resource-constrained devices. In particular, the acquisition of
biosignals, such as electrocardiogram (ECG), is subject to large variations
between training and deployment, necessitating domain generalization (DG) for
robust classification quality across sensors and patients. The continuous
monitoring of ECG also requires the execution of DNN models in convenient
wearable devices, which is achieved by specialized ECG accelerators with small
form factor and ultra-low power consumption. However, combining DG capabilities
with ECG accelerators remains a challenge. This article provides a
comprehensive overview of ECG accelerators and DG methods and discusses the
implication of the combination of both domains, such that multi-domain ECG
monitoring is enabled with emerging algorithm-hardware co-optimized systems.
Within this context, an approach based on correction layers is proposed to
deploy DG capabilities on the edge. Here, the DNN fine-tuning for unknown
domains is limited to a single layer, while the remaining DNN model remains
unmodified. Thus, computational complexity (CC) for DG is reduced with minimal
memory overhead compared to conventional fine-tuning of the whole DNN model.
The DNN model-dependent CC is reduced by more than 2.5x compared to DNN
fine-tuning at an average increase of F1 score by more than 20% on the
generalized target domain. In summary, this article provides a novel
perspective on robust DNN classification on the edge for health monitoring
applications.


http://arxiv.org/abs/2503.09446
Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models. (1%)
Zhihua Tian; Sirun Nan; Ming Xu; Shengfang Zhai; Wenjie Qu; Jian Liu; Kui Ren; Ruoxi Jia; Jiaheng Zhang
  Text-to-image (T2I) diffusion models have achieved remarkable progress in
generating high-quality images but also raise people's concerns about
generating harmful or misleading content. While extensive approaches have been
proposed to erase unwanted concepts without requiring retraining from scratch,
they inadvertently degrade performance on normal generation tasks. In this
work, we propose Interpret then Deactivate (ItD), a novel framework to enable
precise concept removal in T2I diffusion models while preserving overall
performance. ItD first employs a sparse autoencoder (SAE) to interpret each
concept as a combination of multiple features. By permanently deactivating the
specific features associated with target concepts, we repurpose SAE as a
zero-shot classifier that identifies whether the input prompt includes target
concepts, allowing selective concept erasure in diffusion models. Moreover, we
demonstrate that ItD can be easily extended to erase multiple concepts without
requiring further training. Comprehensive experiments across celebrity
identities, artistic styles, and explicit content demonstrate ItD's
effectiveness in eliminating targeted concepts without interfering with normal
concept generation. Additionally, ItD is also robust against adversarial
prompts designed to circumvent content filters. Code is available at:
https://github.com/NANSirun/Interpret-then-deactivate.


http://arxiv.org/abs/2503.09068
Probing Network Decisions: Capturing Uncertainties and Unveiling Vulnerabilities Without Label Information. (1%)
Youngju Joung; Sehyun Lee; Jaesik Choi
  To improve trust and transparency, it is crucial to be able to interpret the
decisions of Deep Neural classifiers (DNNs). Instance-level examinations, such
as attribution techniques, are commonly employed to interpret the model
decisions. However, when interpreting misclassified decisions, human
intervention may be required. Analyzing the attribu tions across each class
within one instance can be particularly labor intensive and influenced by the
bias of the human interpreter. In this paper, we present a novel framework to
uncover the weakness of the classifier via counterfactual examples. A prober is
introduced to learn the correctness of the classifier's decision in terms of
binary code-hit or miss. It enables the creation of the counterfactual example
concerning the prober's decision. We test the performance of our prober's
misclassification detection and verify its effectiveness on the image
classification benchmark datasets. Furthermore, by generating counterfactuals
that penetrate the prober, we demonstrate that our framework effectively
identifies vulnerabilities in the target classifier without relying on label
information on the MNIST dataset.


http://arxiv.org/abs/2503.09291
Prompt Inference Attack on Distributed Large Language Model Inference Frameworks. (1%)
Xinjian Luo; Ting Yu; Xiaokui Xiao
  The inference process of modern large language models (LLMs) demands
prohibitive computational resources, rendering them infeasible for deployment
on consumer-grade devices. To address this limitation, recent studies propose
distributed LLM inference frameworks, which employ split learning principles to
enable collaborative LLM inference on resource-constrained hardware. However,
distributing LLM layers across participants requires the transmission of
intermediate outputs, which may introduce privacy risks to the original input
prompts - a critical issue that has yet to be thoroughly explored in the
literature.
  In this paper, we rigorously examine the privacy vulnerabilities of
distributed LLM inference frameworks by designing and evaluating three prompt
inference attacks aimed at reconstructing input prompts from intermediate LLM
outputs. These attacks are developed under various query and data constraints
to reflect diverse real-world LLM service scenarios. Specifically, the first
attack assumes an unlimited query budget and access to an auxiliary dataset
sharing the same distribution as the target prompts. The second attack also
leverages unlimited queries but uses an auxiliary dataset with a distribution
differing from the target prompts. The third attack operates under the most
restrictive scenario, with limited query budgets and no auxiliary dataset
available. We evaluate these attacks on a range of LLMs, including
state-of-the-art models such as Llama-3.2 and Phi-3.5, as well as widely-used
models like GPT-2 and BERT for comparative analysis. Our experiments show that
the first two attacks achieve reconstruction accuracies exceeding 90%, while
the third achieves accuracies typically above 50%, even under stringent
constraints. These findings highlight privacy risks in distributed LLM
inference frameworks, issuing a strong alert on their deployment in real-world
applications.


http://arxiv.org/abs/2503.08973
Quantitative Analysis of Deeply Quantized Tiny Neural Networks Robust to Adversarial Attacks. (98%)
Idris Zakariyya; Ferheen Ayaz; Mounia Kharbouche-Harrari; Jeremy Singer; Sye Loong Keoh; Danilo Pau; José Cano
  Reducing the memory footprint of Machine Learning (ML) models, especially
Deep Neural Networks (DNNs), is imperative to facilitate their deployment on
resource-constrained edge devices. However, a notable drawback of DNN models
lies in their susceptibility to adversarial attacks, wherein minor input
perturbations can deceive them. A primary challenge revolves around the
development of accurate, resilient, and compact DNN models suitable for
deployment on resource-constrained edge devices. This paper presents the
outcomes of a compact DNN model that exhibits resilience against both black-box
and white-box adversarial attacks. This work has achieved this resilience
through training with the QKeras quantization-aware training framework. The
study explores the potential of QKeras and an adversarial robustness technique,
Jacobian Regularization (JR), to co-optimize the DNN architecture through
per-layer JR methodology. As a result, this paper has devised a DNN model
employing this co-optimization strategy based on Stochastic Ternary
Quantization (STQ). Its performance was compared against existing DNN models in
the face of various white-box and black-box attacks. The experimental findings
revealed that, the proposed DNN model had small footprint and on average, it
exhibited better performance than Quanos and DS-CNN MLCommons/TinyML (MLC/T)
benchmarks when challenged with white-box and black-box attacks, respectively,
on the CIFAR-10 image and Google Speech Commands audio datasets.


http://arxiv.org/abs/2503.08226
A Grey-box Text Attack Framework using Explainable AI. (92%)
Esther Chiramal; Kelvin Soh Boon Kai
  Explainable AI is a strong strategy implemented to understand complex
black-box model predictions in a human interpretable language. It provides the
evidence required to execute the use of trustworthy and reliable AI systems. On
the other hand, however, it also opens the door to locating possible
vulnerabilities in an AI model. Traditional adversarial text attack uses word
substitution, data augmentation techniques and gradient-based attacks on
powerful pre-trained Bidirectional Encoder Representations from Transformers
(BERT) variants to generate adversarial sentences. These attacks are generally
whitebox in nature and not practical as they can be easily detected by humans
E.g. Changing the word from "Poor" to "Rich". We proposed a simple yet
effective Grey-box cum Black-box approach that does not require the knowledge
of the model while using a set of surrogate Transformer/BERT models to perform
the attack using Explainable AI techniques. As Transformers are the current
state-of-the-art models for almost all Natural Language Processing (NLP) tasks,
an attack generated from BERT1 is transferable to BERT2. This transferability
is made possible due to the attention mechanism in the transformer that allows
the model to capture long-range dependencies in a sequence. Using the power of
BERT generalisation via attention, we attempt to exploit how transformers learn
by attacking a few surrogate transformer variants which are all based on a
different architecture. We demonstrate that this approach is highly effective
to generate semantically good sentences by changing as little as one word that
is not detectable by humans while still fooling other BERT models.


http://arxiv.org/abs/2503.10690
Battling Misinformation: An Empirical Study on Adversarial Factuality in Open-Source Large Language Models. (76%)
Shahnewaz Karim Sakib; Anindya Bijoy Das; Shibbir Ahmed
  Adversarial factuality refers to the deliberate insertion of misinformation
into input prompts by an adversary, characterized by varying levels of
expressed confidence. In this study, we systematically evaluate the performance
of several open-source large language models (LLMs) when exposed to such
adversarial inputs. Three tiers of adversarial confidence are considered:
strongly confident, moderately confident, and limited confidence. Our analysis
encompasses eight LLMs: LLaMA 3.1 (8B), Phi 3 (3.8B), Qwen 2.5 (7B),
Deepseek-v2 (16B), Gemma2 (9B), Falcon (7B), Mistrallite (7B), and LLaVA (7B).
Empirical results indicate that LLaMA 3.1 (8B) exhibits a robust capability in
detecting adversarial inputs, whereas Falcon (7B) shows comparatively lower
performance. Notably, for the majority of the models, detection success
improves as the adversary's confidence decreases; however, this trend is
reversed for LLaMA 3.1 (8B) and Phi 3 (3.8B), where a reduction in adversarial
confidence corresponds with diminished detection performance. Further analysis
of the queries that elicited the highest and lowest rates of successful attacks
reveals that adversarial attacks are more effective when targeting less
commonly referenced or obscure information.


http://arxiv.org/abs/2503.08195
Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulation. (70%)
Wenlong Meng; Fan Zhang; Wendao Yao; Zhenyuan Guo; Yuwei Li; Chengkun Wei; Wenzhi Chen
  Large language models (LLMs) have demonstrated significant utility in a wide
range of applications; however, their deployment is plagued by security
vulnerabilities, notably jailbreak attacks. These attacks manipulate LLMs to
generate harmful or unethical content by crafting adversarial prompts. While
much of the current research on jailbreak attacks has focused on single-turn
interactions, it has largely overlooked the impact of historical dialogues on
model behavior. In this paper, we introduce a novel jailbreak paradigm,
Dialogue Injection Attack (DIA), which leverages the dialogue history to
enhance the success rates of such attacks. DIA operates in a black-box setting,
requiring only access to the chat API or knowledge of the LLM's chat template.
We propose two methods for constructing adversarial historical dialogues: one
adapts gray-box prefilling attacks, and the other exploits deferred responses.
Our experiments show that DIA achieves state-of-the-art attack success rates on
recent LLMs, including Llama-3.1 and GPT-4o. Additionally, we demonstrate that
DIA can bypass 5 different defense mechanisms, highlighting its robustness and
effectiveness.


http://arxiv.org/abs/2503.08801
Enhanced Estimation Techniques for Certified Radii in Randomized Smoothing. (56%)
Zixuan Liang
  This paper presents novel methods for estimating certified radii in
randomized smoothing, a technique crucial for certifying the robustness of
neural networks against adversarial perturbations. Our proposed techniques
significantly improve the accuracy of certified test-set accuracy by providing
tighter bounds on the certified radii. We introduce advanced algorithms for
both discrete and continuous domains, demonstrating their effectiveness on
CIFAR-10 and ImageNet datasets. The new methods show considerable improvements
over existing approaches, particularly in reducing discrepancies in certified
radii estimates. We also explore the impact of various hyperparameters,
including sample size, standard deviation, and temperature, on the performance
of these methods. Our findings highlight the potential for more efficient
certification processes and pave the way for future research on tighter
confidence sequences and improved theoretical frameworks. The study concludes
with a discussion of potential future directions, including enhanced estimation
techniques for discrete domains and further theoretical advancements to bridge
the gap between empirical and theoretical performance in randomized smoothing.


http://arxiv.org/abs/2503.08636
Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning. (33%)
Hubert Baniecki; Przemyslaw Biecek
  A common belief is that intrinsically interpretable deep learning models
ensure a correct, intuitive understanding of their behavior and offer greater
robustness against accidental errors or intentional manipulation. However,
these beliefs have not been comprehensively verified, and growing evidence
casts doubt on them. In this paper, we highlight the risks related to
overreliance and susceptibility to adversarial manipulation of these so-called
"intrinsically (aka inherently) interpretable" models by design. We introduce
two strategies for adversarial analysis with prototype manipulation and
backdoor attacks against prototype-based networks, and discuss how concept
bottleneck models defend against these attacks. Fooling the model's reasoning
by exploiting its use of latent prototypes manifests the inherent
uninterpretability of deep neural networks, leading to a false sense of
security reinforced by a visual confirmation bias. The reported limitations of
prototype-based networks put their trustworthiness and applicability into
question, motivating further work on the robustness and alignment of (deep)
interpretable models.


http://arxiv.org/abs/2503.08976
Not All Edges are Equally Robust: Evaluating the Robustness of Ranking-Based Federated Learning. (22%)
Zirui Gong; Yanjun Zhang; Leo Yu Zhang; Zhaoxi Zhang; Yong Xiang; Shirui Pan
  Federated Ranking Learning (FRL) is a state-of-the-art FL framework that
stands out for its communication efficiency and resilience to poisoning
attacks. It diverges from the traditional FL framework in two ways: 1) it
leverages discrete rankings instead of gradient updates, significantly reducing
communication costs and limiting the potential space for malicious updates, and
2) it uses majority voting on the server side to establish the global ranking,
ensuring that individual updates have minimal influence since each client
contributes only a single vote. These features enhance the system's scalability
and position FRL as a promising paradigm for FL training.
  However, our analysis reveals that FRL is not inherently robust, as certain
edges are particularly vulnerable to poisoning attacks. Through a theoretical
investigation, we prove the existence of these vulnerable edges and establish a
lower bound and an upper bound for identifying them in each layer. Based on
this finding, we introduce a novel local model poisoning attack against FRL,
namely the Vulnerable Edge Manipulation (VEM) attack. The VEM attack focuses on
identifying and perturbing the most vulnerable edges in each layer and
leveraging an optimization-based approach to maximize the attack's impact.
Through extensive experiments on benchmark datasets, we demonstrate that our
attack achieves an overall 53.23% attack impact and is 3.7x more impactful than
existing methods. Our findings highlight significant vulnerabilities in
ranking-based FL systems and underline the urgency for the development of new
robust FL frameworks.


http://arxiv.org/abs/2503.08269
Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks. (8%)
Junying Wang; Hongyuan Zhang; Yuan Yuan
  Recent Customized Portrait Generation (CPG) methods, taking a facial image
and a textual prompt as inputs, have attracted substantial attention. Although
these methods generate high-fidelity portraits, they fail to prevent the
generated portraits from being tracked and misused by malicious face
recognition systems. To address this, this paper proposes a Customized Portrait
Generation framework with facial Adversarial attacks (Adv-CPG). Specifically,
to achieve facial privacy protection, we devise a lightweight local ID
encryptor and an encryption enhancer. They implement progressive double-layer
encryption protection by directly injecting the target identity and adding
additional identity guidance, respectively. Furthermore, to accomplish
fine-grained and personalized portrait generation, we develop a multi-modal
image customizer capable of generating controlled fine-grained facial features.
To the best of our knowledge, Adv-CPG is the first study that introduces facial
adversarial attacks into CPG. Extensive experiments demonstrate the superiority
of Adv-CPG, e.g., the average attack success rate of the proposed Adv-CPG is
28.1% and 2.86% higher compared to the SOTA noise-based attack methods and
unconstrained attack methods, respectively.


http://arxiv.org/abs/2503.08038
Generalized Kullback-Leibler Divergence Loss. (2%)
Jiequan Cui; Beier Zhu; Qingshan Xu; Zhuotao Tian; Xiaojuan Qi; Bei Yu; Hanwang Zhang; Richang Hong
  In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss
and mathematically prove that it is equivalent to the Decoupled
Kullback-Leibler (DKL) Divergence loss that consists of (1) a weighted Mean
Square Error (wMSE) loss and (2) a Cross-Entropy loss incorporating soft
labels. Thanks to the decoupled structure of DKL loss, we have identified two
areas for improvement. Firstly, we address the limitation of KL loss in
scenarios like knowledge distillation by breaking its asymmetric optimization
property along with a smoother weight function. This modification effectively
alleviates convergence challenges in optimization, particularly for classes
with high predicted scores in soft labels. Secondly, we introduce class-wise
global information into KL/DKL to reduce bias arising from individual samples.
With these two enhancements, we derive the Generalized Kullback-Leibler (GKL)
Divergence loss and evaluate its effectiveness by conducting experiments on
CIFAR-10/100, ImageNet, and vision-language datasets, focusing on adversarial
training, and knowledge distillation tasks. Specifically, we achieve new
state-of-the-art adversarial robustness on the public leaderboard --
RobustBench and competitive knowledge distillation performance across
CIFAR/ImageNet models and CLIP models, demonstrating the substantial practical
merits. Our code is available at https://github.com/jiequancui/DKL.


http://arxiv.org/abs/2503.08829
Seal Your Backdoor with Variational Defense. (1%)
Ivan Sabolić; Matej Grcić; Siniša Šegvić
  We propose VIBE, a model-agnostic framework that trains classifiers resilient
to backdoor attacks. The key concept behind our approach is to treat malicious
inputs and corrupted labels from the training dataset as observed random
variables, while the actual clean labels are latent. VIBE then recovers the
corresponding latent clean label posterior through variational inference. The
resulting training procedure follows the expectation-maximization (EM)
algorithm. The E-step infers the clean pseudolabels by solving an
entropy-regularized optimal transport problem, while the M-step updates the
classifier parameters via gradient descent. Being modular, VIBE can seamlessly
integrate with recent advancements in self-supervised representation learning,
which enhance its ability to resist backdoor attacks. We experimentally
validate the method effectiveness against contemporary backdoor attacks on
standard datasets, a large-scale setup with 1$k$ classes, and a dataset
poisoned with multiple attacks. VIBE consistently outperforms previous defenses
across all tested scenarios.


http://arxiv.org/abs/2503.07058
Breaking the Limits of Quantization-Aware Defenses: QADT-R for Robustness Against Patch-Based Adversarial Attacks in QNNs. (99%)
Amira Guesmi; Bassem Ouni; Muhammad Shafique
  Quantized Neural Networks (QNNs) have emerged as a promising solution for
reducing model size and computational costs, making them well-suited for
deployment in edge and resource-constrained environments. While quantization is
known to disrupt gradient propagation and enhance robustness against
pixel-level adversarial attacks, its effectiveness against patch-based
adversarial attacks remains largely unexplored. In this work, we demonstrate
that adversarial patches remain highly transferable across quantized models,
achieving over 70\% attack success rates (ASR) even at extreme bit-width
reductions (e.g., 2-bit). This challenges the common assumption that
quantization inherently mitigates adversarial threats. To address this, we
propose Quantization-Aware Defense Training with Randomization (QADT-R), a
novel defense strategy that integrates Adaptive Quantization-Aware Patch
Generation (A-QAPA), Dynamic Bit-Width Training (DBWT), and
Gradient-Inconsistent Regularization (GIR) to enhance resilience against highly
transferable patch-based attacks. A-QAPA generates adversarial patches within
quantized models, ensuring robustness across different bit-widths. DBWT
introduces bit-width cycling during training to prevent overfitting to a
specific quantization setting, while GIR injects controlled gradient
perturbations to disrupt adversarial optimization. Extensive evaluations on
CIFAR-10 and ImageNet show that QADT-R reduces ASR by up to 25\% compared to
prior defenses such as PBAT and DWQ. Our findings further reveal that
PBAT-trained models, while effective against seen patch configurations, fail to
generalize to unseen patches due to quantization shift. Additionally, our
empirical analysis of gradient alignment, spatial sensitivity, and patch
visibility provides insights into the mechanisms that contribute to the high
transferability of patch-based attacks in QNNs.


http://arxiv.org/abs/2503.06966
MIGA: Mutual Information-Guided Attack on Denoising Models for Semantic Manipulation. (92%)
Guanghao Li; Mingzhi Chen; Hao Yu; Shuting Dong; Wenhao Jiang; Ming Tang; Chun Yuan
  Deep learning-based denoising models have been widely employed in vision
tasks, functioning as filters to eliminate noise while retaining crucial
semantic information. Additionally, they play a vital role in defending against
adversarial perturbations that threaten downstream tasks. However, these models
can be intrinsically susceptible to adversarial attacks due to their dependence
on specific noise assumptions. Existing attacks on denoising models mainly aim
at deteriorating visual clarity while neglecting semantic manipulation,
rendering them either easily detectable or limited in effectiveness. In this
paper, we propose Mutual Information-Guided Attack (MIGA), the first method
designed to directly attack deep denoising models by strategically disrupting
their ability to preserve semantic content via adversarial perturbations. By
minimizing the mutual information between the original and denoised images, a
measure of semantic similarity. MIGA forces the denoiser to produce
perceptually clean yet semantically altered outputs. While these images appear
visually plausible, they encode systematically distorted semantics, revealing a
fundamental vulnerability in denoising models. These distortions persist in
denoised outputs and can be quantitatively assessed through downstream task
performance. We propose new evaluation metrics and systematically assess MIGA
on four denoising models across five datasets, demonstrating its consistent
effectiveness in disrupting semantic fidelity. Our findings suggest that
denoising models are not always robust and can introduce security risks in
real-world applications.


http://arxiv.org/abs/2503.06989
Utilizing Jailbreak Probability to Attack and Safeguard Multimodal LLMs. (87%)
Wenzhuo Xu; Zhipeng Wei; Xiongtao Sun; Deyue Zhang; Dongdong Yang; Quanchen Zou; Xiangzheng Zhang
  Recently, Multimodal Large Language Models (MLLMs) have demonstrated their
superior ability in understanding multimodal contents. However, they remain
vulnerable to jailbreak attacks, which exploit weaknesses in their safety
alignment to generate harmful responses. Previous studies categorize jailbreaks
as successful or failed based on whether responses contain malicious content.
However, given the stochastic nature of MLLM responses, this binary
classification of an input's ability to jailbreak MLLMs is inappropriate.
Derived from this viewpoint, we introduce jailbreak probability to quantify the
jailbreak potential of an input, which represents the likelihood that MLLMs
generated a malicious response when prompted with this input. We approximate
this probability through multiple queries to MLLMs. After modeling the
relationship between input hidden states and their corresponding jailbreak
probability using Jailbreak Probability Prediction Network (JPPN), we use
continuous jailbreak probability for optimization. Specifically, we propose
Jailbreak-Probability-based Attack (JPA) that optimizes adversarial
perturbations on inputs to maximize jailbreak probability. To counteract
attacks, we also propose two defensive methods: Jailbreak-Probability-based
Finetuning (JPF) and Jailbreak-Probability-based Defensive Noise (JPDN), which
minimizes jailbreak probability in the MLLM parameters and input space,
respectively. Extensive experiments show that (1) JPA yields improvements (up
to 28.38\%) under both white and black box settings compared to previous
methods with small perturbation bounds and few iterations. (2) JPF and JPDN
significantly reduce jailbreaks by at most over 60\%. Both of the above results
demonstrate the significance of introducing jailbreak probability to make
nuanced distinctions among input jailbreak abilities.


http://arxiv.org/abs/2503.06950
CtrlRAG: Black-box Adversarial Attacks Based on Masked Language Models in Retrieval-Augmented Language Generation. (83%)
Runqi Sui
  Retrieval-Augmented Generation (RAG) systems enhance Large Language Models
(LLMs) by integrating external knowledge bases. However, this integration
introduces a new security threat: adversaries can exploit the retrieval
mechanism to inject malicious content into the knowledge base, thereby
influencing the generated responses. Based on this attack vector, we propose
CtrlRAG, a novel attack method designed for RAG system in the black-box
setting, which aligns with real-world scenarios. Unlike existing attack
methods, CtrlRAG introduces a perturbation mechanism using Masked Language
Model (MLM) to dynamically optimize malicious content in response to changes in
the retrieved context. Experimental results demonstrate that CtrlRAG
outperforms three baseline methods in both Emotional Manipulation and
Hallucination Amplification objectives. Furthermore, we evaluate three existing
defense mechanisms, revealing their limited effectiveness against CtrlRAG and
underscoring the urgent need for more robust defenses.


http://arxiv.org/abs/2503.07882
ReLATE: Resilient Learner Selection for Multivariate Time-Series Classification Against Adversarial Attacks. (82%)
Cagla Ipek Kocal; Onat Gungor; Aaron Tartz; Tajana Rosing; Baris Aksanli
  Minimizing computational overhead in time-series classification, particularly
in deep learning models, presents a significant challenge. This challenge is
further compounded by adversarial attacks, emphasizing the need for resilient
methods that ensure robust performance and efficient model selection. We
introduce ReLATE, a framework that identifies robust learners based on dataset
similarity, reduces computational overhead, and enhances resilience. ReLATE
maintains multiple deep learning models in well-known adversarial attack
scenarios, capturing model performance. ReLATE identifies the most analogous
dataset to a given target using a similarity metric, then applies the optimal
model from the most similar dataset. ReLATE reduces computational overhead by
an average of 81.2%, enhancing adversarial resilience and streamlining robust
model selection, all without sacrificing performance, within 4.2% of Oracle.


http://arxiv.org/abs/2503.07978
Detecting Backdoor Attacks in Federated Learning via Direction Alignment Inspection. (68%)
Jiahao Xu; Zikai Zhang; Rui Hu
  The distributed nature of training makes Federated Learning (FL) vulnerable
to backdoor attacks, where malicious model updates aim to compromise the global
model's performance on specific tasks. Existing defense methods show limited
efficacy as they overlook the inconsistency between benign and malicious model
updates regarding both general and fine-grained directions. To fill this gap,
we introduce AlignIns, a novel defense method designed to safeguard FL systems
against backdoor attacks. AlignIns looks into the direction of each model
update through a direction alignment inspection process. Specifically, it
examines the alignment of model updates with the overall update direction and
analyzes the distribution of the signs of their significant parameters,
comparing them with the principle sign across all model updates. Model updates
that exhibit an unusual degree of alignment are considered malicious and thus
be filtered out. We provide the theoretical analysis of the robustness of
AlignIns and its propagation error in FL. Our empirical results on both
independent and identically distributed (IID) and non-IID datasets demonstrate
that AlignIns achieves higher robustness compared to the state-of-the-art
defense methods. The code is available at
https://github.com/JiiahaoXU/AlignIns.


http://arxiv.org/abs/2503.07568
Runtime Detection of Adversarial Attacks in AI Accelerators Using Performance Counters. (54%)
Habibur Rahaman; Atri Chatterjee; Swarup Bhunia
  Rapid adoption of AI technologies raises several major security concerns,
including the risks of adversarial perturbations, which threaten the
confidentiality and integrity of AI applications. Protecting AI hardware from
misuse and diverse security threats is a challenging task. To address this
challenge, we propose SAMURAI, a novel framework for safeguarding against
malicious usage of AI hardware and its resilience to attacks. SAMURAI
introduces an AI Performance Counter (APC) for tracking dynamic behavior of an
AI model coupled with an on-chip Machine Learning (ML) analysis engine, known
as TANTO (Trained Anomaly Inspection Through Trace Observation). APC records
the runtime profile of the low-level hardware events of different AI
operations. Subsequently, the summary information recorded by the APC is
processed by TANTO to efficiently identify potential security breaches and
ensure secure, responsible use of AI. SAMURAI enables real-time detection of
security threats and misuse without relying on traditional software-based
solutions that require model integration. Experimental results demonstrate that
SAMURAI achieves up to 97% accuracy in detecting adversarial attacks with
moderate overhead on various AI models, significantly outperforming
conventional software-based approaches. It enhances security and regulatory
compliance, providing a comprehensive solution for safeguarding AI against
emergent threats.


http://arxiv.org/abs/2503.07818
Strengthening the Internal Adversarial Robustness in Lifted Neural Networks. (26%)
Christopher Zach
  Lifted neural networks (i.e. neural architectures explicitly optimizing over
respective network potentials to determine the neural activities) can be
combined with a type of adversarial training to gain robustness for internal as
well as input layers, in addition to improved generalization performance. In
this work we first investigate how adversarial robustness in this framework can
be further strengthened by solely modifying the training loss. In a second step
we fix some remaining limitations and arrive at a novel training loss for
lifted neural networks, that combines targeted and untargeted adversarial
perturbations.


http://arxiv.org/abs/2503.07697
PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models. (13%)
Michael-Andrei Panaitescu-Liess; Pankayaraj Pathmanathan; Yigitcan Kaya; Zora Che; Bang An; Sicheng Zhu; Aakriti Agrawal; Furong Huang
  As the capabilities of large language models (LLMs) continue to expand, their
usage has become increasingly prevalent. However, as reflected in numerous
ongoing lawsuits regarding LLM-generated content, addressing copyright
infringement remains a significant challenge. In this paper, we introduce
PoisonedParrot: the first stealthy data poisoning attack that induces an LLM to
generate copyrighted content even when the model has not been directly trained
on the specific copyrighted material. PoisonedParrot integrates small fragments
of copyrighted text into the poison samples using an off-the-shelf LLM. Despite
its simplicity, evaluated in a wide range of experiments, PoisonedParrot is
surprisingly effective at priming the model to generate copyrighted content
with no discernible side effects. Moreover, we discover that existing defenses
are largely ineffective against our attack. Finally, we make the first attempt
at mitigating copyright-infringement poisoning attacks by proposing a defense:
ParrotTrap. We encourage the community to explore this emerging threat model
further.


http://arxiv.org/abs/2503.08731
FairDeFace: Evaluating the Fairness and Adversarial Robustness of Face Obfuscation Methods. (2%)
Seyyed Mohammad Sadegh Moosavi Khorzooghi; Poojitha Thota; Mohit Singhal; Abolfazl Asudeh; Gautam Das; Shirin Nilizadeh
  The lack of a common platform and benchmark datasets for evaluating face
obfuscation methods has been a challenge, with every method being tested using
arbitrary experiments, datasets, and metrics. While prior work has demonstrated
that face recognition systems exhibit bias against some demographic groups,
there exists a substantial gap in our understanding regarding the fairness of
face obfuscation methods. Providing fair face obfuscation methods can ensure
equitable protection across diverse demographic groups, especially since they
can be used to preserve the privacy of vulnerable populations. To address these
gaps, this paper introduces a comprehensive framework, named FairDeFace,
designed to assess the adversarial robustness and fairness of face obfuscation
methods. The framework introduces a set of modules encompassing data
benchmarks, face detection and recognition algorithms, adversarial models,
utility detection models, and fairness metrics. FairDeFace serves as a
versatile platform where any face obfuscation method can be integrated,
allowing for rigorous testing and comparison with other state-of-the-art
methods. In its current implementation, FairDeFace incorporates 6 attacks, and
several privacy, utility and fairness metrics. Using FairDeFace, and by
conducting more than 500 experiments, we evaluated and compared the adversarial
robustness of seven face obfuscation methods. This extensive analysis led to
many interesting findings both in terms of the degree of robustness of existing
methods and their biases against some gender or racial groups. FairDeFace also
uses visualization of focused areas for both obfuscation and verification
attacks to show not only which areas are mostly changed in the obfuscation
process for some demographics, but also why they failed through focus area
comparison of obfuscation and verification.


http://arxiv.org/abs/2503.07464
Learning to Localize Leakage of Cryptographic Sensitive Variables. (2%)
Jimmy Gammell; Anand Raghunathan; Abolfazl Hashemi; Kaushik Roy
  While cryptographic algorithms such as the ubiquitous Advanced Encryption
Standard (AES) are secure, *physical implementations* of these algorithms in
hardware inevitably 'leak' sensitive data such as cryptographic keys. A
particularly insidious form of leakage arises from the fact that hardware
consumes power and emits radiation in a manner that is statistically associated
with the data it processes and the instructions it executes. Supervised deep
learning has emerged as a state-of-the-art tool for carrying out *side-channel
attacks*, which exploit this leakage by learning to map power/radiation
measurements throughout encryption to the sensitive data operated on during
that encryption. In this work we develop a principled deep learning framework
for determining the relative leakage due to measurements recorded at different
points in time, in order to inform *defense* against such attacks. This
information is invaluable to cryptographic hardware designers for understanding
*why* their hardware leaks and how they can mitigate it (e.g. by indicating the
particular sections of code or electronic components which are responsible).
Our framework is based on an adversarial game between a family of classifiers
trained to estimate the conditional distributions of sensitive data given
subsets of measurements, and a budget-constrained noise distribution which
probabilistically erases individual measurements to maximize the loss of these
classifiers. We demonstrate our method's efficacy and ability to overcome
limitations of prior work through extensive experimental comparison with 8
baseline methods using 3 evaluation metrics and 6 publicly-available power/EM
trace datasets from AES, ECC and RSA implementations. We provide an open-source
PyTorch implementation of these experiments.


http://arxiv.org/abs/2503.06559
MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillation. (80%)
Yuzheng Wang; Zhaoyu Chen; Dingkang Yang; Yuanhang Wang; Lizhe Qi
  Adversarial Robustness Distillation (ARD) is a promising task to boost the
robustness of small-capacity models with the guidance of the pre-trained robust
teacher. The ARD can be summarized as a min-max optimization process, i.e.,
synthesizing adversarial examples (inner) & training the student (outer).
Although competitive robustness performance, existing ARD methods still have
issues. In the inner process, the synthetic training examples are far from the
teacher's decision boundary leading to important robust information missing. In
the outer process, the student model is decoupled from learning natural and
robust scenarios, leading to the robustness saturation, i.e., student
performance is highly susceptible to customized teacher selection. To tackle
these issues, this paper proposes a general Min-Max optimization Adversarial
Robustness Distillation (MMARD) method. For the inner process, we introduce the
teacher's robust predictions, which drive the training examples closer to the
teacher's decision boundary to explore more robust knowledge. For the outer
process, we propose a structured information modeling method based on
triangular relationships to measure the mutual information of the model in
natural and robust scenarios and enhance the model's ability to understand
multi-scenario mapping relationships. Experiments show our MMARD achieves
state-of-the-art performance on multiple benchmarks. Besides, MMARD is
plug-and-play and convenient to combine with existing methods.


http://arxiv.org/abs/2503.06461
Long-tailed Adversarial Training with Self-Distillation. (78%)
Seungju Cho; Hongsin Lee; Changick Kim
  Adversarial training significantly enhances adversarial robustness, yet
superior performance is predominantly achieved on balanced datasets.
  Addressing adversarial robustness in the context of unbalanced or long-tailed
distributions is considerably more challenging, mainly due to the scarcity of
tail data instances.
  Previous research on adversarial robustness within long-tailed distributions
has primarily focused on combining traditional long-tailed natural training
with existing adversarial robustness methods.
  In this study, we provide an in-depth analysis for the challenge that
adversarial training struggles to achieve high performance on tail classes in
long-tailed distributions.
  Furthermore, we propose a simple yet effective solution to advance
adversarial robustness on long-tailed distributions through a novel
self-distillation technique.
  Specifically, this approach leverages a balanced self-teacher model, which is
trained using a balanced dataset sampled from the original long-tailed dataset.
Our extensive experiments demonstrate state-of-the-art performance in both
clean and robust accuracy for long-tailed adversarial robustness, with
significant improvements in tail class performance on various datasets. We
improve the accuracy against PGD attacks for tail classes by 20.3, 7.1, and 3.8
percentage points on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively,
while achieving the highest robust accuracy.


http://arxiv.org/abs/2503.08704
Life-Cycle Routing Vulnerabilities of LLM Router. (68%)
Qiqi Lin; Xiaoyang Ji; Shengfang Zhai; Qingni Shen; Zhi Zhang; Yuejian Fang; Yansong Gao
  Large language models (LLMs) have achieved remarkable success in natural
language processing, yet their performance and computational costs vary
significantly. LLM routers play a crucial role in dynamically balancing these
trade-offs. While previous studies have primarily focused on routing
efficiency, security vulnerabilities throughout the entire LLM router life
cycle, from training to inference, remain largely unexplored. In this paper, we
present a comprehensive investigation into the life-cycle routing
vulnerabilities of LLM routers. We evaluate both white-box and black-box
adversarial robustness, as well as backdoor robustness, across several
representative routing models under extensive experimental settings. Our
experiments uncover several key findings: 1) Mainstream DNN-based routers tend
to exhibit the weakest adversarial and backdoor robustness, largely due to
their strong feature extraction capabilities that amplify vulnerabilities
during both training and inference; 2) Training-free routers demonstrate the
strongest robustness across different attack types, benefiting from the absence
of learnable parameters that can be manipulated. These findings highlight
critical security risks spanning the entire life cycle of LLM routers and
provide insights for developing more robust models.


http://arxiv.org/abs/2503.06453
NaviDet: Efficient Input-level Backdoor Detection on Text-to-Image Synthesis via Neuron Activation Variation. (56%)
Shengfang Zhai; Jiajun Li; Yue Liu; Huanran Chen; Zhihua Tian; Wenjie Qu; Qingni Shen; Ruoxi Jia; Yinpeng Dong; Jiaheng Zhang
  In recent years, text-to-image (T2I) diffusion models have garnered
significant attention for their ability to generate high-quality images
reflecting text prompts. However, their growing popularity has also led to the
emergence of backdoor threats, posing substantial risks. Currently, effective
defense strategies against such threats are lacking due to the diversity of
backdoor targets in T2I synthesis. In this paper, we propose NaviDet, the first
general input-level backdoor detection framework for identifying backdoor
inputs across various backdoor targets. Our approach is based on the new
observation that trigger tokens tend to induce significant neuron activation
variation in the early stage of the diffusion generation process, a phenomenon
we term Early-step Activation Variation. Leveraging this insight, NaviDet
detects malicious samples by analyzing neuron activation variations caused by
input tokens. Through extensive experiments, we demonstrate the effectiveness
and efficiency of our method against various T2I backdoor attacks, surpassing
existing baselines with significantly lower computational overhead.
Furthermore, we rigorously demonstrate that our method remains effective
against potential adaptive attacks.


http://arxiv.org/abs/2503.06519
Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluation. (15%)
Wenhui Zhang; Huiyu Xu; Zhibo Wang; Zeqing He; Ziqi Zhu; Kui Ren
  Small language models (SLMs) have emerged as promising alternatives to large
language models (LLMs) due to their low computational demands, enhanced privacy
guarantees and comparable performance in specific domains through light-weight
fine-tuning. Deploying SLMs on edge devices, such as smartphones and smart
vehicles, has become a growing trend. However, the security implications of
SLMs have received less attention than LLMs, particularly regarding jailbreak
attacks, which is recognized as one of the top threats of LLMs by the OWASP. In
this paper, we conduct the first large-scale empirical study of SLMs'
vulnerabilities to jailbreak attacks. Through systematically evaluation on 63
SLMs from 15 mainstream SLM families against 8 state-of-the-art jailbreak
methods, we demonstrate that 47.6% of evaluated SLMs show high susceptibility
to jailbreak attacks (ASR > 40%) and 38.1% of them can not even resist direct
harmful query (ASR > 50%). We further analyze the reasons behind the
vulnerabilities and identify four key factors: model size, model architecture,
training datasets and training techniques. Moreover, we assess the
effectiveness of three prompt-level defense methods and find that none of them
achieve perfect performance, with detection accuracy varying across different
SLMs and attack methods. Notably, we point out that the inherent security
awareness play a critical role in SLM security, and models with strong security
awareness could timely terminate unsafe response with little reminder. Building
upon the findings, we highlight the urgent need for security-by-design
approaches in SLM development and provide valuable insights for building more
trustworthy SLM ecosystem.


http://arxiv.org/abs/2503.08708
TH-Bench: Evaluating Evading Attacks via Humanizing AI Text on Machine-Generated Text Detectors. (10%)
Jingyi Zheng; Junfeng Wang; Zhen Sun; Wenhan Dong; Yule Liu; Xinlei He
  As Large Language Models (LLMs) advance, Machine-Generated Texts (MGTs) have
become increasingly fluent, high-quality, and informative. Existing wide-range
MGT detectors are designed to identify MGTs to prevent the spread of plagiarism
and misinformation. However, adversaries attempt to humanize MGTs to evade
detection (named evading attacks), which requires only minor modifications to
bypass MGT detectors. Unfortunately, existing attacks generally lack a unified
and comprehensive evaluation framework, as they are assessed using different
experimental settings, model architectures, and datasets. To fill this gap, we
introduce the Text-Humanization Benchmark (TH-Bench), the first comprehensive
benchmark to evaluate evading attacks against MGT detectors. TH-Bench evaluates
attacks across three key dimensions: evading effectiveness, text quality, and
computational overhead. Our extensive experiments evaluate 6 state-of-the-art
attacks against 13 MGT detectors across 6 datasets, spanning 19 domains and
generated by 11 widely used LLMs. Our findings reveal that no single evading
attack excels across all three dimensions. Through in-depth analysis, we
highlight the strengths and limitations of different attacks. More importantly,
we identify a trade-off among three dimensions and propose two optimization
insights. Through preliminary experiments, we validate their correctness and
effectiveness, offering potential directions for future research.


http://arxiv.org/abs/2503.06529
AnywhereDoor: Multi-Target Backdoor Attacks on Object Detection. (10%)
Jialin Lu; Junjie Shan; Ziqi Zhao; Ka-Ho Chow
  As object detection becomes integral to many safety-critical applications,
understanding its vulnerabilities is essential. Backdoor attacks, in
particular, pose a serious threat by implanting hidden triggers in victim
models, which adversaries can later exploit to induce malicious behaviors
during inference. However, current understanding is limited to single-target
attacks, where adversaries must define a fixed malicious behavior (target)
before training, making inference-time adaptability impossible. Given the large
output space of object detection (including object existence prediction,
bounding box estimation, and classification), the feasibility of flexible,
inference-time model control remains unexplored. This paper introduces
AnywhereDoor, a multi-target backdoor attack for object detection. Once
implanted, AnywhereDoor allows adversaries to make objects disappear, fabricate
new ones, or mislabel them, either across all object classes or specific ones,
offering an unprecedented degree of control. This flexibility is enabled by
three key innovations: (i) objective disentanglement to scale the number of
supported targets; (ii) trigger mosaicking to ensure robustness even against
region-based detectors; and (iii) strategic batching to address object-level
data imbalances that hinder manipulation. Extensive experiments demonstrate
that AnywhereDoor grants attackers a high degree of control, improving attack
success rates by 26% compared to adaptations of existing methods for such
flexible control.


http://arxiv.org/abs/2503.06140
Boosting the Local Invariance for Better Adversarial Transferability. (99%)
Bohan Liu; Xiaosen Wang
  Transfer-based attacks pose a significant threat to real-world applications
by directly targeting victim models with adversarial examples generated on
surrogate models. While numerous approaches have been proposed to enhance
adversarial transferability, existing works often overlook the intrinsic
relationship between adversarial perturbations and input images. In this work,
we find that adversarial perturbation often exhibits poor translation
invariance for a given clean image and model, which is attributed to local
invariance. Through empirical analysis, we demonstrate that there is a positive
correlation between the local invariance of adversarial perturbations w.r.t.
the input image and their transferability across different models. Based on
this finding, we propose a general adversarial transferability boosting
technique called Local Invariance Boosting approach (LI-Boost). Extensive
experiments on the standard ImageNet dataset demonstrate that LI-Boost could
significantly boost various types of transfer-based attacks (e.g.,
gradient-based, input transformation-based, model-related, advanced objective
function, ensemble, etc.) on CNNs, ViTs, and defense mechanisms. Our approach
presents a promising direction for future research in improving adversarial
transferability across different models.


http://arxiv.org/abs/2503.06269
Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models. (98%)
Thomas Winninger; Boussad Addad; Katarzyna Kapusta
  Traditional white-box methods for creating adversarial perturbations against
LLMs typically rely only on gradient computation from the targeted model,
ignoring the internal mechanisms responsible for attack success or failure.
Conversely, interpretability studies that analyze these internal mechanisms
lack practical applications beyond runtime interventions. We bridge this gap by
introducing a novel white-box approach that leverages mechanistic
interpretability techniques to craft practical adversarial inputs.
Specifically, we first identify acceptance subspaces - sets of feature vectors
that do not trigger the model's refusal mechanisms - then use gradient-based
optimization to reroute embeddings from refusal subspaces to acceptance
subspaces, effectively achieving jailbreaks. This targeted approach
significantly reduces computation cost, achieving attack success rates of
80-95\% on state-of-the-art models including Gemma2, Llama3.2, and Qwen2.5
within minutes or even seconds, compared to existing techniques that often fail
or require hours of computation. We believe this approach opens a new direction
for both attack research and defense development. Furthermore, it showcases a
practical application of mechanistic interpretability where other methods are
less efficient, which highlights its utility. The code and generated datasets
are available at https://github.com/Sckathach/subspace-rerouting.


http://arxiv.org/abs/2503.06188
Attackers Can Do Better: Over- and Understated Factors of Model Stealing Attacks. (80%)
Daryna Oliynyk; Rudolf Mayer; Andreas Rauber
  Machine learning models were shown to be vulnerable to model stealing
attacks, which lead to intellectual property infringement. Among other methods,
substitute model training is an all-encompassing attack applicable to any
machine learning model whose behaviour can be approximated from input-output
queries. Whereas prior works mainly focused on improving the performance of
substitute models by, e.g. developing a new substitute training method, there
have been only limited ablation studies on the impact the attacker's strength
has on the substitute model's performance. As a result, different authors came
to diverse, sometimes contradicting, conclusions. In this work, we exhaustively
examine the ambivalent influence of different factors resulting from varying
the attacker's capabilities and knowledge on a substitute training attack. Our
findings suggest that some of the factors that have been considered important
in the past are, in fact, not that influential; instead, we discover new
correlations between attack conditions and success rate. In particular, we
demonstrate that better-performing target models enable higher-fidelity attacks
and explain the intuition behind this phenomenon. Further, we propose to shift
the focus from the complexity of target models toward the complexity of their
learning tasks. Therefore, for the substitute model, rather than aiming for a
higher architecture complexity, we suggest focusing on getting data of higher
complexity and an appropriate architecture. Finally, we demonstrate that even
in the most limited data-free scenario, there is no need to overcompensate weak
knowledge with millions of queries. Our results often exceed or match the
performance of previous attacks that assume a stronger attacker, suggesting
that these stronger attacks are likely endangering a model owner's intellectual
property to a significantly higher degree than shown until now.


http://arxiv.org/abs/2503.10661
CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models. (73%)
Xiangyu Yin; Jiaxu Liu; Zhen Chen; Jinwei Hu; Yi Dong; Xiaowei Huang; Wenjie Ruan
  Recent advances in large vision-language models (VLMs) have demonstrated
remarkable success across a wide range of visual understanding tasks. However,
the robustness of these models against jailbreak attacks remains an open
challenge. In this work, we propose a universal certified defence framework to
safeguard VLMs rigorously against potential visual jailbreak attacks. First, we
proposed a novel distance metric to quantify semantic discrepancies between
malicious and intended responses, capturing subtle differences often overlooked
by conventional cosine similarity-based measures. Then, we devise a regressed
certification approach that employs randomized smoothing to provide formal
robustness guarantees against both adversarial and structural perturbations,
even under black-box settings. Complementing this, our feature-space defence
introduces noise distributions (e.g., Gaussian, Laplacian) into the latent
embeddings to safeguard against both pixel-level and structure-level
perturbations. Our results highlight the potential of a formally grounded,
integrated strategy toward building more resilient and trustworthy VLMs.


http://arxiv.org/abs/2503.06223
Reinforced Diffuser for Red Teaming Large Vision-Language Models. (68%)
Ruofan Wang; Xiang Zheng; Xiaosen Wang; Cong Wang; Xingjun Ma
  The rapid advancement of large Vision-Language Models (VLMs) has raised
significant safety concerns, particularly regarding their vulnerability to
jailbreak attacks. While existing research primarily focuses on VLMs'
susceptibility to harmful instructions, this work identifies a critical yet
overlooked vulnerability: current alignment mechanisms often fail to address
the risks posed by toxic text continuation tasks. To investigate this issue, we
propose a novel Red Team Diffuser (RTD) framework, which leverages
reinforcement learning to generate red team images that effectively induce
highly toxic continuations from target black-box VLMs. The RTD pipeline begins
with a greedy search for high-quality image prompts that maximize the toxicity
of VLM-generated sentence continuations, guided by a Large Language Model
(LLM). These prompts are then used as input for the reinforcement fine-tuning
of a diffusion model, which employs toxicity and alignment rewards to further
amplify harmful outputs. Experimental results demonstrate the effectiveness of
RTD, increasing the toxicity rate of LLaVA outputs by 10.69% on the original
attack set and 8.91% on a hold-out set. Moreover, RTD exhibits strong
cross-model transferability, raising the toxicity rate by 5.1% on Gemini and
26.83% on LLaMA. These findings reveal significant deficiencies in existing
alignment strategies, particularly their inability to prevent harmful
continuations. Our work underscores the urgent need for more robust and
adaptive alignment mechanisms to ensure the safe deployment of VLMs in
real-world applications.


http://arxiv.org/abs/2503.06361
Adversarial Robustness of Discriminative Self-Supervised Learning in Vision. (16%)
Ömer Veysel Çağatan; Ömer Faruk Tal; M. Emre Gürsoy
  Self-supervised learning (SSL) has advanced significantly in visual
representation learning, yet comprehensive evaluations of its adversarial
robustness remain limited. In this study, we evaluate the adversarial
robustness of seven discriminative self-supervised models and one supervised
model across diverse tasks, including ImageNet classification, transfer
learning, segmentation, and detection. Our findings suggest that discriminative
SSL models generally exhibit better robustness to adversarial attacks compared
to their supervised counterpart on ImageNet, with this advantage extending to
transfer learning when using linear evaluation. However, when fine-tuning is
applied, the robustness gap between SSL and supervised models narrows
considerably. Similarly, this robustness advantage diminishes in segmentation
and detection tasks. We also investigate how various factors might influence
adversarial robustness, including architectural choices, training duration,
data augmentations, and batch sizes. Our analysis contributes to the ongoing
exploration of adversarial robustness in visual self-supervised representation
systems.


http://arxiv.org/abs/2503.06254
Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation. (13%)
Yinuo Liu; Zenghui Yuan; Guiyao Tie; Jiawen Shi; Lichao Sun; Neil Zhenqiang Gong
  Multimodal retrieval-augmented generation (RAG) enhances the visual reasoning
capability of vision-language models (VLMs) by dynamically accessing
information from external knowledge bases. In this work, we introduce
\textit{Poisoned-MRAG}, the first knowledge poisoning attack on multimodal RAG
systems. Poisoned-MRAG injects a few carefully crafted image-text pairs into
the multimodal knowledge database, manipulating VLMs to generate the
attacker-desired response to a target query. Specifically, we formalize the
attack as an optimization problem and propose two cross-modal attack
strategies, dirty-label and clean-label, tailored to the attacker's knowledge
and goals. Our extensive experiments across multiple knowledge databases and
VLMs show that Poisoned-MRAG outperforms existing methods, achieving up to 98\%
attack success rate with just five malicious image-text pairs injected into the
InfoSeek database (481,782 pairs). Additionally, We evaluate 4 different
defense strategies, including paraphrasing, duplicate removal, structure-driven
mitigation, and purification, demonstrating their limited effectiveness and
trade-offs against Poisoned-MRAG. Our results highlight the effectiveness and
scalability of Poisoned-MRAG, underscoring its potential as a significant
threat to multimodal RAG systems.


http://arxiv.org/abs/2503.06253
MAD-MAX: Modular And Diverse Malicious Attack MiXtures for Automated LLM Red Teaming. (10%)
Stefan Schoepf; Muhammad Zaid Hameed; Ambrish Rawat; Kieran Fraser; Giulio Zizzo; Giandomenico Cornacchia; Mark Purcell
  With LLM usage rapidly increasing, their vulnerability to jailbreaks that
create harmful outputs are a major security risk. As new jailbreaking
strategies emerge and models are changed by fine-tuning, continuous testing for
security vulnerabilities is necessary. Existing Red Teaming methods fall short
in cost efficiency, attack success rate, attack diversity, or extensibility as
new attack types emerge. We address these challenges with Modular And Diverse
Malicious Attack MiXtures (MAD-MAX) for Automated LLM Red Teaming. MAD-MAX uses
automatic assignment of attack strategies into relevant attack clusters,
chooses the most relevant clusters for a malicious goal, and then combines
strategies from the selected clusters to achieve diverse novel attacks with
high attack success rates. MAD-MAX further merges promising attacks together at
each iteration of Red Teaming to boost performance and introduces a similarity
filter to prune out similar attacks for increased cost efficiency. The MAD-MAX
approach is designed to be easily extensible with newly discovered attack
strategies and outperforms the prominent Red Teaming method Tree of Attacks
with Pruning (TAP) significantly in terms of Attack Success Rate (ASR) and
queries needed to achieve jailbreaks. MAD-MAX jailbreaks 97% of malicious goals
in our benchmarks on GPT-4o and Gemini-Pro compared to TAP with 66%. MAD-MAX
does so with only 10.9 average queries to the target LLM compared to TAP with
23.3.
  WARNING: This paper contains contents which are offensive in nature.


http://arxiv.org/abs/2503.06276
Exploring Adversarial Transferability between Kolmogorov-arnold Networks. (5%)
Songping Wang; Xinquan Yue; Yueming Lyu; Caifeng Shan
  Kolmogorov-Arnold Networks (KANs) have emerged as a transformative model
paradigm, significantly impacting various fields. However, their adversarial
robustness remains less underexplored, especially across different KAN
architectures. To explore this critical safety issue, we conduct an analysis
and find that due to overfitting to the specific basis functions of KANs, they
possess poor adversarial transferability among different KANs. To tackle this
challenge, we propose AdvKAN, the first transfer attack method for KANs. AdvKAN
integrates two key components: 1) a Breakthrough-Defense Surrogate Model
(BDSM), which employs a breakthrough-defense training strategy to mitigate
overfitting to the specific structures of KANs. 2) a Global-Local Interaction
(GLI) technique, which promotes sufficient interaction between adversarial
gradients of hierarchical levels, further smoothing out loss surfaces of KANs.
Both of them work together to enhance the strength of transfer attack among
different KANs. Extensive experimental results on various KANs and datasets
demonstrate the effectiveness of AdvKAN, which possesses notably superior
attack capabilities and deeply reveals the vulnerabilities of KANs. Code will
be released upon acceptance.


http://arxiv.org/abs/2503.07661
Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracy. (2%)
Wei Junhao; Yu Zhe; Sakuma Jun
  Model merging is a technique that combines multiple finetuned models into a
single model without additional training, allowing a free-rider to cheaply
inherit specialized capabilities. This study investigates methodologies to
suppress unwanted model merging by free-riders. Existing methods such as model
watermarking or fingerprinting can only detect merging in hindsight. In
contrast, we propose a first proactive defense against model merging.
Specifically, our defense method modifies the model parameters so that the
model is disrupted if the model is merged with any other model, while its
functionality is kept unchanged if not merged with others. Our approach
consists of two modules, rearranging MLP parameters and scaling attention
heads, which push the model out of the shared basin in parameter space, causing
the merging performance with other models to degrade significantly. We conduct
extensive experiments on image classification, image generation, and text
classification to demonstrate that our defense severely disrupts merging while
retaining the functionality of the post-protect model. Moreover, we analyze
potential adaptive attacks and further propose a dropout-based pruning to
improve our proposal's robustness.


http://arxiv.org/abs/2503.06340
Backdoor Attacks on Discrete Graph Diffusion Models. (1%)
Jiawen Wang; Samin Karim; Yuan Hong; Binghui Wang
  Diffusion models are powerful generative models in continuous data domains
such as image and video data. Discrete graph diffusion models (DGDMs) have
recently extended them for graph generation, which are crucial in fields like
molecule and protein modeling, and obtained the SOTA performance. However, it
is risky to deploy DGDMs for safety-critical applications (e.g., drug
discovery) without understanding their security vulnerabilities. In this work,
we perform the first study on graph diffusion models against backdoor attacks,
a severe attack that manipulates both the training and inference/generation
phases in graph diffusion models. We first define the threat model, under which
we design the attack such that the backdoored graph diffusion model can
generate 1) high-quality graphs without backdoor activation, 2) effective,
stealthy, and persistent backdoored graphs with backdoor activation, and 3)
graphs that are permutation invariant and exchangeable--two core properties in
graph generative models. 1) and 2) are validated via empirical evaluations
without and with backdoor defenses, while 3) is validated via theoretical
results.


http://arxiv.org/abs/2503.05303
Robust Intrusion Detection System with Explainable Artificial Intelligence. (83%)
Betül Güvenç Paltun; Ramin Fuladi; Rim El Malki
  Machine learning (ML) models serve as powerful tools for threat detection and
mitigation; however, they also introduce potential new risks. Adversarial input
can exploit these models through standard interfaces, thus creating new attack
pathways that threaten critical network operations. As ML advancements
progress, adversarial strategies become more advanced, and conventional
defenses such as adversarial training are costly in computational terms and
often fail to provide real-time detection. These methods typically require a
balance between robustness and model performance, which presents challenges for
applications that demand instant response. To further investigate this
vulnerability, we suggest a novel strategy for detecting and mitigating
adversarial attacks using eXplainable Artificial Intelligence (XAI). This
approach is evaluated in real time within intrusion detection systems (IDS),
leading to the development of a zero-touch mitigation strategy. Additionally,
we explore various scenarios in the Radio Resource Control (RRC) layer within
the Open Radio Access Network (O-RAN) framework, emphasizing the critical need
for enhanced mitigation techniques to strengthen IDS defenses against advanced
threats and implement a zero-touch mitigation solution. Extensive testing
across different scenarios in the RRC layer of the O-RAN infrastructure
validates the ability of the framework to detect and counteract integrated
RRC-layer attacks when paired with adversarial strategies, emphasizing the
essential need for robust defensive mechanisms to strengthen IDS against
complex threats.


http://arxiv.org/abs/2503.05445
Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attacks. (22%)
Meiyu Lin; Haichuan Zhang; Jiale Lao; Renyuan Li; Yuanchun Zhou; Carl Yang; Yang Cao; Mingjie Tang
  Large language models (LLMs) have shown state-of-the-art results in
translating natural language questions into SQL queries (Text-to-SQL), a
long-standing challenge within the database community. However, security
concerns remain largely unexplored, particularly the threat of backdoor
attacks, which can introduce malicious behaviors into models through
fine-tuning with poisoned datasets. In this work, we systematically investigate
the vulnerabilities of LLM-based Text-to-SQL models and present ToxicSQL, a
novel backdoor attack framework. Our approach leverages stealthy {semantic and
character-level triggers} to make backdoors difficult to detect and remove,
ensuring that malicious behaviors remain covert while maintaining high model
accuracy on benign inputs. Furthermore, we propose leveraging SQL injection
payloads as backdoor targets, enabling the generation of malicious yet
executable SQL queries, which pose severe security and privacy risks in
language model-based SQL development. We demonstrate that injecting only 0.44%
of poisoned data can result in an attack success rate of 79.41%, posing a
significant risk to database security. Additionally, we propose detection and
mitigation strategies to enhance model reliability. Our findings highlight the
urgent need for security-aware Text-to-SQL development, emphasizing the
importance of robust defenses against backdoor threats.


http://arxiv.org/abs/2503.05477
Enhancing Network Security: A Hybrid Approach for Detection and Mitigation of Distributed Denial-of-Service Attacks Using Machine Learning. (1%)
Nizo Jaman Shohan; Gazi Tanbhir; Faria Elahi; Ahsan Ullah; Md. Nazmus Sakib
  The distributed denial-of-service (DDoS) attack stands out as a highly
formidable cyber threat, representing an advanced form of the denial-of-service
(DoS) attack. A DDoS attack involves multiple computers working together to
overwhelm a system, making it unavailable. On the other hand, a DoS attack is a
one-on-one attempt to make a system or website inaccessible. Thus, it is
crucial to construct an effective model for identifying various DDoS incidents.
Although extensive research has focused on binary detection models for DDoS
identification, they face challenges to adapt evolving threats, necessitating
frequent updates. Whereas multiclass detection models offer a comprehensive
defense against diverse DDoS attacks, ensuring adaptability in the
ever-changing cyber threat landscape. In this paper, we propose a Hybrid Model
to strengthen network security by combining the featureextraction abilities of
1D Convolutional Neural Networks (CNNs) with the classification skills of
Random Forest (RF) and Multi-layer Perceptron (MLP) classifiers. Using the
CIC-DDoS2019 dataset, we perform multiclass classification of various DDoS
attacks and conduct a comparative analysis of evaluation metrics for RF, MLP,
and our proposed Hybrid Model. After analyzing the results, we draw meaningful
conclusions and confirm the superiority of our Hybrid Model by performing
thorough cross-validation. Additionally, we integrate our machine learning
model with Snort, which provides a robust and adaptive solution for detecting
and mitigating various DDoS attacks.


http://arxiv.org/abs/2503.05916
SAS: Segment Anything Small for Ultrasound -- A Non-Generative Data Augmentation Technique for Robust Deep Learning in Ultrasound Imaging. (1%)
Danielle L. Ferreira; Ahana Gangopadhyay; Hsi-Ming Chang; Ravi Soni; Gopal Avinash
  Accurate segmentation of anatomical structures in ultrasound (US) images,
particularly small ones, is challenging due to noise and variability in imaging
conditions (e.g., probe position, patient anatomy, tissue characteristics and
pathology). To address this, we introduce Segment Anything Small (SAS), a
simple yet effective scale- and texture-aware data augmentation technique
designed to enhance the performance of deep learning models for segmenting
small anatomical structures in ultrasound images. SAS employs a dual
transformation strategy: (1) simulating diverse organ scales by resizing and
embedding organ thumbnails into a black background, and (2) injecting noise
into regions of interest to simulate varying tissue textures. These
transformations generate realistic and diverse training data without
introducing hallucinations or artifacts, improving the model's robustness to
noise and variability. We fine-tuned a promptable foundation model on a
controlled organ-specific medical imaging dataset and evaluated its performance
on one internal and five external datasets. Experimental results demonstrate
significant improvements in segmentation performance, with Dice score gains of
up to 0.35 and an average improvement of 0.16 [95% CI 0.132,0.188].
Additionally, our iterative point prompts provide precise control and adaptive
refinement, achieving performance comparable to bounding box prompts with just
two points. SAS enhances model robustness and generalizability across diverse
anatomical structures and imaging conditions, particularly for small
structures, without compromising the accuracy of larger ones. By offering a
computationally efficient solution that eliminates the need for extensive human
labeling efforts, SAS emerges as a powerful tool for advancing medical image
analysis, particularly in resource-constrained settings.


http://arxiv.org/abs/2503.04385
Scale-Invariant Adversarial Attack against Arbitrary-scale Super-resolution. (98%)
Yihao Huang; Xin Luo; Qing Guo; Felix Juefei-Xu; Xiaojun Jia; Weikai Miao; Geguang Pu; Yang Liu
  The advent of local continuous image function (LIIF) has garnered significant
attention for arbitrary-scale super-resolution (SR) techniques. However, while
the vulnerabilities of fixed-scale SR have been assessed, the robustness of
continuous representation-based arbitrary-scale SR against adversarial attacks
remains an area warranting further exploration. The elaborately designed
adversarial attacks for fixed-scale SR are scale-dependent, which will cause
time-consuming and memory-consuming problems when applied to arbitrary-scale
SR. To address this concern, we propose a simple yet effective
``scale-invariant'' SR adversarial attack method with good transferability,
termed SIAGT. Specifically, we propose to construct resource-saving attacks by
exploiting finite discrete points of continuous representation. In addition, we
formulate a coordinate-dependent loss to enhance the cross-model
transferability of the attack. The attack can significantly deteriorate the SR
images while introducing imperceptible distortion to the targeted
low-resolution (LR) images. Experiments carried out on three popular LIIF-based
SR approaches and four classical SR datasets show remarkable attack performance
and transferability of SIAGT.


http://arxiv.org/abs/2503.04853
From Pixels to Trajectory: Universal Adversarial Example Detection via Temporal Imprints. (98%)
Yansong Gao; Huaibing Peng; Hua Ma; Zhiyang Dai; Shuo Wang; Hongsheng Hu; Anmin Fu; Minhui Xue
  For the first time, we unveil discernible temporal (or historical) trajectory
imprints resulting from adversarial example (AE) attacks. Standing in contrast
to existing studies all focusing on spatial (or static) imprints within the
targeted underlying victim models, we present a fresh temporal paradigm for
understanding these attacks. Of paramount discovery is that these imprints are
encapsulated within a single loss metric, spanning universally across diverse
tasks such as classification and regression, and modalities including image,
text, and audio. Recognizing the distinct nature of loss between adversarial
and clean examples, we exploit this temporal imprint for AE detection by
proposing TRAIT (TRaceable Adversarial temporal trajectory ImprinTs). TRAIT
operates under minimal assumptions without prior knowledge of attacks, thereby
framing the detection challenge as a one-class classification problem. However,
detecting AEs is still challenged by significant overlaps between the
constructed synthetic losses of adversarial and clean examples due to the
absence of ground truth for incoming inputs. TRAIT addresses this challenge by
converting the synthetic loss into a spectrum signature, using the technique of
Fast Fourier Transform to highlight the discrepancies, drawing inspiration from
the temporal nature of the imprints, analogous to time-series signals. Across
12 AE attacks including SMACK (USENIX Sec'2023), TRAIT demonstrates consistent
outstanding performance across comprehensively evaluated modalities, tasks,
datasets, and model architectures. In all scenarios, TRAIT achieves an AE
detection accuracy exceeding 97%, often around 99%, while maintaining a false
rejection rate of 1%. TRAIT remains effective under the formulated strong
adaptive attacks.


http://arxiv.org/abs/2503.04315
Provable Robust Overfitting Mitigation in Wasserstein Distributionally Robust Optimization. (97%)
Shuang Liu; Yihan Wang; Yifan Zhu; Yibo Miao; Xiao-Shan Gao
  Wasserstein distributionally robust optimization (WDRO) optimizes against
worst-case distributional shifts within a specified uncertainty set, leading to
enhanced generalization on unseen adversarial examples, compared to standard
adversarial training which focuses on pointwise adversarial perturbations.
However, WDRO still suffers fundamentally from the robust overfitting problem,
as it does not consider statistical error. We address this gap by proposing a
novel robust optimization framework under a new uncertainty set for adversarial
noise via Wasserstein distance and statistical error via Kullback-Leibler
divergence, called the Statistically Robust WDRO. We establish a robust
generalization bound for the new optimization framework, implying that
out-of-distribution adversarial performance is at least as good as the
statistically robust training loss with high probability. Furthermore, we
derive conditions under which Stackelberg and Nash equilibria exist between the
learner and the adversary, giving an optimal robust model in certain sense.
Finally, through extensive experiments, we demonstrate that our method
significantly mitigates robust overfitting and enhances robustness within the
framework of WDRO.


http://arxiv.org/abs/2503.04963
Energy-Latency Attacks: A New Adversarial Threat to Deep Learning. (81%)
Hanene F. Z. Brachemi Meftah; Wassim Hamidouche; Sid Ahmed Fezza; Olivier Deforges
  The growing computational demand for deep neural networks ( DNNs) has raised
concerns about their energy consumption and carbon footprint, particularly as
the size and complexity of the models continue to increase. To address these
challenges, energy-efficient hardware and custom accelerators have become
essential. Additionally, adaptable DNN s are being developed to dynamically
balance performance and efficiency. The use of these strategies became more
common to enable sustainable AI deployment. However, these efficiency-focused
designs may also introduce vulnerabilities, as attackers can potentially
exploit them to increase latency and energy usage by triggering their
worst-case-performance scenarios. This new type of attack, called
energy-latency attacks, has recently gained significant research attention,
focusing on the vulnerability of DNN s to this emerging attack paradigm, which
can trigger denial-of-service ( DoS) attacks. This paper provides a
comprehensive overview of current research on energy-latency attacks,
categorizing them using the established taxonomy for traditional adversarial
attacks. We explore different metrics used to measure the success of these
attacks and provide an analysis and comparison of existing attack strategies.
We also analyze existing defense mechanisms and highlight current challenges
and potential areas for future research in this developing field. The GitHub
page for this work can be accessed at
https://github.com/hbrachemi/Survey_energy_attacks/


http://arxiv.org/abs/2503.04480
Poisoning Bayesian Inference via Data Deletion and Replication. (62%)
Matthieu Carreau; Roi Naveiro; William N. Caballero
  Research in adversarial machine learning (AML) has shown that statistical
models are vulnerable to maliciously altered data. However, despite advances in
Bayesian machine learning models, most AML research remains concentrated on
classical techniques. Therefore, we focus on extending the white-box model
poisoning paradigm to attack generic Bayesian inference, highlighting its
vulnerability in adversarial contexts. A suite of attacks are developed that
allow an attacker to steer the Bayesian posterior toward a target distribution
through the strategic deletion and replication of true observations, even when
only sampling access to the posterior is available. Analytic properties of
these algorithms are proven and their performance is empirically examined in
both synthetic and real-world scenarios. With relatively little effort, the
attacker is able to substantively alter the Bayesian's beliefs and, by
accepting more risk, they can mold these beliefs to their will. By carefully
constructing the adversarial posterior, surgical poisoning is achieved such
that only targeted inferences are corrupted and others are minimally disturbed.


http://arxiv.org/abs/2503.04474
Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges. (22%)
Francisco Eiras; Eliott Zemour; Eric Lin; Vaikkunth Mugunthan
  Large Language Model (LLM) based judges form the underpinnings of key safety
evaluation processes such as offline benchmarking, automated red-teaming, and
online guardrailing. This widespread requirement raises the crucial question:
can we trust the evaluations of these evaluators? In this paper, we highlight
two critical challenges that are typically overlooked: (i) evaluations in the
wild where factors like prompt sensitivity and distribution shifts can affect
performance and (ii) adversarial attacks that target the judge. We highlight
the importance of these through a study of commonly used safety judges, showing
that small changes such as the style of the model output can lead to jumps of
up to 0.24 in the false negative rate on the same dataset, whereas adversarial
attacks on the model generation can fool some judges into misclassifying 100%
of harmful generations as safe ones. These findings reveal gaps in commonly
used meta-evaluation benchmarks and weaknesses in the robustness of current LLM
judges, indicating that low attack success under certain judges could create a
false sense of security.


http://arxiv.org/abs/2503.04856
One-Shot is Enough: Consolidating Multi-Turn Attacks into Efficient Single-Turn Prompts for LLMs. (12%)
Junwoo Ha; Hyunjun Kim; Sangyoon Yu; Haon Park; Ashkan Yousefpour; Yuna Park; Suhyun Kim
  Despite extensive safety enhancements in large language models (LLMs),
multi-turn "jailbreak" conversations crafted by skilled human adversaries can
still breach even the most sophisticated guardrails. However, these multi-turn
attacks demand considerable manual effort, limiting their scalability. In this
work, we introduce a novel approach called Multi-turn-to-Single-turn (M2S) that
systematically converts multi-turn jailbreak prompts into single-turn attacks.
Specifically, we propose three conversion strategies - Hyphenize, Numberize,
and Pythonize - each preserving sequential context yet packaging it in a single
query. Our experiments on the Multi-turn Human Jailbreak (MHJ) dataset show
that M2S often increases or maintains high Attack Success Rates (ASRs) compared
to original multi-turn conversations. Notably, using a StrongREJECT-based
evaluation of harmfulness, M2S achieves up to 95.9% ASR on Mistral-7B and
outperforms original multi-turn prompts by as much as 17.5% in absolute
improvement on GPT-4o. Further analysis reveals that certain adversarial
tactics, when consolidated into a single prompt, exploit structural formatting
cues to evade standard policy checks. These findings underscore that
single-turn attacks - despite being simpler and cheaper to conduct - can be
just as potent, if not more, than their multi-turn counterparts. Our findings
underscore the urgent need to reevaluate and reinforce LLM safety strategies,
given how adversarial queries can be compacted into a single prompt while still
retaining sufficient complexity to bypass existing safety measures.


http://arxiv.org/abs/2503.03842
Task-Agnostic Attacks Against Vision Foundation Models. (99%)
Brian Pulfer; Yury Belousov; Vitaliy Kinakh; Teddy Furon; Slava Voloshynovskiy
  The study of security in machine learning mainly focuses on downstream
task-specific attacks, where the adversarial example is obtained by optimizing
a loss function specific to the downstream task. At the same time, it has
become standard practice for machine learning practitioners to adopt publicly
available pre-trained vision foundation models, effectively sharing a common
backbone architecture across a multitude of applications such as
classification, segmentation, depth estimation, retrieval, question-answering
and more. The study of attacks on such foundation models and their impact to
multiple downstream tasks remains vastly unexplored. This work proposes a
general framework that forges task-agnostic adversarial examples by maximally
disrupting the feature representation obtained with foundation models. We
extensively evaluate the security of the feature representations obtained by
popular vision foundation models by measuring the impact of this attack on
multiple downstream tasks and its transferability between models.


http://arxiv.org/abs/2503.03272
Towards Effective and Sparse Adversarial Attack on Spiking Neural Networks via Breaking Invisible Surrogate Gradients. (99%)
Li Lun; Kunyu Feng; Qinglong Ni; Ling Liang; Yuan Wang; Ying Li; Dunshan Yu; Xiaoxin Cui
  Spiking neural networks (SNNs) have shown their competence in handling
spatial-temporal event-based data with low energy consumption. Similar to
conventional artificial neural networks (ANNs), SNNs are also vulnerable to
gradient-based adversarial attacks, wherein gradients are calculated by
spatial-temporal back-propagation (STBP) and surrogate gradients (SGs).
However, the SGs may be invisible for an inference-only model as they do not
influence the inference results, and current gradient-based attacks are
ineffective for binary dynamic images captured by the dynamic vision sensor
(DVS). While some approaches addressed the issue of invisible SGs through
universal SGs, their SGs lack a correlation with the victim model, resulting in
sub-optimal performance. Moreover, the imperceptibility of existing SNN-based
binary attacks is still insufficient. In this paper, we introduce an innovative
potential-dependent surrogate gradient (PDSG) method to establish a robust
connection between the SG and the model, thereby enhancing the adaptability of
adversarial attacks across various models with invisible SGs. Additionally, we
propose the sparse dynamic attack (SDA) to effectively attack binary dynamic
images. Utilizing a generation-reduction paradigm, SDA can fully optimize the
sparsity of adversarial perturbations. Experimental results demonstrate that
our PDSG and SDA outperform state-of-the-art SNN-based attacks across various
models and datasets. Specifically, our PDSG achieves 100% attack success rate
on ImageNet, and our SDA obtains 82% attack success rate by modifying only
0.24% of the pixels on CIFAR10DVS. The code is available at
https://github.com/ryime/PDSG-SDA .


http://arxiv.org/abs/2503.03613
CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP. (99%)
Songlong Xing; Zhengyu Zhao; Nicu Sebe
  Despite its prevalent use in image-text matching tasks in a zero-shot manner,
CLIP has been shown to be highly vulnerable to adversarial perturbations added
onto images. Recent studies propose to finetune the vision encoder of CLIP with
adversarial samples generated on the fly, and show improved robustness against
adversarial attacks on a spectrum of downstream datasets, a property termed as
zero-shot robustness. In this paper, we show that malicious perturbations that
seek to maximise the classification loss lead to `falsely stable' images, and
propose to leverage the pre-trained vision encoder of CLIP to counterattack
such adversarial images during inference to achieve robustness. Our paradigm is
simple and training-free, providing the first method to defend CLIP from
adversarial attacks at test time, which is orthogonal to existing methods
aiming to boost zero-shot adversarial robustness of CLIP. We conduct
experiments across 16 classification datasets, and demonstrate stable and
consistent gains compared to test-time defence methods adapted from existing
adversarial robustness studies that do not rely on external networks, without
noticeably impairing performance on clean images. We also show that our
paradigm can be employed on CLIP models that have been adversarially finetuned
to further enhance their robustness at test time. Our code is available
\href{https://github.com/Sxing2/CLIP-Test-time-Counterattacks}{here}.


http://arxiv.org/abs/2503.04825
Adversarial Example Based Fingerprinting for Robust Copyright Protection in Split Learning. (99%)
Zhangting Lin; Mingfu Xue; Kewei Chen; Wenmao Liu; Xiang Gao; Leo Yu Zhang; Jian Wang; Yushu Zhang
  Currently, deep learning models are easily exposed to data leakage risks. As
a distributed model, Split Learning thus emerged as a solution to address this
issue. The model is splitted to avoid data uploading to the server and reduce
computing requirements while ensuring data privacy and security. However, the
transmission of data between clients and server creates a potential
vulnerability. In particular, model is vulnerable to intellectual property (IP)
infringement such as piracy. Alarmingly, a dedicated copyright protection
framework tailored for Split Learning models is still lacking. To this end, we
propose the first copyright protection scheme for Split Learning model,
leveraging fingerprint to ensure effective and robust copyright protection. The
proposed method first generates a set of specifically designed adversarial
examples. Then, we select those examples that would induce misclassifications
to form the fingerprint set. These adversarial examples are embedded as
fingerprints into the model during the training process. Exhaustive experiments
highlight the effectiveness of the scheme. This is demonstrated by a remarkable
fingerprint verification success rate (FVSR) of 100% on MNIST, 98% on CIFAR-10,
and 100% on ImageNet, respectively. Meanwhile, the model's accuracy only
decreases slightly, indicating that the embedded fingerprints do not compromise
model performance. Even under label inference attack, our approach consistently
achieves a high fingerprint verification success rate that ensures robust
verification.


http://arxiv.org/abs/2503.04833
Adversarial Training for Multimodal Large Language Models against Jailbreak Attacks. (95%)
Liming Lu; Shuchao Pang; Siyuan Liang; Haotian Zhu; Xiyu Zeng; Aishan Liu; Yunhuai Liu; Yongbin Zhou
  Multimodal large language models (MLLMs) have made remarkable strides in
cross-modal comprehension and generation tasks. However, they remain vulnerable
to jailbreak attacks, where crafted perturbations bypass security guardrails
and elicit harmful outputs. In this paper, we present the first adversarial
training (AT) paradigm tailored to defend against jailbreak attacks during the
MLLM training phase. Extending traditional AT to this domain poses two critical
challenges: efficiently tuning massive parameters and ensuring robustness
against attacks across multiple modalities. To address these challenges, we
introduce Projection Layer Against Adversarial Training (ProEAT), an end-to-end
AT framework. ProEAT incorporates a projector-based adversarial training
architecture that efficiently handles large-scale parameters while maintaining
computational feasibility by focusing adversarial training on a lightweight
projector layer instead of the entire model; additionally, we design a dynamic
weight adjustment mechanism that optimizes the loss function's weight
allocation based on task demands, streamlining the tuning process. To enhance
defense performance, we propose a joint optimization strategy across visual and
textual modalities, ensuring robust resistance to jailbreak attacks originating
from either modality. Extensive experiments conducted on five major jailbreak
attack methods across three mainstream MLLMs demonstrate the effectiveness of
our approach. ProEAT achieves state-of-the-art defense performance,
outperforming existing baselines by an average margin of +34% across text and
image modalities, while incurring only a 1% reduction in clean accuracy.
Furthermore, evaluations on real-world embodied intelligent systems highlight
the practical applicability of our framework, paving the way for the
development of more secure and reliable multimodal systems.


http://arxiv.org/abs/2503.03454
Data Poisoning Attacks to Locally Differentially Private Range Query Protocols. (87%)
Ting-Wei Liao; Chih-Hsun Lin; Yu-Lin Tsai; Takao Murakami; Chia-Mu Yu; Jun Sakuma; Chun-Ying Huang; Hiroaki Kikuchi
  Local Differential Privacy (LDP) has been widely adopted to protect user
privacy in decentralized data collection. However, recent studies have revealed
that LDP protocols are vulnerable to data poisoning attacks, where malicious
users manipulate their reported data to distort aggregated results. In this
work, we present the first study on data poisoning attacks targeting LDP range
query protocols, focusing on both tree-based and grid-based approaches. We
identify three key challenges in executing such attacks, including crafting
consistent and effective fake data, maintaining data consistency across levels
or grids, and preventing server detection. To address the first two challenges,
we propose novel attack methods that are provably optimal, including a
tree-based attack and a grid-based attack, designed to manipulate range query
results with high effectiveness. \textbf{Our key finding is that the common
post-processing procedure, Norm-Sub, in LDP range query protocols can help the
attacker massively amplify their attack effectiveness.} In addition, we study a
potential countermeasure, but also propose an adaptive attack capable of
evading this defense to address the third challenge. We evaluate our methods
through theoretical analysis and extensive experiments on synthetic and
real-world datasets. Our results show that the proposed attacks can
significantly amplify estimations for arbitrary range queries by manipulating a
small fraction of users, providing 5-10x more influence than a normal user to
the estimation.


http://arxiv.org/abs/2503.03201
Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution. (83%)
Jizhao Zhu; Akang Shi; Zixuan Li; Long Bai; Xiaolong Jin; Jiafeng Guo; Xueqi Cheng
  In this paper, we aim to enhance the robustness of Universal Information
Extraction (UIE) by introducing a new benchmark dataset, a comprehensive
evaluation, and a feasible solution. Existing robust benchmark datasets have
two key limitations: 1) They generate only a limited range of perturbations for
a single Information Extraction (IE) task, which fails to evaluate the
robustness of UIE models effectively; 2) They rely on small models or
handcrafted rules to generate perturbations, often resulting in unnatural
adversarial examples. Considering the powerful generation capabilities of Large
Language Models (LLMs), we introduce a new benchmark dataset for Robust UIE,
called RUIE-Bench, which utilizes LLMs to generate more diverse and realistic
perturbations across different IE tasks. Based on this dataset, we
comprehensively evaluate existing UIE models and reveal that both LLM-based
models and other models suffer from significant performance drops. To improve
robustness and reduce training costs, we propose a data-augmentation solution
that dynamically selects hard samples for iterative training based on the
model's inference loss. Experimental results show that training with only
\textbf{15\%} of the data leads to an average \textbf{7.5\%} relative
performance improvement across three IE tasks.


http://arxiv.org/abs/2503.03502
CURVALID: Geometrically-guided Adversarial Prompt Detection. (75%)
Canaan Yung; Hanxun Huang; Sarah Monazam Erfani; Christopher Leckie
  Adversarial prompts capable of jailbreaking large language models (LLMs) and
inducing undesirable behaviours pose a significant obstacle to their safe
deployment. Current mitigation strategies rely on activating built-in defence
mechanisms or fine-tuning the LLMs, but the fundamental distinctions between
adversarial and benign prompts are yet to be understood. In this work, we
introduce CurvaLID, a novel defense framework that efficiently detects
adversarial prompts by leveraging their geometric properties. It is agnostic to
the type of LLM, offering a unified detection framework across diverse
adversarial prompts and LLM architectures. CurvaLID builds on the geometric
analysis of text prompts to uncover their underlying differences. We
theoretically extend the concept of curvature via the Whewell equation into an
$n$-dimensional word embedding space, enabling us to quantify local geometric
properties, including semantic shifts and curvature in the underlying
manifolds. Additionally, we employ Local Intrinsic Dimensionality (LID) to
capture geometric features of text prompts within adversarial subspaces. Our
findings reveal that adversarial prompts differ fundamentally from benign
prompts in terms of their geometric characteristics. Our results demonstrate
that CurvaLID delivers superior detection and rejection of adversarial queries,
paving the way for safer LLM deployment. The source code can be found at
https://github.com/Cancanxxx/CurvaLID


http://arxiv.org/abs/2503.03944
GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoors. (13%)
Yaopei Zeng; Yuanpu Cao; Lu Lin
  The growing accessibility of diffusion models has revolutionized image
editing but also raised significant concerns about unauthorized modifications,
such as misinformation and plagiarism. Existing countermeasures largely rely on
adversarial perturbations designed to disrupt diffusion model outputs. However,
these approaches are found to be easily neutralized by simple image
preprocessing techniques, such as compression and noise addition. To address
this limitation, we propose GuardDoor, a novel and robust protection mechanism
that fosters collaboration between image owners and model providers.
Specifically, the model provider participating in the mechanism fine-tunes the
image encoder to embed a protective backdoor, allowing image owners to request
the attachment of imperceptible triggers to their images. When unauthorized
users attempt to edit these protected images with this diffusion model, the
model produces meaningless outputs, reducing the risk of malicious image
editing. Our method demonstrates enhanced robustness against image
preprocessing operations and is scalable for large-scale deployment. This work
underscores the potential of cooperative frameworks between model providers and
image owners to safeguard digital content in the era of generative AI.


http://arxiv.org/abs/2503.07483
Poisoning Attacks to Local Differential Privacy Protocols for Trajectory Data. (13%)
I-Jung Hsu; Chih-Hsun Lin; Chia-Mu Yu; Sy-Yen Kuo; Chun-Ying Huang
  Trajectory data, which tracks movements through geographic locations, is
crucial for improving real-world applications. However, collecting such
sensitive data raises considerable privacy concerns. Local differential privacy
(LDP) offers a solution by allowing individuals to locally perturb their
trajectory data before sharing it. Despite its privacy benefits, LDP protocols
are vulnerable to data poisoning attacks, where attackers inject fake data to
manipulate aggregated results. In this work, we make the first attempt to
analyze vulnerabilities in several representative LDP trajectory protocols. We
propose \textsc{TraP}, a heuristic algorithm for data \underline{P}oisoning
attacks using a prefix-suffix method to optimize fake \underline{Tra}jectory
selection, significantly reducing computational complexity. Our experimental
results demonstrate that our attack can substantially increase target pattern
occurrences in the perturbed trajectory dataset with few fake users. This study
underscores the urgent need for robust defenses and better protocol designs to
safeguard LDP trajectory data against malicious manipulation.


http://arxiv.org/abs/2503.03710
Improving LLM Safety Alignment with Dual-Objective Optimization. (9%)
Xuandong Zhao; Will Cai; Tianneng Shi; David Huang; Licong Lin; Song Mei; Dawn Song
  Existing training-time safety alignment techniques for large language models
(LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization
(DPO), a widely deployed alignment method, exhibits limitations in both
experimental and theoretical contexts as its loss function proves suboptimal
for refusal learning. Through gradient-based analysis, we identify these
shortcomings and propose an improved safety alignment that disentangles DPO
objectives into two components: (1) robust refusal training, which encourages
refusal even when partial unsafe generations are produced, and (2) targeted
unlearning of harmful knowledge. This approach significantly increases LLM
robustness against a wide range of jailbreak attacks, including prefilling,
suffix, and multi-turn attacks across both in-distribution and
out-of-distribution scenarios. Furthermore, we introduce a method to emphasize
critical refusal tokens by incorporating a reward-based token-level weighting
mechanism for refusal learning, which further improves the robustness against
adversarial exploits. Our research also suggests that robustness to jailbreak
attacks is correlated with token distribution shifts in the training process
and internal representations of refusal and harmful tokens, offering valuable
directions for future research in LLM safety alignment. The code is available
at https://github.com/wicai24/DOOR-Alignment


http://arxiv.org/abs/2503.02986
Mind the Gap: Detecting Black-box Adversarial Attacks in the Making through Query Update Analysis. (99%)
Jeonghwan Park; Niall McLaughlin; Ihsen Alouani
  Adversarial attacks remain a significant threat that can jeopardize the
integrity of Machine Learning (ML) models. In particular, query-based black-box
attacks can generate malicious noise without having access to the victim
model's architecture, making them practical in real-world contexts. The
community has proposed several defenses against adversarial attacks, only to be
broken by more advanced and adaptive attack strategies. In this paper, we
propose a framework that detects if an adversarial noise instance is being
generated. Unlike existing stateful defenses that detect adversarial noise
generation by monitoring the input space, our approach learns adversarial
patterns in the input update similarity space. In fact, we propose to observe a
new metric called Delta Similarity (DS), which we show it captures more
efficiently the adversarial behavior. We evaluate our approach against 8
state-of-the-art attacks, including adaptive attacks, where the adversary is
aware of the defense and tries to evade detection. We find that our approach is
significantly more robust than existing defenses both in terms of specificity
and sensitivity.


http://arxiv.org/abs/2503.03170
AttackSeqBench: Benchmarking Large Language Models' Understanding of Sequential Patterns in Cyber Attacks. (1%)
Javier Yong; Haokai Ma; Yunshan Ma; Anis Yusof; Zhenkai Liang; Ee-Chien Chang
  The observations documented in Cyber Threat Intelligence (CTI) reports play a
critical role in describing adversarial behaviors, providing valuable insights
for security practitioners to respond to evolving threats. Recent advancements
of Large Language Models (LLMs) have demonstrated significant potential in
various cybersecurity applications, including CTI report understanding and
attack knowledge graph construction. While previous works have proposed
benchmarks that focus on the CTI extraction ability of LLMs, the sequential
characteristic of adversarial behaviors within CTI reports remains largely
unexplored, which holds considerable significance in developing a comprehensive
understanding of how adversaries operate. To address this gap, we introduce
AttackSeqBench, a benchmark tailored to systematically evaluate LLMs'
capability to understand and reason attack sequences in CTI reports. Our
benchmark encompasses three distinct Question Answering (QA) tasks, each task
focuses on the varying granularity in adversarial behavior. To alleviate the
laborious effort of QA construction, we carefully design an automated dataset
construction pipeline to create scalable and well-formulated QA datasets based
on real-world CTI reports. To ensure the quality of our dataset, we adopt a
hybrid approach of combining human evaluation and systematic evaluation
metrics. We conduct extensive experiments and analysis with both fast-thinking
and slow-thinking LLMs, while highlighting their strengths and limitations in
analyzing the sequential patterns in cyber attacks. The overarching goal of
this work is to provide a benchmark that advances LLM-driven CTI report
understanding and fosters its application in real-world cybersecurity
operations. Our dataset and code are available at
https://github.com/Javiery3889/AttackSeqBench .


http://arxiv.org/abs/2503.02169
One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy. (99%)
Jiacheng Zhang; Benjamin I. P. Rubinstein; Jingfeng Zhang; Feng Liu
  Statistical adversarial data detection (SADD) detects whether an upcoming
batch contains adversarial examples (AEs) by measuring the distributional
discrepancies between clean examples (CEs) and AEs. In this paper, we explore
the strength of SADD-based methods by theoretically showing that minimizing
distributional discrepancy can help reduce the expected loss on AEs. Despite
these advantages, SADD-based methods have a potential limitation: they discard
inputs that are detected as AEs, leading to the loss of useful information
within those inputs. To address this limitation, we propose a two-pronged
adversarial defense method, named Distributional-discrepancy-based Adversarial
Defense (DAD). In the training phase, DAD first optimizes the test power of the
maximum mean discrepancy (MMD) to derive MMD-OPT, which is a stone that kills
two birds. MMD-OPT first serves as a guiding signal to minimize the
distributional discrepancy between CEs and AEs to train a denoiser. Then, it
serves as a discriminator to differentiate CEs and AEs during inference.
Overall, in the inference stage, DAD consists of a two-pronged process: (1)
directly feeding the detected CEs into the classifier, and (2) removing noise
from the detected AEs by the distributional-discrepancy-based denoiser.
Extensive experiments show that DAD outperforms current state-of-the-art (SOTA)
defense methods by simultaneously improving clean and robust accuracy on
CIFAR-10 and ImageNet-1K against adaptive white-box attacks. Codes are publicly
available at: https://github.com/tmlr-group/DAD.


http://arxiv.org/abs/2503.01407
Divide and Conquer: Heterogeneous Noise Integration for Diffusion-based Adversarial Purification. (80%)
Gaozheng Pei; Shaojie Lyu; Gong Chen; Ke Ma; Qianqian Xu; Yingfei Sun; Qingming Huang
  Existing diffusion-based purification methods aim to disrupt adversarial
perturbations by introducing a certain amount of noise through a forward
diffusion process, followed by a reverse process to recover clean examples.
However, this approach is fundamentally flawed: the uniform operation of the
forward process across all pixels compromises normal pixels while attempting to
combat adversarial perturbations, resulting in the target model producing
incorrect predictions. Simply relying on low-intensity noise is insufficient
for effective defense. To address this critical issue, we implement a
heterogeneous purification strategy grounded in the interpretability of neural
networks. Our method decisively applies higher-intensity noise to specific
pixels that the target model focuses on while the remaining pixels are
subjected to only low-intensity noise. This requirement motivates us to
redesign the sampling process of the diffusion model, allowing for the
effective removal of varying noise levels. Furthermore, to evaluate our method
against strong adaptative attack, our proposed method sharply reduces time cost
and memory usage through a single-step resampling. The empirical evidence from
extensive experiments across three datasets demonstrates that our method
outperforms most current adversarial training and purification techniques by a
substantial margin.


http://arxiv.org/abs/2503.02017
A Lightweight and Secure Deep Learning Model for Privacy-Preserving Federated Learning in Intelligent Enterprises. (1%)
Reza Fotohi; Fereidoon Shams Aliee; Bahar Farahani
  The ever growing Internet of Things (IoT) connections drive a new type of
organization, the Intelligent Enterprise. In intelligent enterprises, machine
learning based models are adopted to extract insights from data. Due to the
efficiency and privacy challenges of these traditional models, a new federated
learning (FL) paradigm has emerged. In FL, multiple enterprises can jointly
train a model to update a final model. However, firstly, FL trained models
usually perform worse than centralized models, especially when enterprises
training data is non-IID (Independent and Identically Distributed). Second, due
to the centrality of FL and the untrustworthiness of local enterprises,
traditional FL solutions are vulnerable to poisoning and inference attacks and
violate privacy. Thirdly, the continuous transfer of parameters between
enterprises and servers increases communication costs. To this end, the
FedAnil+ model is proposed, a novel, lightweight, and secure Federated Deep
Learning Model that includes three main phases. In the first phase, the goal is
to solve the data type distribution skew challenge. Addressing privacy concerns
against poisoning and inference attacks is covered in the second phase.
Finally, to alleviate the communication overhead, a novel compression approach
is proposed that significantly reduces the size of the updates. The experiment
results validate that FedAnil+ is secure against inference and poisoning
attacks with better accuracy. In addition, it shows improvements over existing
approaches in terms of model accuracy (13%, 16%, and 26%), communication cost
(17%, 21%, and 25%), and computation cost (7%, 9%, and 11%).


http://arxiv.org/abs/2503.01944
Protecting DeFi Platforms against Non-Price Flash Loan Attacks. (1%)
Abdulrahman Alhaidari; Balaji Palanisamy; Prashant Krishnamurthy
  Smart contracts in Decentralized Finance (DeFi) platforms are attractive
targets for attacks as their vulnerabilities can lead to massive amounts of
financial losses. Flash loan attacks, in particular, pose a major threat to
DeFi protocols that hold a Total Value Locked (TVL) exceeding \$106 billion.
These attacks use the atomicity property of blockchains to drain funds from
smart contracts in a single transaction. While existing research primarily
focuses on price manipulation attacks, such as oracle manipulation, mitigating
non-price flash loan attacks that often exploit smart contracts' zero-day
vulnerabilities remains largely unaddressed. These attacks are challenging to
detect because of their unique patterns, time sensitivity, and complexity. In
this paper, we present FlashGuard, a runtime detection and mitigation method
for non-price flash loan attacks. Our approach targets smart contract function
signatures to identify attacks in real-time and counterattack by disrupting the
attack transaction atomicity by leveraging the short window when transactions
are visible in the mempool but not yet confirmed. When FlashGuard detects an
attack, it dispatches a stealthy dusting counterattack transaction to miners to
change the victim contract's state which disrupts the attack's atomicity and
forces the attack transaction to revert. We evaluate our approach using 20
historical attacks and several unseen attacks. FlashGuard achieves an average
real-time detection latency of 150.31ms, a detection accuracy of over 99.93\%,
and an average disruption time of 410.92ms. FlashGuard could have potentially
rescued over \$405.71 million in losses if it were deployed prior to these
attack instances. FlashGuard demonstrates significant potential as a DeFi
security solution to mitigate and handle rising threats of non-price flash loan
attacks.


http://arxiv.org/abs/2503.00932
Improving the Transferability of Adversarial Attacks by an Input Transpose. (99%)
Qing Wan; Shilong Deng; Xun Wang
  Deep neural networks (DNNs) are highly susceptible to adversarial
examples--subtle perturbations applied to inputs that are often imperceptible
to humans yet lead to incorrect model predictions. In black-box scenarios,
however, existing adversarial examples exhibit limited transferability and
struggle to effectively compromise multiple unseen DNN models. Previous
strategies enhance the cross-model generalization of adversarial examples by
introducing versatility into adversarial perturbations, thereby improving
transferability. However, further refining perturbation versatility often
demands intricate algorithm development and substantial computation
consumption. In this work, we propose an input transpose method that requires
almost no additional labor and computation costs but can significantly improve
the transferability of existing adversarial strategies. Even without adding
adversarial perturbations, our method demonstrates considerable effectiveness
in cross-model attacks. Our exploration finds that on specific datasets, a mere
$1^\circ$ left or right rotation might be sufficient for most adversarial
examples to deceive unseen models. Our further analysis suggests that this
transferability improvement triggered by rotating only $1^\circ$ may stem from
visible pattern shifts in the DNN's low-level feature maps. Moreover, this
transferability exhibits optimal angles that, when identified under
unrestricted query conditions, could potentially yield even greater
performance.


http://arxiv.org/abs/2503.00957
Exploiting Vulnerabilities in Speech Translation Systems through Targeted Adversarial Attacks. (99%)
Chang Liu; Haolin Wu; Xi Yang; Kui Zhang; Cong Wu; Weiming Zhang; Nenghai Yu; Tianwei Zhang; Qing Guo; Jie Zhang
  As speech translation (ST) systems become increasingly prevalent,
understanding their vulnerabilities is crucial for ensuring robust and reliable
communication. However, limited work has explored this issue in depth. This
paper explores methods of compromising these systems through imperceptible
audio manipulations. Specifically, we present two innovative approaches: (1)
the injection of perturbation into source audio, and (2) the generation of
adversarial music designed to guide targeted translation, while also conducting
more practical over-the-air attacks in the physical world. Our experiments
reveal that carefully crafted audio perturbations can mislead translation
models to produce targeted, harmful outputs, while adversarial music achieve
this goal more covertly, exploiting the natural imperceptibility of music.
These attacks prove effective across multiple languages and translation models,
highlighting a systemic vulnerability in current ST architectures. The
implications of this research extend beyond immediate security concerns,
shedding light on the interpretability and robustness of neural speech
processing systems. Our findings underscore the need for advanced defense
mechanisms and more resilient architectures in the realm of audio systems. More
details and samples can be found at https://adv-st.github.io.


http://arxiv.org/abs/2503.00917
AMUN: Adversarial Machine UNlearning. (92%)
Ali Ebrahimpour-Boroojeny; Hari Sundaram; Varun Chandrasekaran
  Machine unlearning, where users can request the deletion of a forget dataset,
is becoming increasingly important because of numerous privacy regulations.
Initial works on ``exact'' unlearning (e.g., retraining) incur large
computational overheads. However, while computationally inexpensive,
``approximate'' methods have fallen short of reaching the effectiveness of
exact unlearning: models produced fail to obtain comparable accuracy and
prediction confidence on both the forget and test (i.e., unseen) dataset.
Exploiting this observation, we propose a new unlearning method, Adversarial
Machine UNlearning (AMUN), that outperforms prior state-of-the-art (SOTA)
methods for image classification. AMUN lowers the confidence of the model on
the forget samples by fine-tuning the model on their corresponding adversarial
examples. Adversarial examples naturally belong to the distribution imposed by
the model on the input space; fine-tuning the model on the adversarial examples
closest to the corresponding forget samples (a) localizes the changes to the
decision boundary of the model around each forget sample and (b) avoids drastic
changes to the global behavior of the model, thereby preserving the model's
accuracy on test samples. Using AMUN for unlearning a random $10\%$ of CIFAR-10
samples, we observe that even SOTA membership inference attacks cannot do
better than random guessing.


http://arxiv.org/abs/2503.01924
TAET: Two-Stage Adversarial Equalization Training on Long-Tailed Distributions. (54%)
Wang YuHang; Junkang Guo; Aolei Liu; Kaihao Wang; Zaitong Wu; Zhenyu Liu; Wenfei Yin; Jian Liu
  Adversarial robustness is a critical challenge in deploying deep neural
networks for real-world applications. While adversarial training is a widely
recognized defense strategy, most existing studies focus on balanced datasets,
overlooking the prevalence of long-tailed distributions in real-world data,
which significantly complicates robustness. This paper provides a comprehensive
analysis of adversarial training under long-tailed distributions and identifies
limitations in the current state-of-the-art method, AT-BSL, in achieving robust
performance under such conditions. To address these challenges, we propose a
novel training framework, TAET, which integrates an initial stabilization phase
followed by a stratified equalization adversarial training phase. Additionally,
prior work on long-tailed robustness has largely ignored the crucial evaluation
metric of balanced accuracy. To bridge this gap, we introduce the concept of
balanced robustness, a comprehensive metric tailored for assessing robustness
under long-tailed distributions. Extensive experiments demonstrate that our
method surpasses existing advanced defenses, achieving significant improvements
in both memory and computational efficiency. This work represents a substantial
advancement in addressing robustness challenges in real-world applications. Our
code is available at:
https://github.com/BuhuiOK/TAET-Two-Stage-Adversarial-Equalization-Training-on-Long-Tailed-Distributions.


http://arxiv.org/abs/2503.01926
Unnatural Languages Are Not Bugs but Features for LLMs. (1%)
Keyu Duan; Yiran Zhao; Zhili Feng; Jinjie Ni; Tianyu Pang; Qian Liu; Tianle Cai; Longxu Dou; Kenji Kawaguchi; Anirudh Goyal; J. Zico Kolter; Michael Qizhe Shieh
  Large Language Models (LLMs) have been observed to process non-human-readable
text sequences, such as jailbreak prompts, often viewed as a bug for aligned
LLMs. In this work, we present a systematic investigation challenging this
perception, demonstrating that unnatural languages - strings that appear
incomprehensible to humans but maintain semantic meanings for LLMs - contain
latent features usable by models. Notably, unnatural languages possess latent
features that can be generalized across different models and tasks during
inference. Furthermore, models fine-tuned on unnatural versions of instruction
datasets perform on-par with those trained on natural language, achieving 49.71
win rates in Length-controlled AlpacaEval 2.0 in average across various base
models. In addition, through comprehensive analysis, we demonstrate that LLMs
process unnatural languages by filtering noise and inferring contextual meaning
from filtered words.


http://arxiv.org/abs/2503.00384
A Survey of Adversarial Defenses in Vision-based Systems: Categorization, Methods and Challenges. (99%)
Nandish Chattopadhyay; Abdul Basit; Bassem Ouni; Muhammad Shafique
  Adversarial attacks have emerged as a major challenge to the trustworthy
deployment of machine learning models, particularly in computer vision
applications. These attacks have a varied level of potency and can be
implemented in both white box and black box approaches. Practical attacks
include methods to manipulate the physical world and enforce adversarial
behaviour by the corresponding target neural network models. Multiple different
approaches to mitigate different kinds of such attacks are available in the
literature, each with their own advantages and limitations. In this survey, we
present a comprehensive systematization of knowledge on adversarial defenses,
focusing on two key computer vision tasks: image classification and object
detection. We review the state-of-the-art adversarial defense techniques and
categorize them for easier comparison. In addition, we provide a schematic
representation of these categories within the context of the overall machine
learning pipeline, facilitating clearer understanding and benchmarking of
defenses. Furthermore, we map these defenses to the types of adversarial
attacks and datasets where they are most effective, offering practical insights
for researchers and practitioners. This study is necessary for understanding
the scope of how the available defenses are able to address the adversarial
threats, and their shortcomings as well, which is necessary for driving the
research in this area in the most appropriate direction, with the aim of
building trustworthy AI systems for regular practical use-cases.


http://arxiv.org/abs/2503.00377
Adversarial Attacks on Event-Based Pedestrian Detectors: A Physical Approach. (82%)
Guixu Lin; Muyao Niu; Qingtian Zhu; Zhengwei Yin; Zhuoxiao Li; Shengfeng He; Yinqiang Zheng
  Event cameras, known for their low latency and high dynamic range, show great
potential in pedestrian detection applications. However, while recent research
has primarily focused on improving detection accuracy, the robustness of
event-based visual models against physical adversarial attacks has received
limited attention. For example, adversarial physical objects, such as specific
clothing patterns or accessories, can exploit inherent vulnerabilities in these
systems, leading to misdetections or misclassifications. This study is the
first to explore physical adversarial attacks on event-driven pedestrian
detectors, specifically investigating whether certain clothing patterns worn by
pedestrians can cause these detectors to fail, effectively rendering them
unable to detect the person. To address this, we developed an end-to-end
adversarial framework in the digital domain, framing the design of adversarial
clothing textures as a 2D texture optimization problem. By crafting an
effective adversarial loss function, the framework iteratively generates
optimal textures through backpropagation. Our results demonstrate that the
textures identified in the digital domain possess strong adversarial
properties. Furthermore, we translated these digitally optimized textures into
physical clothing and tested them in real-world scenarios, successfully
demonstrating that the designed textures significantly degrade the performance
of event-based pedestrian detection models. This work highlights the
vulnerability of such models to physical adversarial attacks.


http://arxiv.org/abs/2503.00596
BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judge. (16%)
Terry Tong; Fei Wang; Zhe Zhao; Muhao Chen
  This paper proposes a novel backdoor threat attacking the LLM-as-a-Judge
evaluation regime, where the adversary controls both the candidate and
evaluator model. The backdoored evaluator victimizes benign users by unfairly
assigning inflated scores to adversary. A trivial single token backdoor
poisoning 1% of the evaluator training data triples the adversary's score with
respect to their legitimate score. We systematically categorize levels of data
access corresponding to three real-world settings, (1) web poisoning, (2)
malicious annotator, and (3) weight poisoning. These regimes reflect a weak to
strong escalation of data access that highly correlates with attack severity.
Under the weakest assumptions - web poisoning (1), the adversary still induces
a 20% score inflation. Likewise, in the (3) weight poisoning regime, the
stronger assumptions enable the adversary to inflate their scores from 1.5/5 to
4.9/5. The backdoor threat generalizes across different evaluator
architectures, trigger designs, evaluation tasks, and poisoning rates. By
poisoning 10% of the evaluator training data, we control toxicity judges
(Guardrails) to misclassify toxic prompts as non-toxic 89% of the time, and
document reranker judges in RAG to rank the poisoned document first 97% of the
time. LLM-as-a-Judge is uniquely positioned at the intersection of ethics and
technology, where social implications of mislead model selection and evaluation
constrain the available defensive tools. Amidst these challenges, model merging
emerges as a principled tool to offset the backdoor, reducing ASR to near 0%
whilst maintaining SOTA performance. Model merging's low computational cost and
convenient integration into the current LLM Judge training pipeline position it
as a promising avenue for backdoor mitigation in the LLM-as-a-Judge setting.


http://arxiv.org/abs/2503.00615
xIDS-EnsembleGuard: An Explainable Ensemble Learning-based Intrusion Detection System. (1%)
Muhammad Adil; Mian Ahmad Jan; Safayat Bin Hakim; Houbing Herbert Song; Zhanpeng Jin
  In this paper, we focus on addressing the challenges of detecting malicious
attacks in networks by designing an advanced Explainable Intrusion Detection
System (xIDS). The existing machine learning and deep learning approaches have
invisible limitations, such as potential biases in predictions, a lack of
interpretability, and the risk of overfitting to training data. These issues
can create doubt about their usefulness, transparency, and a decrease in trust
among stakeholders. To overcome these challenges, we propose an ensemble
learning technique called "EnsembleGuard." This approach uses the predicted
outputs of multiple models, including tree-based methods (LightGBM, GBM,
Bagging, XGBoost, CatBoost) and deep learning models such as LSTM (long
short-term memory) and GRU (gated recurrent unit), to maintain a balance and
achieve trustworthy results. Our work is unique because it combines both
tree-based and deep learning models to design an interpretable and explainable
meta-model through model distillation. By considering the predictions of all
individual models, our meta-model effectively addresses key challenges and
ensures both explainable and reliable results. We evaluate our model using
well-known datasets, including UNSW-NB15, NSL-KDD, and CIC-IDS-2017, to assess
its reliability against various types of attacks. During analysis, we found
that our model outperforms both tree-based models and other comparative
approaches in different attack scenarios.


http://arxiv.org/abs/2503.00687
Transformer Meets Twicing: Harnessing Unattended Residual Information. (1%)
Laziz Abdullaev; Tan M. Nguyen
  Transformer-based deep learning models have achieved state-of-the-art
performance across numerous language and vision tasks. While the self-attention
mechanism, a core component of transformers, has proven capable of handling
complex data patterns, it has been observed that the representational capacity
of the attention matrix degrades significantly across transformer layers,
thereby hurting its overall performance. In this work, we leverage the
connection between self-attention computations and low-pass non-local means
(NLM) smoothing filters and propose the Twicing Attention, a novel attention
mechanism that uses kernel twicing procedure in nonparametric regression to
alleviate the low-pass behavior of associated NLM smoothing with compelling
theoretical guarantees and enhanced adversarial robustness. This approach
enables the extraction and reuse of meaningful information retained in the
residuals following the imperfect smoothing operation at each layer. Our
proposed method offers two key advantages over standard self-attention: 1) a
provably slower decay of representational capacity and 2) improved robustness
and accuracy across various data modalities and tasks. We empirically
demonstrate the performance gains of our model over baseline transformers on
multiple tasks and benchmarks, including image classification and language
modeling, on both clean and corrupted data.


http://arxiv.org/abs/2502.21171
QFAL: Quantum Federated Adversarial Learning. (99%)
Walid El Maouaki; Nouhaila Innan; Alberto Marchisio; Taoufik Said; Mohamed Bennai; Muhammad Shafique
  Quantum federated learning (QFL) merges the privacy advantages of federated
systems with the computational potential of quantum neural networks (QNNs), yet
its vulnerability to adversarial attacks remains poorly understood. This work
pioneers the integration of adversarial training into QFL, proposing a robust
framework, quantum federated adversarial learning (QFAL), where clients
collaboratively defend against perturbations by combining local adversarial
example generation with federated averaging (FedAvg). We systematically
evaluate the interplay between three critical factors: client count (5, 10,
15), adversarial training coverage (0-100%), and adversarial attack
perturbation strength (epsilon = 0.01-0.5), using the MNIST dataset. Our
experimental results show that while fewer clients often yield higher
clean-data accuracy, larger federations can more effectively balance accuracy
and robustness when partially adversarially trained. Notably, even limited
adversarial coverage (e.g., 20%-50%) can significantly improve resilience to
moderate perturbations, though at the cost of reduced baseline performance.
Conversely, full adversarial training (100%) may regain high clean accuracy but
is vulnerable under stronger attacks. These findings underscore an inherent
trade-off between robust and standard objectives, which is further complicated
by quantum-specific factors. We conclude that a carefully chosen combination of
client count and adversarial coverage is critical for mitigating adversarial
vulnerabilities in QFL. Moreover, we highlight opportunities for future
research, including adaptive adversarial training schedules, more diverse
quantum encoding schemes, and personalized defense strategies to further
enhance the robustness-accuracy trade-off in real-world quantum federated
environments.


http://arxiv.org/abs/2502.21048
Data-free Universal Adversarial Perturbation with Pseudo-semantic Prior. (99%)
Chanhui Lee; Yeonghwan Song; Jeany Son
  Data-free Universal Adversarial Perturbation (UAP) is an image-agnostic
adversarial attack that deceives deep neural networks using a single
perturbation generated solely from random noise without relying on data priors.
However, traditional data-free UAP methods often suffer from limited
transferability due to the absence of semantic content in random noise. To
address this issue, we propose a novel data-free universal attack method that
recursively extracts pseudo-semantic priors directly from the UAPs during
training to enrich the semantic content within the data-free UAP framework. Our
approach effectively leverages latent semantic information within UAPs via
region sampling, enabling successful input transformations-typically
ineffective in traditional data-free UAP methods due to the lack of semantic
cues-and significantly enhancing black-box transferability. Furthermore, we
introduce a sample reweighting technique to mitigate potential imbalances from
random sampling and transformations, emphasizing hard examples less affected by
the UAPs. Comprehensive experiments on ImageNet show that our method achieves
state-of-the-art performance in average fooling rate by a substantial margin,
notably improves attack transferability across various CNN architectures
compared to existing data-free UAP methods, and even surpasses data-dependent
UAP methods. Code is available at: https://github.com/ChnanChan/PSP-UAP.


http://arxiv.org/abs/2502.20948
Concealed Adversarial attacks on neural networks for sequential data. (98%)
Petr Sokerin; Dmitry Anikin; Sofia Krehova; Alexey Zaytsev
  The emergence of deep learning led to the broad usage of neural networks in
the time series domain for various applications, including finance and
medicine. While powerful, these models are prone to adversarial attacks: a
benign targeted perturbation of input data leads to significant changes in a
classifier's output. However, formally small attacks in the time series domain
become easily detected by the human eye or a simple detector model.
  We develop a concealed adversarial attack for different time-series models:
it provides more realistic perturbations, being hard to detect by a human or
model discriminator. To achieve this goal, the proposed adversarial attack
maximizes an aggregation of a classifier and a trained discriminator loss. To
make the attack stronger, we also propose a training procedure for a
discriminator that provides broader coverage of possible attacks. Extensive
benchmarking on six UCR time series datasets across four diverse architectures
- including recurrent, convolutional, state-space, and transformer-based models
- demonstrates the superiority of our attack for a concealability-efficiency
trade-off. Our findings highlight the growing challenge of designing robust
time series models, emphasizing the need for improved defenses against
realistic and effective attacks.


http://arxiv.org/abs/2503.00140
Approaching the Harm of Gradient Attacks While Only Flipping Labels. (92%)
Abdessamad El-Kabid; El-Mahdi El-Mhamdi
  Machine learning systems deployed in distributed or federated environments
are highly susceptible to adversarial manipulations, particularly availability
attacks -adding imperceptible perturbations to training data, thereby rendering
the trained model unavailable. Prior research in distributed machine learning
has demonstrated such adversarial effects through the injection of gradients or
data poisoning. In this study, we aim to enhance comprehension of the potential
of weaker (and more probable) adversaries by posing the following inquiry: Can
availability attacks be inflicted solely through the flipping of a subset of
training labels, without altering features, and under a strict flipping budget?
We analyze the extent of damage caused by constrained label flipping attacks.
Focusing on a distributed classification problem, (1) we propose a novel
formalization of label flipping attacks on logistic regression models and
derive a greedy algorithm that is provably optimal at each training step. (2)
To demonstrate that availability attacks can be approached by label flipping
alone, we show that a budget of only $0.1\%$ of labels at each training step
can reduce the accuracy of the model by $6\%$, and that some models can perform
worse than random guessing when up to $25\%$ of labels are flipped. (3) We shed
light on an interesting interplay between what the attacker gains from more
write-access versus what they gain from more flipping budget. (4) we define and
compare the power of targeted label flipping attack to that of an untargeted
label flipping attack.


http://arxiv.org/abs/2502.20924
Decoder Gradient Shield: Provable and High-Fidelity Prevention of Gradient-Based Box-Free Watermark Removal. (83%)
Haonan An; Guang Hua; Zhengru Fang; Guowen Xu; Susanto Rahardja; Yuguang Fang
  The intellectual property of deep image-to-image models can be protected by
the so-called box-free watermarking. It uses an encoder and a decoder,
respectively, to embed into and extract from the model's output images
invisible copyright marks. Prior works have improved watermark robustness,
focusing on the design of better watermark encoders. In this paper, we reveal
an overlooked vulnerability of the unprotected watermark decoder which is
jointly trained with the encoder and can be exploited to train a watermark
removal network. To defend against such an attack, we propose the decoder
gradient shield (DGS) as a protection layer in the decoder API to prevent
gradient-based watermark removal with a closed-form solution. The fundamental
idea is inspired by the classical adversarial attack, but is utilized for the
first time as a defensive mechanism in the box-free model watermarking. We then
demonstrate that DGS can reorient and rescale the gradient directions of
watermarked queries and stop the watermark remover's training loss from
converging to the level without DGS, while retaining decoder output image
quality. Experimental results verify the effectiveness of proposed method. Code
of paper will be made available upon acceptance.


http://arxiv.org/abs/2502.21041
Fast Adversarial Training against Sparse Attacks Requires Loss Smoothing. (69%)
Xuyang Zhong; Yixiao Huang; Chen Liu
  This paper studies fast adversarial training against sparse adversarial
perturbations bounded by $l_0$ norm. We demonstrate the challenges of employing
$1$-step attacks on $l_0$ bounded perturbations for fast adversarial training,
including degraded performance and the occurrence of catastrophic overfitting
(CO). We highlight that CO in $l_0$ adversarial training is caused by
sub-optimal perturbation locations of $1$-step attack. Theoretical and
empirical analyses reveal that the loss landscape of $l_0$ adversarial training
is more craggy compared to its $l_\infty$, $l_2$ and $l_1$ counterparts.
Moreover, we corroborate that the craggy loss landscape can aggravate CO. To
address these issues, we propose Fast-LS-$l_0$ that incorporates soft labels
and the trade-off loss function to smooth the adversarial loss landscape.
Extensive experiments demonstrate our method can overcome the challenge of
catastrophic overfitting, achieve state-of-the-art performance, and narrow down
the performance gap between $1$-step and multi-step adversarial training
against sparse attacks.


http://arxiv.org/abs/2502.20995
The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systems. (15%)
Chanwoo Choi; Jinsoo Kim; Sukmin Cho; Soyeong Jeong; Buru Chang
  With the growing adoption of retrieval-augmented generation (RAG) systems,
various attack methods have been proposed to degrade their performance.
However, most existing approaches rely on unrealistic assumptions in which
external attackers have access to internal components such as the retriever. To
address this issue, we introduce a realistic black-box attack based on the RAG
paradox, a structural vulnerability arising from the system's effort to enhance
trust by revealing both the retrieved documents and their sources to users.
This transparency enables attackers to observe which sources are used and how
information is phrased, allowing them to craft poisoned documents that are more
likely to be retrieved and upload them to the identified sources. Moreover, as
RAG systems directly provide retrieved content to users, these documents must
not only be retrievable but also appear natural and credible to maintain user
confidence in the search results. Unlike prior work that focuses solely on
improving document retrievability, our attack method explicitly considers both
retrievability and user trust in the retrieved content. Both offline and online
experiments demonstrate that our method significantly degrades system
performance without internal access, while generating natural-looking poisoned
documents.


http://arxiv.org/abs/2502.21286
Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis. (2%)
Li Yang; Mirna El Rajab; Abdallah Shami; Sami Muhaidat
  Zero-Touch Networks (ZTNs) represent a state-of-the-art paradigm shift
towards fully automated and intelligent network management, enabling the
automation and intelligence required to manage the complexity, scale, and
dynamic nature of next-generation (6G) networks. ZTNs leverage Artificial
Intelligence (AI) and Machine Learning (ML) to enhance operational efficiency,
support intelligent decision-making, and ensure effective resource allocation.
However, the implementation of ZTNs is subject to security challenges that need
to be resolved to achieve their full potential. In particular, two critical
challenges arise: the need for human expertise in developing AI/ML-based
security mechanisms, and the threat of adversarial attacks targeting AI/ML
models. In this survey paper, we provide a comprehensive review of current
security issues in ZTNs, emphasizing the need for advanced AI/ML-based security
mechanisms that require minimal human intervention and protect AI/ML models
themselves. Furthermore, we explore the potential of Automated ML (AutoML)
technologies in developing robust security solutions for ZTNs. Through case
studies, we illustrate practical approaches to securing ZTNs against both
conventional and AI/ML-specific threats, including the development of
autonomous intrusion detection systems and strategies to combat Adversarial ML
(AML) attacks. The paper concludes with a discussion of the future research
directions for the development of ZTN security approaches.


http://arxiv.org/abs/2502.21279
L-Lipschitz Gershgorin ResNet Network. (1%)
Marius F. R. Juston; William R. Norris; Dustin Nottage; Ahmet Soylemezoglu
  Deep residual networks (ResNets) have demonstrated outstanding success in
computer vision tasks, attributed to their ability to maintain gradient flow
through deep architectures. Simultaneously, controlling the Lipschitz bound in
neural networks has emerged as an essential area of research for enhancing
adversarial robustness and network certifiability. This paper uses a rigorous
approach to design $\mathcal{L}$-Lipschitz deep residual networks using a
Linear Matrix Inequality (LMI) framework. The ResNet architecture was
reformulated as a pseudo-tri-diagonal LMI with off-diagonal elements and
derived closed-form constraints on network parameters to ensure
$\mathcal{L}$-Lipschitz continuity. To address the lack of explicit eigenvalue
computations for such matrix structures, the Gershgorin circle theorem was
employed to approximate eigenvalue locations, guaranteeing the LMI's negative
semi-definiteness. Our contributions include a provable parameterization
methodology for constructing Lipschitz-constrained networks and a compositional
framework for managing recursive systems within hierarchical architectures.
These findings enable robust network designs applicable to adversarial
robustness, certified training, and control systems. However, a limitation was
identified in the Gershgorin-based approximations, which over-constrain the
system, suppressing non-linear dynamics and diminishing the network's
expressive capacity.


http://arxiv.org/abs/2502.20562
LISArD: Learning Image Similarity to Defend Against Gray-box Adversarial Attacks. (99%)
Joana C. Costa; Tiago Roxo; Hugo Proença; Pedro R. M. Inácio
  State-of-the-art defense mechanisms are typically evaluated in the context of
white-box attacks, which is not realistic, as it assumes the attacker can
access the gradients of the target network. To protect against this scenario,
Adversarial Training (AT) and Adversarial Distillation (AD) include adversarial
examples during the training phase, and Adversarial Purification uses a
generative model to reconstruct all the images given to the classifier. This
paper considers an even more realistic evaluation scenario: gray-box attacks,
which assume that the attacker knows the architecture and the dataset used to
train the target network, but cannot access its gradients. We provide empirical
evidence that models are vulnerable to gray-box attacks and propose LISArD, a
defense mechanism that does not increase computational and temporal costs but
provides robustness against gray-box and white-box attacks without including
AT. Our method approximates a cross-correlation matrix, created with the
embeddings of perturbed and clean images, to a diagonal matrix while
simultaneously conducting classification learning. Our results show that LISArD
can effectively protect against gray-box attacks, can be used in multiple
architectures, and carries over its resilience to the white-box scenario. Also,
state-of-the-art AD models underperform greatly when removing AT and/or moving
to gray-box settings, highlighting the lack of robustness from existing
approaches to perform in various conditions (aside from white-box settings).
All the source code is available at https://github.com/Joana-Cabral/LISArD.


http://arxiv.org/abs/2503.00063
NoPain: No-box Point Cloud Attack via Optimal Transport Singular Boundary. (99%)
Zezeng Li; Xiaoyu Du; Na Lei; Liming Chen; Weimin Wang
  Adversarial attacks exploit the vulnerability of deep models against
adversarial samples. Existing point cloud attackers are tailored to specific
models, iteratively optimizing perturbations based on gradients in either a
white-box or black-box setting. Despite their promising attack performance,
they often struggle to produce transferable adversarial samples due to
overfitting the specific parameters of surrogate models. To overcome this
issue, we shift our focus to the data distribution itself and introduce a novel
approach named NoPain, which employs optimal transport (OT) to identify the
inherent singular boundaries of the data manifold for cross-network point cloud
attacks. Specifically, we first calculate the OT mapping from noise to the
target feature space, then identify singular boundaries by locating
non-differentiable positions. Finally, we sample along singular boundaries to
generate adversarial point clouds. Once the singular boundaries are determined,
NoPain can efficiently produce adversarial samples without the need of
iterative updates or guidance from the surrogate classifiers. Extensive
experiments demonstrate that the proposed end-to-end method outperforms
baseline approaches in terms of both transferability and efficiency, while also
maintaining notable advantages even against defense strategies. Code and model
are available at https://github.com/cognaclee/nopain


http://arxiv.org/abs/2502.20650
Gungnir: Exploiting Stylistic Features in Images for Backdoor Attacks on Diffusion Models. (86%)
Yu Pan; Jiahao Chen; Bingrong Dai; Lin Wang; Yi Du; Jiao Liu
  In recent years, Diffusion Models (DMs) have demonstrated significant
advances in the field of image generation. However, according to current
research, DMs are vulnerable to backdoor attacks, which allow attackers to
control the model's output by inputting data containing covert triggers, such
as a specific visual patch or phrase. Existing defense strategies are well
equipped to thwart such attacks through backdoor detection and trigger
inversion because previous attack methods are constrained by limited input
spaces and low-dimensional triggers. For example, visual triggers are easily
observed by defenders, text-based or attention-based triggers are more
susceptible to neural network detection. To explore more possibilities of
backdoor attack in DMs, we propose Gungnir, a novel method that enables
attackers to activate the backdoor in DMs through style triggers within input
images. Our approach proposes using stylistic features as triggers for the
first time and implements backdoor attacks successfully in image-to-image tasks
by introducing Reconstructing-Adversarial Noise (RAN) and Short-Term
Timesteps-Retention (STTR). Our technique generates trigger-embedded images
that are perceptually indistinguishable from clean images, thus bypassing both
manual inspection and automated detection neural networks. Experiments
demonstrate that Gungnir can easily bypass existing defense methods. Among
existing DM defense frameworks, our approach achieves a 0 backdoor detection
rate (BDR). Our codes are available at https://github.com/paoche11/Gungnir.


http://arxiv.org/abs/2502.20604
Exploring the Impact of Temperature Scaling in Softmax for Classification and Adversarial Robustness. (74%)
Hao Xuan; Bokai Yang; Xingyu Li
  The softmax function is a fundamental component in deep learning. This study
delves into the often-overlooked parameter within the softmax function, known
as "temperature," providing novel insights into the practical and theoretical
aspects of temperature scaling for image classification. Our empirical studies,
adopting convolutional neural networks and transformers on multiple benchmark
datasets, reveal that moderate temperatures generally introduce better overall
performance. Through extensive experiments and rigorous theoretical analysis,
we explore the role of temperature scaling in model training and unveil that
temperature not only influences learning step size but also shapes the model's
optimization direction. Moreover, for the first time, we discover a surprising
benefit of elevated temperatures: enhanced model robustness against common
corruption, natural perturbation, and non-targeted adversarial attacks like
Projected Gradient Descent. We extend our discoveries to adversarial training,
demonstrating that, compared to the standard softmax function with the default
temperature value, higher temperatures have the potential to enhance
adversarial training. The insights of this work open new avenues for improving
model performance and security in deep learning applications.


http://arxiv.org/abs/2502.20314
Adversarial Robustness in Parameter-Space Classifiers. (69%)
Tamir Shor; Ethan Fetaya; Chaim Baskin; Alex Bronstein
  Implicit Neural Representations (INRs) have been recently garnering
increasing interest in various research fields, mainly due to their ability to
represent large, complex data in a compact and continuous manner. Past work
further showed that numerous popular downstream tasks can be performed directly
in the INR parameter-space. Doing so can substantially reduce the computational
resources required to process the represented data in their native domain. A
major difficulty in using modern machine-learning approaches, is their high
susceptibility to adversarial attacks, which have been shown to greatly limit
the reliability and applicability of such methods in a wide range of settings.
In this work, we show that parameter-space models trained for classification
are inherently robust to adversarial attacks -- without the need of any robust
training. To support our claims, we develop a novel suite of adversarial
attacks targeting parameter-space classifiers, and furthermore analyze
practical considerations of attacking parameter-space classifiers. Code for
reproducing all experiments and implementation of all proposed methods will be
released upon publication.


http://arxiv.org/abs/2502.20325
On Adversarial Attacks In Acoustic Drone Localization. (67%)
Tamir Shor; Chaim Baskin; Alex Bronstein
  Multi-rotor aerial autonomous vehicles (MAVs, more widely known as "drones")
have been generating increased interest in recent years due to their growing
applicability in a vast and diverse range of fields (e.g., agriculture,
commercial delivery, search and rescue). The sensitivity of visual-based
methods to lighting conditions and occlusions had prompted growing study of
navigation reliant on other modalities, such as acoustic sensing. A major
concern in using drones in scale for tasks in non-controlled environments is
the potential threat of adversarial attacks over their navigational systems,
exposing users to mission-critical failures, security breaches, and compromised
safety outcomes that can endanger operators and bystanders. While previous work
shows impressive progress in acoustic-based drone localization, prior research
in adversarial attacks over drone navigation only addresses visual
sensing-based systems. In this work, we aim to compensate for this gap by
supplying a comprehensive analysis of the effect of PGD adversarial attacks
over acoustic drone localization. We furthermore develop an algorithm for
adversarial perturbation recovery, capable of markedly diminishing the affect
of such attacks in our setting. The code for reproducing all experiments will
be released upon publication.


http://arxiv.org/abs/2503.00065
ADAGE: Active Defenses Against GNN Extraction. (33%)
Jing Xu; Franziska Boenisch; Adam Dziedzic
  Graph Neural Networks (GNNs) achieve high performance in various real-world
applications, such as drug discovery, traffic states prediction, and
recommendation systems. The fact that building powerful GNNs requires a large
amount of training data, powerful computing resources, and human expertise
turns the models into lucrative targets for model stealing attacks. Prior work
has revealed that the threat vector of stealing attacks against GNNs is large
and diverse, as an attacker can leverage various heterogeneous signals ranging
from node labels to high-dimensional node embeddings to create a local copy of
the target GNN at a fraction of the original training costs. This diversity in
the threat vector renders the design of effective and general defenses
challenging and existing defenses usually focus on one particular stealing
setup. Additionally, they solely provide means to identify stolen model copies
rather than preventing the attack. To close this gap, we propose the first and
general Active Defense Against GNN Extraction (ADAGE). By analyzing the queries
to the GNN, tracking their diversity in terms of proximity to different
communities identified in the underlying graph, and increasing the defense
strength with the growing fraction of communities that have been queried, ADAGE
can prevent stealing in all common attack setups. Our extensive experimental
evaluation using six benchmark datasets, four GNN models, and three types of
adaptive attackers shows that ADAGE penalizes attackers to the degree of
rendering stealing impossible, whilst not harming predictive performance for
legitimate users. ADAGE, thereby, contributes towards securely sharing valuable
GNNs in the future.


http://arxiv.org/abs/2503.00062
CRFU: Compressive Representation Forgetting Against Privacy Leakage on Machine Unlearning. (31%)
Weiqi Wang; Chenhan Zhang; Zhiyi Tian; Shushu Liu; Shui Yu
  Machine unlearning allows data owners to erase the impact of their specified
data from trained models. Unfortunately, recent studies have shown that
adversaries can recover the erased data, posing serious threats to user
privacy. An effective unlearning method removes the information of the
specified data from the trained model, resulting in different outputs for the
same input before and after unlearning. Adversaries can exploit these output
differences to conduct privacy leakage attacks, such as reconstruction and
membership inference attacks. However, directly applying traditional defenses
to unlearning leads to significant model utility degradation. In this paper, we
introduce a Compressive Representation Forgetting Unlearning scheme (CRFU),
designed to safeguard against privacy leakage on unlearning. CRFU achieves data
erasure by minimizing the mutual information between the trained compressive
representation (learned through information bottleneck theory) and the erased
data, thereby maximizing the distortion of data. This ensures that the model's
output contains less information that adversaries can exploit. Furthermore, we
introduce a remembering constraint and an unlearning rate to balance the
forgetting of erased data with the preservation of previously learned
knowledge, thereby reducing accuracy degradation. Theoretical analysis
demonstrates that CRFU can effectively defend against privacy leakage attacks.
Our experimental results show that CRFU significantly increases the
reconstruction mean square error (MSE), achieving a defense effect improvement
of approximately $200\%$ against privacy reconstruction attacks with only
$1.5\%$ accuracy degradation on MNIST.


http://arxiv.org/abs/2502.20306
SecureGaze: Defending Gaze Estimation Against Backdoor Attacks. (13%)
Lingyu Du; Yupei Liu; Jinyuan Jia; Guohao Lan
  Gaze estimation models are widely used in applications such as driver
attention monitoring and human-computer interaction. While many methods for
gaze estimation exist, they rely heavily on data-hungry deep learning to
achieve high performance. This reliance often forces practitioners to harvest
training data from unverified public datasets, outsource model training, or
rely on pre-trained models. However, such practices expose gaze estimation
models to backdoor attacks. In such attacks, adversaries inject backdoor
triggers by poisoning the training data, creating a backdoor vulnerability: the
model performs normally with benign inputs, but produces manipulated gaze
directions when a specific trigger is present. This compromises the security of
many gaze-based applications, such as causing the model to fail in tracking the
driver's attention. To date, there is no defense that addresses backdoor
attacks on gaze estimation models. In response, we introduce SecureGaze, the
first solution designed to protect gaze estimation models from such attacks.
Unlike classification models, defending gaze estimation poses unique challenges
due to its continuous output space and globally activated backdoor behavior. By
identifying distinctive characteristics of backdoored gaze estimation models,
we develop a novel and effective approach to reverse-engineer the trigger
function for reliable backdoor detection. Extensive evaluations in both digital
and physical worlds demonstrate that SecureGaze effectively counters a range of
backdoor attacks and outperforms seven state-of-the-art defenses adapted from
classification models.


http://arxiv.org/abs/2502.19806
From Data to Sliding Mode Control of Uncertain Large-Scale Networks with Unknown Dynamics. (1%)
Behrad Samari; Gian Paolo Incremona; Antonella Ferrara; Abolfazl Lavaei
  Large-scale interconnected networks, composed of multiple low-dimensional
subsystems, serve as a crucial framework for modeling a wide range of
real-world applications. Despite offering computational scalability, the
inherent interdependence among subsystems poses significant challenges to the
effective control of such networks. This complexity is further exacerbated in
the presence of external perturbations and when the dynamics of individual
subsystems, and accordingly the overall network, are unknown-scenarios
frequently encountered in modern practical applications. In this paper, we
develop a compositional data-driven approach to ensure the global asymptotic
stability (GAS) of large-scale nonlinear networks with unknown mathematical
models, subjected to external perturbations. To achieve this, we first gather
two sets of data from each unknown nominal subsystem without perturbation,
which we refer to as two input-state trajectories. The collected data from each
subsystem is then utilized to design an input-to-state stable (ISS) Lyapunov
function and its corresponding controller for each nominal subsystem, rendering
them ISS. To cancel the effect of external perturbations on the dynamic of each
subsystem, and accordingly the whole network, we then design a local integral
sliding mode (ISM) controller for each subsystem using the collected data.
Under a small-gain compositional condition, we employ data-driven ISS Lyapunov
functions designed for subsystems and construct a control Lyapunov function for
the network, rendering the assurance of GAS property over the nominal network.
We then extend this compositional result to network perturbed models,
demonstrating that the synthesized ISM controllers ensure the GAS property even
in the presence of perturbations.


http://arxiv.org/abs/2502.20589
LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times and Network Traffic Analysis. (1%)
Saeif Alhazbi; Ahmed Mohamed Hussain; Gabriele Oligeri; Panos Papadimitratos
As Large Language Models (LLMs) become increasingly integrated into many technological ecosystems across various domains and industries, identifying which model is deployed or being interacted with is critical for the security and trustworthiness of the systems. Current verification methods typically rely on analyzing the generated output to determine the source model. However, these techniques are susceptible to adversarial attacks, operate in a post-hoc manner, and may require access to the model weights to inject a verifiable fingerprint. In this paper, we propose a novel passive and non-invasive fingerprinting technique that operates in real-time and remains effective even under encrypted network traffic conditions. Our method leverages the intrinsic autoregressive generation nature of language models, which generate text one token at a time based on all previously generated tokens, creating a unique temporal pattern like a rhythm or heartbeat that persists even when the output is streamed over a network. We find that measuring the Inter-Token Times (ITTs)-time intervals between consecutive tokens-can identify different language models with high accuracy. We develop a Deep Learning (DL) pipeline to capture these timing patterns using network traffic analysis and evaluate it on 16 Small Language Models (SLMs) and 10 proprietary LLMs across different deployment scenarios, including local host machine (GPU/CPU), Local Area Network (LAN), Remote Network, and Virtual Private Network (VPN). The experimental results confirm that our proposed technique is effective and maintains high accuracy even when tested in different network conditions. This work opens a new avenue for model identification in real-world scenarios and contributes to more secure and trustworthy language model deployment.

http://arxiv.org/abs/2502.20178
SSD: A State-based Stealthy Backdoor Attack For Navigation System in UAV Route Planning. (1%)
Zhaoxuan Wang; Yang Li; Jie Zhang; Xingshuo Han; Kangbo Liu; Lyu Yang; yuan Zhou; Tianwei Zhang; Quan Pan
  Unmanned aerial vehicles (UAVs) are increasingly employed to perform
high-risk tasks that require minimal human intervention. However, UAVs face
escalating cybersecurity threats, particularly from GNSS spoofing attacks.
While previous studies have extensively investigated the impacts of GNSS
spoofing on UAVs, few have focused on its effects on specific tasks. Moreover,
the influence of UAV motion states on the assessment of network security risks
is often overlooked. To address these gaps, we first provide a detailed
evaluation of how motion states affect the effectiveness of network attacks. We
demonstrate that nonlinear motion states not only enhance the effectiveness of
position spoofing in GNSS spoofing attacks but also reduce the probability of
speed-related attack detection. Building upon this, we propose a
state-triggered backdoor attack method (SSD) to deceive GNSS systems and assess
its risk to trajectory planning tasks. Extensive validation of SSD's
effectiveness and stealthiness is conducted. Experimental results show that,
with appropriately tuned hyperparameters, SSD significantly increases
positioning errors and the risk of task failure, while maintaining 100% stealth
across three state-of-the-art detectors.


http://arxiv.org/abs/2502.19672
Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attack. (99%)
Chenhe Gu; Jindong Gu; Andong Hua; Yao Qin
  Multimodal Large Language Models (MLLMs), built upon LLMs, have recently
gained attention for their capabilities in image recognition and understanding.
However, while MLLMs are vulnerable to adversarial attacks, the transferability
of these attacks across different models remains limited, especially under
targeted attack setting. Existing methods primarily focus on vision-specific
perturbations but struggle with the complex nature of vision-language modality
alignment. In this work, we introduce the Dynamic Vision-Language Alignment
(DynVLA) Attack, a novel approach that injects dynamic perturbations into the
vision-language connector to enhance generalization across diverse
vision-language alignment of different models. Our experimental results show
that DynVLA significantly improves the transferability of adversarial examples
across various MLLMs, including BLIP2, InstructBLIP, MiniGPT4, LLaVA, and
closed-source models such as Gemini.


http://arxiv.org/abs/2502.19757
Snowball Adversarial Attack on Traffic Sign Classification. (99%)
Anthony Etim; Jakub Szefer
  Adversarial attacks on machine learning models often rely on small,
imperceptible perturbations to mislead classifiers. Such strategy focuses on
minimizing the visual perturbation for humans so they are not confused, and
also maximizing the misclassification for machine learning algorithms. An
orthogonal strategy for adversarial attacks is to create perturbations that are
clearly visible but do not confuse humans, yet still maximize misclassification
for machine learning algorithms. This work follows the later strategy, and
demonstrates instance of it through the Snowball Adversarial Attack in the
context of traffic sign recognition. The attack leverages the human brain's
superior ability to recognize objects despite various occlusions, while machine
learning algorithms are easily confused. The evaluation shows that the Snowball
Adversarial Attack is robust across various images and is able to confuse
state-of-the-art traffic sign recognition algorithm. The findings reveal that
Snowball Adversarial Attack can significantly degrade model performance with
minimal effort, raising important concerns about the vulnerabilities of deep
neural networks and highlighting the necessity for improved defenses for image
recognition machine learning models.


http://arxiv.org/abs/2502.19697
Prompt-driven Transferable Adversarial Attack on Person Re-Identification with Attribute-aware Textual Inversion. (99%)
Yuan Bian; Min Liu; Yunqi Yi; Xueping Wang; Yaonan Wang
  Person re-identification (re-id) models are vital in security surveillance
systems, requiring transferable adversarial attacks to explore the
vulnerabilities of them. Recently, vision-language models (VLM) based attacks
have shown superior transferability by attacking generalized image and textual
features of VLM, but they lack comprehensive feature disruption due to the
overemphasis on discriminative semantics in integral representation. In this
paper, we introduce the Attribute-aware Prompt Attack (AP-Attack), a novel
method that leverages VLM's image-text alignment capability to explicitly
disrupt fine-grained semantic features of pedestrian images by destroying
attribute-specific textual embeddings. To obtain personalized textual
descriptions for individual attributes, textual inversion networks are designed
to map pedestrian images to pseudo tokens that represent semantic embeddings,
trained in the contrastive learning manner with images and a predefined prompt
template that explicitly describes the pedestrian attributes. Inverted benign
and adversarial fine-grained textual semantics facilitate attacker in
effectively conducting thorough disruptions, enhancing the transferability of
adversarial examples. Extensive experiments show that AP-Attack achieves
state-of-the-art transferability, significantly outperforming previous methods
by 22.9% on mean Drop Rate in cross-model&dataset attack scenarios.


http://arxiv.org/abs/2502.19710
SAP-DIFF: Semantic Adversarial Patch Generation for Black-Box Face Recognition Models via Diffusion Models. (98%)
Mingsi Wang; Shuaiyin Yao; Chang Yue; Lijie Zhang; Guozhu Meng
  Given the need to evaluate the robustness of face recognition (FR) models,
many efforts have focused on adversarial patch attacks that mislead FR models
by introducing localized perturbations. Impersonation attacks are a significant
threat because adversarial perturbations allow attackers to disguise themselves
as legitimate users. This can lead to severe consequences, including data
breaches, system damage, and misuse of resources. However, research on such
attacks in FR remains limited. Existing adversarial patch generation methods
exhibit limited efficacy in impersonation attacks due to (1) the need for high
attacker capabilities, (2) low attack success rates, and (3) excessive query
requirements. To address these challenges, we propose a novel method SAP-DIFF
that leverages diffusion models to generate adversarial patches via semantic
perturbations in the latent space rather than direct pixel manipulation. We
introduce an attention disruption mechanism to generate features unrelated to
the original face, facilitating the creation of adversarial samples and a
directional loss function to guide perturbations toward the target identity
feature space, thereby enhancing attack effectiveness and efficiency. Extensive
experiments on popular FR models and datasets demonstrate that our method
outperforms state-of-the-art approaches, achieving an average attack success
rate improvement of 45.66% (all exceeding 40%), and a reduction in the number
of queries by about 40% compared to the SOTA approach


http://arxiv.org/abs/2502.19095
XSS Adversarial Attacks Based on Deep Reinforcement Learning: A Replication and Extension Study. (87%)
Samuele Pasini; Gianluca Maragliano; Jinhan Kim; Paolo Tonella
  Cross-site scripting (XSS) poses a significant threat to web application
security. While Deep Learning (DL) has shown remarkable success in detecting
XSS attacks, it remains vulnerable to adversarial attacks due to the
discontinuous nature of its input-output mapping. These adversarial attacks
employ mutation-based strategies for different components of XSS attack
vectors, allowing adversarial agents to iteratively select mutations to evade
detection. Our work replicates a state-of-the-art XSS adversarial attack,
highlighting threats to validity in the reference work and extending it toward
a more effective evaluation strategy. Moreover, we introduce an XSS Oracle to
mitigate these threats. The experimental results show that our approach
achieves an escape rate above 96% when the threats to validity of the
replicated technique are addressed.


http://arxiv.org/abs/2502.19755
HALO: Robust Out-of-Distribution Detection via Joint Optimisation. (86%)
Hugo Lyons Keenan; Sarah Erfani; Christopher Leckie
  Effective out-of-distribution (OOD) detection is crucial for the safe
deployment of machine learning models in real-world scenarios. However, recent
work has shown that OOD detection methods are vulnerable to adversarial
attacks, potentially leading to critical failures in high-stakes applications.
This discovery has motivated work on robust OOD detection methods that are
capable of maintaining performance under various attack settings. Prior
approaches have made progress on this problem but face a number of limitations:
often only exhibiting robustness to attacks on OOD data or failing to maintain
strong clean performance. In this work, we adapt an existing robust
classification framework, TRADES, extending it to the problem of robust OOD
detection and discovering a novel objective function. Recognising the critical
importance of a strong clean/robust trade-off for OOD detection, we introduce
an additional loss term which boosts classification and detection performance.
Our approach, called HALO (Helper-based AdversariaL OOD detection), surpasses
existing methods and achieves state-of-the-art performance across a number of
datasets and attack settings. Extensive experiments demonstrate an average
AUROC improvement of 3.15 in clean settings and 7.07 under adversarial attacks
when compared to the next best method. Furthermore, HALO exhibits resistance to
transferred attacks, offers tuneable performance through hyperparameter
selection, and is compatible with existing OOD detection frameworks
out-of-the-box, leaving open the possibility of future performance gains. Code
is available at: https://github.com/hugo0076/HALO


http://arxiv.org/abs/2502.19047
A Dual-Purpose Framework for Backdoor Defense and Backdoor Amplification in Diffusion Models. (83%)
Vu Tuan Truong Long; Bao Le
  Diffusion models have emerged as state-of-the-art generative frameworks,
excelling in producing high-quality multi-modal samples. However, recent
studies have revealed their vulnerability to backdoor attacks, where backdoored
models generate specific, undesirable outputs called backdoor target (e.g.,
harmful images) when a pre-defined trigger is embedded to their inputs. In this
paper, we propose PureDiffusion, a dual-purpose framework that simultaneously
serves two contrasting roles: backdoor defense and backdoor attack
amplification. For defense, we introduce two novel loss functions to invert
backdoor triggers embedded in diffusion models. The first leverages
trigger-induced distribution shifts across multiple timesteps of the diffusion
process, while the second exploits the denoising consistency effect when a
backdoor is activated. Once an accurate trigger inversion is achieved, we
develop a backdoor detection method that analyzes both the inverted trigger and
the generated backdoor targets to identify backdoor attacks. In terms of attack
amplification with the role of an attacker, we describe how our trigger
inversion algorithm can be used to reinforce the original trigger embedded in
the backdoored diffusion model. This significantly boosts attack performance
while reducing the required backdoor training time. Experimental results
demonstrate that PureDiffusion achieves near-perfect detection accuracy,
outperforming existing defenses by a large margin, particularly against complex
trigger patterns. Additionally, in an attack scenario, our attack amplification
approach elevates the attack success rate (ASR) of existing backdoor attacks to
nearly 100\% while reducing training time by up to 20x.


http://arxiv.org/abs/2502.18943
Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models. (41%)
Yu He; Boheng Li; Liu Liu; Zhongjie Ba; Wei Dong; Yiming Li; Zhan Qin; Kui Ren; Chun Chen
  Membership Inference Attacks (MIAs) aim to predict whether a data sample
belongs to the model's training set or not. Although prior research has
extensively explored MIAs in Large Language Models (LLMs), they typically
require accessing to complete output logits (\ie, \textit{logits-based
attacks}), which are usually not available in practice. In this paper, we study
the vulnerability of pre-trained LLMs to MIAs in the \textit{label-only
setting}, where the adversary can only access generated tokens (text). We first
reveal that existing label-only MIAs have minor effects in attacking
pre-trained LLMs, although they are highly effective in inferring fine-tuning
datasets used for personalized LLMs. We find that their failure stems from two
main reasons, including better generalization and overly coarse perturbation.
Specifically, due to the extensive pre-training corpora and exposing each
sample only a few times, LLMs exhibit minimal robustness differences between
members and non-members. This makes token-level perturbations too coarse to
capture such differences.
  To alleviate these problems, we propose \textbf{PETAL}: a label-only
membership inference attack based on \textbf{PE}r-\textbf{T}oken
sem\textbf{A}ntic simi\textbf{L}arity. Specifically, PETAL leverages
token-level semantic similarity to approximate output probabilities and
subsequently calculate the perplexity. It finally exposes membership based on
the common assumption that members are `better' memorized and have smaller
perplexity. We conduct extensive experiments on the WikiMIA benchmark and the
more challenging MIMIR benchmark. Empirically, our PETAL performs better than
the extensions of existing label-only attacks against personalized LLMs and
even on par with other advanced logit-based attacks across all metrics on five
prevalent open-source LLMs.


http://arxiv.org/abs/2503.00061
Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents. (26%)
Qiusi Zhan; Richard Fang; Henil Shalin Panchal; Daniel Kang
  Large Language Model (LLM) agents exhibit remarkable performance across
diverse applications by using external tools to interact with environments.
However, integrating external tools introduces security risks, such as indirect
prompt injection (IPI) attacks. Despite defenses designed for IPI attacks,
their robustness remains questionable due to insufficient testing against
adaptive attacks. In this paper, we evaluate eight different defenses and
bypass all of them using adaptive attacks, consistently achieving an attack
success rate of over 50%. This reveals critical vulnerabilities in current
defenses. Our research underscores the need for adaptive attack evaluation when
designing defenses to ensure robustness and reliability. The code is available
at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent.


http://arxiv.org/abs/2502.19612
Evaluation of Hate Speech Detection Using Large Language Models and Geographical Contextualization. (26%)
Anwar Hossain Zahid; Monoshi Kumar Roy; Swarna Das
  The proliferation of hate speech on social media is one of the serious issues
that is bringing huge impacts to society: an escalation of violence,
discrimination, and social fragmentation. The problem of detecting hate speech
is intrinsically multifaceted due to cultural, linguistic, and contextual
complexities and adversarial manipulations. In this study, we systematically
investigate the performance of LLMs on detecting hate speech across
multilingual datasets and diverse geographic contexts. Our work presents a new
evaluation framework in three dimensions: binary classification of hate speech,
geography-aware contextual detection, and robustness to adversarially generated
text. Using a dataset of 1,000 comments from five diverse regions, we evaluate
three state-of-the-art LLMs: Llama2 (13b), Codellama (7b), and DeepSeekCoder
(6.7b). Codellama had the best binary classification recall with 70.6% and an
F1-score of 52.18%, whereas DeepSeekCoder had the best performance in
geographic sensitivity, correctly detecting 63 out of 265 locations. The tests
for adversarial robustness also showed significant weaknesses; Llama2
misclassified 62.5% of manipulated samples. These results bring to light the
trade-offs between accuracy, contextual understanding, and robustness in the
current versions of LLMs. This work has thus set the stage for developing
contextually aware, multilingual hate speech detection systems by underlining
key strengths and limitations, therefore offering actionable insights for
future research and real-world applications.


http://arxiv.org/abs/2502.19041
Beyond Surface-Level Patterns: An Essence-Driven Defense Framework Against Jailbreak Attacks in LLMs. (3%)
Shiyu Xiang; Ansen Zhang; Yanfei Cao; Yang Fan; Ronghao Chen
  Although Aligned Large Language Models (LLMs) are trained to refuse harmful
requests, they remain vulnerable to jailbreak attacks. Unfortunately, existing
methods often focus on surface-level patterns, overlooking the deeper attack
essences. As a result, defenses fail when attack prompts change, even though
the underlying "attack essence" remains the same. To address this issue, we
introduce EDDF, an \textbf{E}ssence-\textbf{D}riven \textbf{D}efense
\textbf{F}ramework Against Jailbreak Attacks in LLMs. EDDF is a plug-and-play
input-filtering method and operates in two stages: 1) offline essence database
construction, and 2) online adversarial query detection. The key idea behind
EDDF is to extract the "attack essence" from a diverse set of known attack
instances and store it in an offline vector database. Experimental results
demonstrate that EDDF significantly outperforms existing methods by reducing
the Attack Success Rate by at least 20\%, underscoring its superior robustness
against jailbreak attacks.


http://arxiv.org/abs/2502.18724
Adversarial Universal Stickers: Universal Perturbation Attacks on Traffic Sign using Stickers. (99%)
Anthony Etim; Jakub Szefer
  Adversarial attacks on deep learning models have proliferated in recent
years. In many cases, a different adversarial perturbation is required to be
added to each image to cause the deep learning model to misclassify it. This is
ineffective as each image has to be modified in a different way. Meanwhile,
research on universal perturbations focuses on designing a single perturbation
that can be applied to all images in a data set, and cause a deep learning
model to misclassify the images. This work advances the field of universal
perturbations by exploring universal perturbations in the context of traffic
signs and autonomous vehicle systems. This work introduces a novel method for
generating universal perturbations that visually look like simple black and
white stickers, and using them to cause incorrect street sign predictions.
Unlike traditional adversarial perturbations, the adversarial universal
stickers are designed to be applicable to any street sign: same sticker, or
stickers, can be applied in same location to any street sign and cause it to be
misclassified. Further, to enable safe experimentation with adversarial images
and street signs, this work presents a virtual setting that leverages Street
View images of street signs, rather than the need to physically modify street
signs, to test the attacks. The experiments in the virtual setting demonstrate
that these stickers can consistently mislead deep learning models used commonly
in street sign recognition, and achieve high attack success rates on dataset of
US traffic signs. The findings highlight the practical security risks posed by
simple stickers applied to traffic signs, and the ease with which adversaries
can generate adversarial universal stickers that can be applied to many street
signs.


http://arxiv.org/abs/2502.17972
Model-Free Adversarial Purification via Coarse-To-Fine Tensor Network Representation. (99%)
Guang Lin; Duc Thien Nguyen; Zerui Tao; Konstantinos Slavakis; Toshihisa Tanaka; Qibin Zhao
  Deep neural networks are known to be vulnerable to well-designed adversarial
attacks. Although numerous defense strategies have been proposed, many are
tailored to the specific attacks or tasks and often fail to generalize across
diverse scenarios. In this paper, we propose Tensor Network Purification (TNP),
a novel model-free adversarial purification method by a specially designed
tensor network decomposition algorithm. TNP depends neither on the pre-trained
generative model nor the specific dataset, resulting in strong robustness
across diverse adversarial scenarios. To this end, the key challenge lies in
relaxing Gaussian-noise assumptions of classical decompositions and
accommodating the unknown distribution of adversarial perturbations. Unlike the
low-rank representation of classical decompositions, TNP aims to reconstruct
the unobserved clean examples from an adversarial example. Specifically, TNP
leverages progressive downsampling and introduces a novel adversarial
optimization objective to address the challenge of minimizing reconstruction
error but without inadvertently restoring adversarial perturbations. Extensive
experiments conducted on CIFAR-10, CIFAR-100, and ImageNet demonstrate that our
method generalizes effectively across various norm threats, attack types, and
tasks, providing a versatile and promising adversarial purification technique.


http://arxiv.org/abs/2502.18176
CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification. (92%)
Mingkun Zhang; Keping Bi; Wei Chen; Jiafeng Guo; Xueqi Cheng
  In this paper, we aim to build an adversarially robust zero-shot image
classifier. We ground our work on CLIP, a vision-language pre-trained encoder
model that can perform zero-shot classification by matching an image with text
prompts ``a photo of a <class-name>.''. Purification is the path we choose
since it does not require adversarial training on specific attack types and
thus can cope with any foreseen attacks. We then formulate purification risk as
the KL divergence between the joint distributions of the purification process
of denoising the adversarial samples and the attack process of adding
perturbations to benign samples, through bidirectional Stochastic Differential
Equations (SDEs). The final derived results inspire us to explore purification
in the multi-modal latent space of CLIP. We propose two variants for our
CLIPure approach: CLIPure-Diff which models the likelihood of images' latent
vectors with the DiffusionPrior module in DaLLE-2 (modeling the generation
process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood
with the cosine similarity between the embeddings of an image and ``a photo of
a.''. As far as we know, CLIPure is the first purification method in
multi-modal latent space and CLIPure-Cos is the first purification method that
is not based on generative models, which substantially improves defense
efficiency. We conducted extensive experiments on CIFAR-10, ImageNet, and 13
datasets that previous CLIP-based defense methods used for evaluating zero-shot
classification robustness. Results show that CLIPure boosts the SOTA robustness
by a large margin, e.g., from 71.7% to 91.1% on CIFAR10, from 59.6% to 72.6% on
ImageNet, and 108% relative improvements of average robustness on the 13
datasets over previous SOTA. The code is available at
https://github.com/TMLResearchGroup-CAS/CLIPure.


http://arxiv.org/abs/2503.01865
Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraints. (15%)
Junxiao Yang; Zhexin Zhang; Shiyao Cui; Hongning Wang; Minlie Huang
  Jailbreaking attacks can effectively induce unsafe behaviors in Large
Language Models (LLMs); however, the transferability of these attacks across
different models remains limited. This study aims to understand and enhance the
transferability of gradient-based jailbreaking methods, which are among the
standard approaches for attacking white-box models. Through a detailed analysis
of the optimization process, we introduce a novel conceptual framework to
elucidate transferability and identify superfluous constraints-specifically,
the response pattern constraint and the token tail constraint-as significant
barriers to improved transferability. Removing these unnecessary constraints
substantially enhances the transferability and controllability of
gradient-based attacks. Evaluated on Llama-3-8B-Instruct as the source model,
our method increases the overall Transfer Attack Success Rate (T-ASR) across a
set of target models with varying safety levels from 18.4% to 50.3%, while also
improving the stability and controllability of jailbreak behaviors on both
source and target models.


http://arxiv.org/abs/2503.00038
from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors. (4%)
Yu Yan; Sheng Sun; Zenghao Duan; Teli Liu; Min Liu; Zhiyi Yin; Jiangyu Lei; Qi Li
  Current studies have exposed the risk of Large Language Models (LLMs)
generating harmful content by jailbreak attacks. However, they overlook that
the direct generation of harmful content from scratch is more difficult than
inducing LLM to calibrate benign content into harmful forms. In our study, we
introduce a novel attack framework that exploits AdVersArial meTAphoR (AVATAR)
to induce the LLM to calibrate malicious metaphors for jailbreaking.
Specifically, to answer harmful queries, AVATAR adaptively identifies a set of
benign but logically related metaphors as the initial seed. Then, driven by
these metaphors, the target LLM is induced to reason and calibrate about the
metaphorical content, thus jailbroken by either directly outputting harmful
responses or calibrating residuals between metaphorical and professional
harmful content. Experimental results demonstrate that AVATAR can effectively
and transferable jailbreak LLMs and achieve a state-of-the-art attack success
rate across multiple advanced LLMs.


http://arxiv.org/abs/2502.18771
Exploring Graph Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation. (2%)
Yuxiang Wang; Xinnan Dai; Wenqi Fan; Yao Ma
  Graph-structured data has become increasingly prevalent across various
domains, raising the demand for effective models to handle graph tasks like
node classification and link prediction. Traditional graph learning models like
Graph Neural Networks (GNNs) have made significant strides, but their
capabilities in handling graph data remain limited in certain contexts. In
recent years, large language models (LLMs) have emerged as promising candidates
for graph tasks, yet most studies focus primarily on performance benchmarks and
fail to address their broader potential, including their ability to handle
limited data, their transferability across tasks, and their robustness. In this
work, we provide a comprehensive exploration of LLMs applied to graph tasks. We
evaluate the performance of pure LLMs, including those without parameter
optimization and those fine-tuned with instructions, across various scenarios.
Our analysis goes beyond accuracy, assessing LLM ability to perform in
few-shot/zero-shot settings, transfer across domains, understand graph
structures, and demonstrate robustness in challenging scenarios. We conduct
extensive experiments with 16 graph learning models alongside 6 LLMs (e.g.,
Llama3B, GPT-4o, Qwen-plus), comparing their performance on datasets like Cora,
PubMed, ArXiv, and Products. Our findings show that LLMs, particularly those
with instruction tuning, outperform traditional models in few-shot settings,
exhibit strong domain transferability, and demonstrate excellent generalization
and robustness. This work offers valuable insights into the capabilities of
LLMs for graph learning, highlighting their advantages and potential for
real-world applications, and paving the way for future research in this area.
Codes and datasets are released in
https://github.com/myflashbarry/LLM-benchmarking.


http://arxiv.org/abs/2502.18077
Examining the Threat Landscape: Foundation Models and Model Stealing. (2%)
Ankita Raj; Deepankar Varma; Chetan Arora
  Foundation models (FMs) for computer vision learn rich and robust
representations, enabling their adaptation to task/domain-specific deployments
with little to no fine-tuning. However, we posit that the very same strength
can make applications based on FMs vulnerable to model stealing attacks.
Through empirical analysis, we reveal that models fine-tuned from FMs harbor
heightened susceptibility to model stealing, compared to conventional vision
architectures like ResNets. We hypothesize that this behavior is due to the
comprehensive encoding of visual patterns and features learned by FMs during
pre-training, which are accessible to both the attacker and the victim. We
report that an attacker is able to obtain 94.28% agreement (matched predictions
with victim) for a Vision Transformer based victim model (ViT-L/16) trained on
CIFAR-10 dataset, compared to only 73.20% agreement for a ResNet-18 victim,
when using ViT-L/16 as the thief model. We arguably show, for the first time,
that utilizing FMs for downstream tasks may not be the best choice for
deployment in commercial APIs due to their susceptibility to model theft. We
thereby alert model owners towards the associated security risks, and highlight
the need for robust security measures to safeguard such models against theft.
Code is available at https://github.com/rajankita/foundation_model_stealing.


http://arxiv.org/abs/2502.18592
DeBUGCN -- Detecting Backdoors in CNNs Using Graph Convolutional Networks. (1%)
Akash Vartak; Khondoker Murad Hossain; Tim Oates
  Deep neural networks (DNNs) are becoming commonplace in critical
applications, making their susceptibility to backdoor (trojan) attacks a
significant problem. In this paper, we introduce a novel backdoor attack
detection pipeline, detecting attacked models using graph convolution networks
(DeBUGCN). To the best of our knowledge, ours is the first use of GCNs for
trojan detection. We use the static weights of a DNN to create a graph
structure of its layers. A GCN is then used as a binary classifier on these
graphs, yielding a trojan or clean determination for the DNN. To demonstrate
the efficacy of our pipeline, we train hundreds of clean and trojaned CNN
models on the MNIST handwritten digits and CIFAR-10 image datasets, and show
the DNN classification results using DeBUGCN. For a true In-the-Wild use case,
our pipeline is evaluated on the TrojAI dataset which consists of various CNN
architectures, thus showing the robustness and model-agnostic behaviour of
DeBUGCN. Furthermore, on comparing our results on several datasets with
state-of-the-art trojan detection algorithms, DeBUGCN is faster and more
accurate.


http://arxiv.org/abs/2502.18623
On the Privacy-Preserving Properties of Spiking Neural Networks with Unique Surrogate Gradients and Quantization Levels. (1%)
Ayana Moshruba; Shay Snyder; Hamed Poursiami; Maryam Parsa
  As machine learning models increasingly process sensitive data, understanding
their vulnerability to privacy attacks is vital. Membership inference attacks
(MIAs) exploit model responses to infer whether specific data points were used
during training, posing a significant privacy risk. Prior research suggests
that spiking neural networks (SNNs), which rely on event-driven computation and
discrete spike-based encoding, exhibit greater resilience to MIAs than
artificial neural networks (ANNs). This resilience stems from their
non-differentiable activations and inherent stochasticity, which obscure the
correlation between model responses and individual training samples. To enhance
privacy in SNNs, we explore two techniques: quantization and surrogate
gradients. Quantization, which reduces precision to limit information leakage,
has improved privacy in ANNs. Given SNNs' sparse and irregular activations,
quantization may further disrupt the activation patterns exploited by MIAs. We
assess the vulnerability of SNNs and ANNs under weight and activation
quantization across multiple datasets, using the attack model's receiver
operating characteristic (ROC) curve area under the curve (AUC) metric, where
lower values indicate stronger privacy, and evaluate the privacy-accuracy
trade-off. Our findings show that quantization enhances privacy in both
architectures with minimal performance loss, though full-precision SNNs remain
more resilient than quantized ANNs. Additionally, we examine the impact of
surrogate gradients on privacy in SNNs. Among five evaluated gradients, spike
rate escape provides the best privacy-accuracy trade-off, while arctangent
increases vulnerability to MIAs. These results reinforce SNNs' inherent privacy
advantages and demonstrate that quantization and surrogate gradient selection
significantly influence privacy-accuracy trade-offs in SNNs.


http://arxiv.org/abs/2502.18314
Learning atomic forces from uncertainty-calibrated adversarial attacks. (1%)
Henrique Musseli Cezar; Tilmann Bodenstein; Henrik Andersen Sveinsson; Morten Ledum; Simen Reine; Sigbjørn Løland Bore
  Adversarial approaches, which intentionally challenge machine learning models
by generating difficult examples, are increasingly being adopted to improve
machine learning interatomic potentials (MLIPs). While already providing great
practical value, little is known about the actual prediction errors of MLIPs on
adversarial structures and whether these errors can be controlled. We propose
the Calibrated Adversarial Geometry Optimization (CAGO) algorithm to discover
adversarial structures with user-assigned errors. Through uncertainty
calibration, the estimated uncertainty of MLIPs is unified with real errors. By
performing geometry optimization for calibrated uncertainty, we reach
adversarial structures with the user-assigned target MLIP prediction error.
Integrating with active learning pipelines, we benchmark CAGO, demonstrating
stable MLIPs that systematically converge structural, dynamical, and
thermodynamical properties for liquid water and water adsorption in a
metal-organic framework within only hundreds of training structures, where
previously many thousands were typically required.


http://arxiv.org/abs/2502.17880
VVRec: Reconstruction Attacks on DL-based Volumetric Video Upstreaming via Latent Diffusion Model with Gamma Distribution. (1%)
Rui Lu; Bihai Zhang; Dan Wang
  With the popularity of 3D volumetric video applications, such as Autonomous
Driving, Virtual Reality, and Mixed Reality, current developers have turned to
deep learning for compressing volumetric video frames, i.e., point clouds for
video upstreaming. The latest deep learning-based solutions offer higher
efficiency, lower distortion, and better hardware support compared to
traditional ones like MPEG and JPEG. However, privacy threats arise, especially
reconstruction attacks targeting to recover the original input point cloud from
the intermediate results. In this paper, we design VVRec, to the best of our
knowledge, which is the first targeting DL-based Volumetric Video
Reconstruction attack scheme. VVRec demonstrates the ability to reconstruct
high-quality point clouds from intercepted transmission intermediate results
using four well-trained neural network modules we design. Leveraging the latest
latent diffusion models with Gamma distribution and a refinement algorithm,
VVRec excels in reconstruction quality, color recovery, and surpasses existing
defenses. We evaluate VVRec using three volumetric video datasets. The results
demonstrate that VVRec achieves 64.70dB reconstruction accuracy, with an
impressive 46.39% reduction of distortion over baselines.


http://arxiv.org/abs/2502.18290
Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models. (1%)
Zhaoyi Liu; Huan Zhang
  Self-supervised learning (SSL) vision encoders learn high-quality image
representations and thus have become a vital part of developing vision modality
of large vision language models (LVLMs). Due to the high cost of training such
encoders, pre-trained encoders are widely shared and deployed into many LVLMs,
which are security-critical or bear societal significance. Under this practical
scenario, we reveal a new backdoor threat that significant visual
hallucinations can be induced into these LVLMs by merely compromising vision
encoders. Because of the sharing and reuse of these encoders, many downstream
LVLMs may inherit backdoor behaviors from encoders, leading to widespread
backdoors. In this work, we propose BadVision, the first method to exploit this
vulnerability in SSL vision encoders for LVLMs with novel trigger optimization
and backdoor learning techniques. We evaluate BadVision on two types of SSL
encoders and LVLMs across eight benchmarks. We show that BadVision effectively
drives the LVLMs to attacker-chosen hallucination with over 99% attack success
rate, causing a 77.6% relative visual understanding error while maintaining the
stealthiness. SoTA backdoor detection methods cannot detect our attack
effectively.


http://arxiv.org/abs/2502.17392
Emoti-Attack: Zero-Perturbation Adversarial Attacks on NLP Systems via Emoji Sequences. (99%)
Yangshijie Zhang
  Deep neural networks (DNNs) have achieved remarkable success in the field of
natural language processing (NLP), leading to widely recognized applications
such as ChatGPT. However, the vulnerability of these models to adversarial
attacks remains a significant concern. Unlike continuous domains like images,
text exists in a discrete space, making even minor alterations at the sentence,
word, or character level easily perceptible to humans. This inherent
discreteness also complicates the use of conventional optimization techniques,
as text is non-differentiable. Previous research on adversarial attacks in text
has focused on character-level, word-level, sentence-level, and multi-level
approaches, all of which suffer from inefficiency or perceptibility issues due
to the need for multiple queries or significant semantic shifts.
  In this work, we introduce a novel adversarial attack method, Emoji-Attack,
which leverages the manipulation of emojis to create subtle, yet effective,
perturbations. Unlike character- and word-level strategies, Emoji-Attack
targets emojis as a distinct layer of attack, resulting in less noticeable
changes with minimal disruption to the text. This approach has been largely
unexplored in previous research, which typically focuses on emoji insertion as
an extension of character-level attacks. Our experiments demonstrate that
Emoji-Attack achieves strong attack performance on both large and small models,
making it a promising technique for enhancing adversarial robustness in NLP
systems.


http://arxiv.org/abs/2502.17003
Improving the Transferability of Adversarial Examples by Inverse Knowledge Distillation. (99%)
Wenyuan Wu; Zheng Liu; Yong Chen; Chao Su; Dezhong Peng; Xu Wang
  In recent years, the rapid development of deep neural networks has brought
increased attention to the security and robustness of these models. While
existing adversarial attack algorithms have demonstrated success in improving
adversarial transferability, their performance remains suboptimal due to a lack
of consideration for the discrepancies between target and source models. To
address this limitation, we propose a novel method, Inverse Knowledge
Distillation (IKD), designed to enhance adversarial transferability
effectively. IKD introduces a distillation-inspired loss function that
seamlessly integrates with gradient-based attack methods, promoting diversity
in attack gradients and mitigating overfitting to specific model architectures.
By diversifying gradients, IKD enables the generation of adversarial samples
with superior generalization capabilities across different models,
significantly enhancing their effectiveness in black-box attack scenarios.
Extensive experiments on the ImageNet dataset validate the effectiveness of our
approach, demonstrating substantial improvements in the transferability and
attack success rates of adversarial samples across a wide range of models.


http://arxiv.org/abs/2502.17121
Adversarial Training for Defense Against Label Poisoning Attacks. (87%)
Melis Ilayda Bal; Volkan Cevher; Michael Muehlebach
  As machine learning models grow in complexity and increasingly rely on
publicly sourced data, such as the human-annotated labels used in training
large language models, they become more vulnerable to label poisoning attacks.
These attacks, in which adversaries subtly alter the labels within a training
dataset, can severely degrade model performance, posing significant risks in
critical applications. In this paper, we propose FLORAL, a novel adversarial
training defense strategy based on support vector machines (SVMs) to counter
these threats. Utilizing a bilevel optimization framework, we cast the training
process as a non-zero-sum Stackelberg game between an attacker, who
strategically poisons critical training labels, and the model, which seeks to
recover from such attacks. Our approach accommodates various model
architectures and employs a projected gradient descent algorithm with kernel
SVMs for adversarial training. We provide a theoretical analysis of our
algorithm's convergence properties and empirically evaluate FLORAL's
effectiveness across diverse classification tasks. Compared to robust baselines
and foundation models such as RoBERTa, FLORAL consistently achieves higher
robust accuracy under increasing attacker budgets. These results underscore the
potential of FLORAL to enhance the resilience of machine learning models
against label poisoning threats, thereby ensuring robust classification in
adversarial settings.


http://arxiv.org/abs/2502.17254
REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective. (87%)
Simon Geisler; Tom Wollschläger; M. H. I. Abdalla; Vincent Cohen-Addad; Johannes Gasteiger; Stephan Günnemann
  To circumvent the alignment of large language models (LLMs), current
optimization-based adversarial attacks usually craft adversarial prompts by
maximizing the likelihood of a so-called affirmative response. An affirmative
response is a manually designed start of a harmful answer to an inappropriate
request. While it is often easy to craft prompts that yield a substantial
likelihood for the affirmative response, the attacked model frequently does not
complete the response in a harmful manner. Moreover, the affirmative objective
is usually not adapted to model-specific preferences and essentially ignores
the fact that LLMs output a distribution over responses. If low attack success
under such an objective is taken as a measure of robustness, the true
robustness might be grossly overestimated. To alleviate these flaws, we propose
an adaptive and semantic optimization problem over the population of responses.
We derive a generally applicable objective via the REINFORCE policy-gradient
formalism and demonstrate its efficacy with the state-of-the-art jailbreak
algorithms Greedy Coordinate Gradient (GCG) and Projected Gradient Descent
(PGD). For example, our objective doubles the attack success rate (ASR) on
Llama3 and increases the ASR from 2% to 50% with circuit breaker defense.


http://arxiv.org/abs/2502.17537
On the Vulnerability of Concept Erasure in Diffusion Models. (76%)
Lucas Beerens; Alex D. Richardson; Kaicheng Zhang; Dongdong Chen
  The proliferation of text-to-image diffusion models has raised significant
privacy and security concerns, particularly regarding the generation of
copyrighted or harmful images. To address these issues, research on machine
unlearning has developed various concept erasure methods, which aim to remove
the effect of unwanted data through post-hoc training. However, we show these
erasure techniques are vulnerable, where images of supposedly erased concepts
can still be generated using adversarially crafted prompts. We introduce
RECORD, a coordinate-descent-based algorithm that discovers prompts capable of
eliciting the generation of erased content. We demonstrate that RECORD
significantly beats the attack success rate of current state-of-the-art attack
methods. Furthermore, our findings reveal that models subjected to concept
erasure are more susceptible to adversarial attacks than previously
anticipated, highlighting the urgency for more robust unlearning approaches. We
open source all our code at https://github.com/LucasBeerens/RECORD


http://arxiv.org/abs/2502.17832
MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks. (8%)
Hyeonjeong Ha; Qiusi Zhan; Jeonghwan Kim; Dimitrios Bralios; Saikrishna Sanniboina; Nanyun Peng; Kai-Wei Chang; Daniel Kang; Heng Ji
  Multimodal large language models (MLLMs) equipped with Retrieval Augmented
Generation (RAG) leverage both their rich parametric knowledge and the dynamic,
external knowledge to excel in tasks such as Question Answering. While RAG
enhances MLLMs by grounding responses in query-relevant external knowledge,
this reliance poses a critical yet underexplored safety risk: knowledge
poisoning attacks, where misinformation or irrelevant knowledge is
intentionally injected into external knowledge bases to manipulate model
outputs to be incorrect and even harmful. To expose such vulnerabilities in
multimodal RAG, we propose MM-PoisonRAG, a novel knowledge poisoning attack
framework with two attack strategies: Localized Poisoning Attack (LPA), which
injects query-specific misinformation in both text and images for targeted
manipulation, and Globalized Poisoning Attack (GPA) to provide false guidance
during MLLM generation to elicit nonsensical responses across all queries. We
evaluate our attacks across multiple tasks, models, and access settings,
demonstrating that LPA successfully manipulates the MLLM to generate
attacker-controlled answers, with a success rate of up to 56% on MultiModalQA.
Moreover, GPA completely disrupts model generation to 0% accuracy with just a
single irrelevant knowledge injection. Our results highlight the urgent need
for robust defenses against knowledge poisoning to safeguard multimodal RAG
frameworks.


http://arxiv.org/abs/2502.17602
A stochastic smoothing framework for nonconvex-nonconcave min-sum-max problems with applications to Wasserstein distributionally robust optimization. (1%)
Wei Liu; Muhammad Khan; Gabriel Mancino-Ball; Yangyang Xu
  Applications such as adversarially robust training and Wasserstein
Distributionally Robust Optimization (WDRO) can be naturally formulated as
min-sum-max optimization problems. While this formulation can be rewritten as
an equivalent min-max problem, the summation of max terms introduces
computational challenges, including increased complexity and memory demands,
which must be addressed. These challenges are particularly evident in WDRO,
where existing tractable algorithms often rely on restrictive assumptions on
the objective function, limiting their applicability to state-of-the-art
machine learning problems such as the training of deep neural networks. This
study introduces a novel stochastic smoothing framework based on the
\mbox{log-sum-exp} function, efficiently approximating the max operator in
min-sum-max problems. By leveraging the Clarke regularity of the max operator,
we develop an iterative smoothing algorithm that addresses these computational
difficulties and guarantees almost surely convergence to a Clarke/directional
stationary point. We further prove that the proposed algorithm finds an
$\epsilon$-scaled Clarke stationary point of the original problem, with a
worst-case iteration complexity of $\widetilde{O}(\epsilon^{-3})$. Our
numerical experiments demonstrate that our approach outperforms or is
competitive with state-of-the-art methods in solving the newsvendor problem,
deep learning regression, and adversarially robust deep learning. The results
highlight that our method yields more accurate and robust solutions in these
challenging problem settings.


http://arxiv.org/abs/2502.16793
VGFL-SA: Vertical Graph Federated Learning Structure Attack Based on Contrastive Learning. (97%)
Yang Chen; Bin Zhou
  Graph Neural Networks (GNNs) have gained attention for their ability to learn
representations from graph data. Due to privacy concerns and conflicts of
interest that prevent clients from directly sharing graph data with one
another, Vertical Graph Federated Learning (VGFL) frameworks have been
developed. Recent studies have shown that VGFL is vulnerable to adversarial
attacks that degrade performance. However, it is a common problem that client
nodes are often unlabeled in the realm of VGFL. Consequently, the existing
attacks, which rely on the availability of labeling information to obtain
gradients, are inherently constrained in their applicability. This limitation
precludes their deployment in practical, real-world environments. To address
the above problems, we propose a novel graph adversarial attack against VGFL,
referred to as VGFL-SA, to degrade the performance of VGFL by modifying the
local clients structure without using labels. Specifically, VGFL-SA uses a
contrastive learning method to complete the attack before the local clients are
trained. VGFL-SA first accesses the graph structure and node feature
information of the poisoned clients, and generates the contrastive views by
node-degree-based edge augmentation and feature shuffling augmentation. Then,
VGFL-SA uses the shared graph encoder to get the embedding of each view, and
the gradients of the adjacency matrices are obtained by the contrastive
function. Finally, perturbed edges are generated using gradient modification
rules. We validated the performance of VGFL-SA by performing a node
classification task on real-world datasets, and the results show that VGFL-SA
achieves good attack effectiveness and transferability.


http://arxiv.org/abs/2502.18520
Class-Conditional Neural Polarizer: A Lightweight and Effective Backdoor Defense by Purifying Poisoned Features. (93%)
Mingli Zhu; Shaokui Wei; Hongyuan Zha; Baoyuan Wu
  Recent studies have highlighted the vulnerability of deep neural networks to
backdoor attacks, where models are manipulated to rely on embedded triggers
within poisoned samples, despite the presence of both benign and trigger
information. While several defense methods have been proposed, they often
struggle to balance backdoor mitigation with maintaining benign performance.In
this work, inspired by the concept of optical polarizer-which allows light
waves of specific polarizations to pass while filtering others-we propose a
lightweight backdoor defense approach, NPD. This method integrates a neural
polarizer (NP) as an intermediate layer within the compromised model,
implemented as a lightweight linear transformation optimized via bi-level
optimization. The learnable NP filters trigger information from poisoned
samples while preserving benign content. Despite its effectiveness, we identify
through empirical studies that NPD's performance degrades when the target
labels (required for purification) are inaccurately estimated. To address this
limitation while harnessing the potential of targeted adversarial mitigation,
we propose class-conditional neural polarizer-based defense (CNPD). The key
innovation is a fusion module that integrates the backdoored model's predicted
label with the features to be purified. This architecture inherently mimics
targeted adversarial defense mechanisms without requiring label estimation used
in NPD. We propose three implementations of CNPD: the first is r-CNPD, which
trains a replicated NP layer for each class and, during inference, selects the
appropriate NP layer for defense based on the predicted class from the
backdoored model. To efficiently handle a large number of classes, two variants
are designed: e-CNPD, which embeds class information as additional features,
and a-CNPD, which directs network attention using class information.


http://arxiv.org/abs/2502.16593
Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images. (93%)
Yubo Wang; Jianting Tang; Chaohu Liu; Linli Xu
  Large vision-language models (LVLMs) have demonstrated remarkable image
understanding and dialogue capabilities, allowing them to handle a variety of
visual question answering tasks. However, their widespread availability raises
concerns about unauthorized usage and copyright infringement, where users or
individuals can develop their own LVLMs by fine-tuning published models. In
this paper, we propose a novel method called Parameter Learning Attack (PLA)
for tracking the copyright of LVLMs without modifying the original model.
Specifically, we construct adversarial images through targeted attacks against
the original model, enabling it to generate specific outputs. To ensure these
attacks remain effective on potential fine-tuned models to trigger copyright
tracking, we allow the original model to learn the trigger images by updating
parameters in the opposite direction during the adversarial attack process.
Notably, the proposed method can be applied after the release of the original
model, thus not affecting the model's performance and behavior. To simulate
real-world applications, we fine-tune the original model using various
strategies across diverse datasets, creating a range of models for copyright
verification. Extensive experiments demonstrate that our method can more
effectively identify the original copyright of fine-tuned models compared to
baseline methods. Therefore, this work provides a powerful tool for tracking
copyrights and detecting unlicensed usage of LVLMs.


http://arxiv.org/abs/2502.16737
Keeping up with dynamic attackers: Certifying robustness to adaptive online data poisoning. (68%)
Avinandan Bose; Laurent Lessard; Maryam Fazel; Krishnamurthy Dj Dvijotham
  The rise of foundation models fine-tuned on human feedback from potentially
untrusted users has increased the risk of adversarial data poisoning,
necessitating the study of robustness of learning algorithms against such
attacks. Existing research on provable certified robustness against data
poisoning attacks primarily focuses on certifying robustness for static
adversaries who modify a fraction of the dataset used to train the model before
the training algorithm is applied. In practice, particularly when learning from
human feedback in an online sense, adversaries can observe and react to the
learning process and inject poisoned samples that optimize adversarial
objectives better than when they are restricted to poisoning a static dataset
once, before the learning algorithm is applied. Indeed, it has been shown in
prior work that online dynamic adversaries can be significantly more powerful
than static ones. We present a novel framework for computing certified bounds
on the impact of dynamic poisoning, and use these certificates to design robust
learning algorithms. We give an illustration of the framework for the mean
estimation and binary classification problems and outline directions for
extending this in further work. The code to implement our certificates and
replicate our results is available at
https://github.com/Avinandan22/Certified-Robustness.


http://arxiv.org/abs/2502.16545
Multi-Target Federated Backdoor Attack Based on Feature Aggregation. (26%)
Lingguag Hao; Kuangrong Hao; Bing Wei; Xue-song Tang
  Current federated backdoor attacks focus on collaboratively training backdoor
triggers, where multiple compromised clients train their local trigger patches
and then merge them into a global trigger during the inference phase. However,
these methods require careful design of the shape and position of trigger
patches and lack the feature interactions between trigger patches during
training, resulting in poor backdoor attack success rates. Moreover, the pixels
of the patches remain untruncated, thereby making abrupt areas in backdoor
examples easily detectable by the detection algorithm. To this end, we propose
a novel benchmark for the federated backdoor attack based on feature
aggregation. Specifically, we align the dimensions of triggers with images,
delimit the trigger's pixel boundaries, and facilitate feature interaction
among local triggers trained by each compromised client. Furthermore,
leveraging the intra-class attack strategy, we propose the simultaneous
generation of backdoor triggers for all target classes, significantly reducing
the overall production time for triggers across all target classes and
increasing the risk of the federated model being attacked. Experiments
demonstrate that our method can not only bypass the detection of defense
methods while patch-based methods fail, but also achieve a zero-shot backdoor
attack with a success rate of 77.39%. To the best of our knowledge, our work is
the first to implement such a zero-shot attack in federated learning. Finally,
we evaluate attack performance by varying the trigger's training factors,
including poison location, ratio, pixel bound, and trigger training duration
(local epochs and communication rounds).


http://arxiv.org/abs/2502.16750
Guardians of the Agentic System: Preventing Many Shots Jailbreak with Agentic System. (16%)
Saikat Barua; Mostafizur Rahman; Md Jafor Sadek; Rafiul Islam; Shehenaz Khaled; Ahmedul Kabir
  The autonomous AI agents using large language models can create undeniable
values in all span of the society but they face security threats from
adversaries that warrants immediate protective solutions because trust and
safety issues arise. Considering the many-shot jailbreaking and deceptive
alignment as some of the main advanced attacks, that cannot be mitigated by the
static guardrails used during the supervised training, points out a crucial
research priority for real world robustness. The combination of static
guardrails in dynamic multi-agent system fails to defend against those attacks.
We intend to enhance security for LLM-based agents through the development of
new evaluation frameworks which identify and counter threats for safe
operational deployment. Our work uses three examination methods to detect rogue
agents through a Reverse Turing Test and analyze deceptive alignment through
multi-agent simulations and develops an anti-jailbreaking system by testing it
with GEMINI 1.5 pro and llama-3.3-70B, deepseek r1 models using tool-mediated
adversarial scenarios. The detection capabilities are strong such as 94\%
accuracy for GEMINI 1.5 pro yet the system suffers persistent vulnerabilities
when under long attacks as prompt length increases attack success rates (ASR)
and diversity metrics become ineffective in prediction while revealing multiple
complex system faults. The findings demonstrate the necessity of adopting
flexible security systems based on active monitoring that can be performed by
the agents themselves together with adaptable interventions by system admin as
the current models can create vulnerabilities that can lead to the unreliable
and vulnerable system. So, in our work, we try to address such situations and
propose a comprehensive framework to counteract the security issues.


http://arxiv.org/abs/2502.16734
Towards Optimal Adversarial Robust Reinforcement Learning with Infinity Measurement Error. (13%)
Haoran Li; Zicheng Zhang; Wang Luo; Congying Han; Jiayu Lv; Tiande Guo; Yudong Hu
  Ensuring the robustness of deep reinforcement learning (DRL) agents against
adversarial attacks is critical for their trustworthy deployment. Recent
research highlights the challenges of achieving state-adversarial robustness
and suggests that an optimal robust policy (ORP) does not always exist,
complicating the enforcement of strict robustness constraints. In this paper,
we further explore the concept of ORP. We first introduce the Intrinsic
State-adversarial Markov Decision Process (ISA-MDP), a novel formulation where
adversaries cannot fundamentally alter the intrinsic nature of state
observations. ISA-MDP, supported by empirical and theoretical evidence,
universally characterizes decision-making under state-adversarial paradigms. We
rigorously prove that within ISA-MDP, a deterministic and stationary ORP
exists, aligning with the Bellman optimal policy. Our findings theoretically
reveal that improving DRL robustness does not necessarily compromise
performance in natural environments. Furthermore, we demonstrate the necessity
of infinity measurement error (IME) in both $Q$-function and probability spaces
to achieve ORP, unveiling vulnerabilities of previous DRL algorithms that rely
on $1$-measurement errors. Motivated by these insights, we develop the
Consistent Adversarial Robust Reinforcement Learning (CAR-RL) framework, which
optimizes surrogates of IME. We apply CAR-RL to both value-based and
policy-based DRL algorithms, achieving superior performance and validating our
theoretical analysis.


http://arxiv.org/abs/2502.16523
Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension. (8%)
Yulong Wu; Viktor Schlegel; Riza Batista-Navarro
  As neural language models achieve human-comparable performance on Machine
Reading Comprehension (MRC) and see widespread adoption, ensuring their
robustness in real-world scenarios has become increasingly important. Current
robustness evaluation research, though, primarily develops synthetic
perturbation methods, leaving unclear how well they reflect real life
scenarios. Considering this, we present a framework to automatically examine
MRC models on naturally occurring textual perturbations, by replacing paragraph
in MRC benchmarks with their counterparts based on available Wikipedia edit
history. Such perturbation type is natural as its design does not stem from an
arteficial generative process, inherently distinct from the previously
investigated synthetic approaches. In a large-scale study encompassing SQUAD
datasets and various model architectures we observe that natural perturbations
result in performance degradation in pre-trained encoder language models. More
worryingly, these state-of-the-art Flan-T5 and Large Language Models (LLMs)
inherit these errors. Further experiments demonstrate that our findings
generalise to natural perturbations found in other more challenging MRC
benchmarks. In an effort to mitigate these errors, we show that it is possible
to improve the robustness to natural perturbations by training on naturally or
synthetically perturbed examples, though a noticeable gap still remains
compared to performance on unperturbed data.


http://arxiv.org/abs/2502.16776
AISafetyLab: A Comprehensive Framework for AI Safety Evaluation and Improvement. (1%)
Zhexin Zhang; Leqi Lei; Junxiao Yang; Xijie Huang; Yida Lu; Shiyao Cui; Renmiao Chen; Qinglin Zhang; Xinyuan Wang; Hao Wang; Hao Li; Xianqi Lei; Chengwei Pan; Lei Sha; Hongning Wang; Minlie Huang
  As AI models are increasingly deployed across diverse real-world scenarios,
ensuring their safety remains a critical yet underexplored challenge. While
substantial efforts have been made to evaluate and enhance AI safety, the lack
of a standardized framework and comprehensive toolkit poses significant
obstacles to systematic research and practical adoption. To bridge this gap, we
introduce AISafetyLab, a unified framework and toolkit that integrates
representative attack, defense, and evaluation methodologies for AI safety.
AISafetyLab features an intuitive interface that enables developers to
seamlessly apply various techniques while maintaining a well-structured and
extensible codebase for future advancements. Additionally, we conduct empirical
studies on Vicuna, analyzing different attack and defense strategies to provide
valuable insights into their comparative effectiveness. To facilitate ongoing
research and development in AI safety, AISafetyLab is publicly available at
https://github.com/thu-coai/AISafetyLab, and we are committed to its continuous
maintenance and improvement.


http://arxiv.org/abs/2502.18508
REFINE: Inversion-Free Backdoor Defense via Model Reprogramming. (84%)
Yukun Chen; Shuo Shao; Enhao Huang; Yiming Li; Pin-Yu Chen; Zhan Qin; Kui Ren
  Backdoor attacks on deep neural networks (DNNs) have emerged as a significant
security threat, allowing adversaries to implant hidden malicious behaviors
during the model training phase. Pre-processing-based defense, which is one of
the most important defense paradigms, typically focuses on input
transformations or backdoor trigger inversion (BTI) to deactivate or eliminate
embedded backdoor triggers during the inference process. However, these methods
suffer from inherent limitations: transformation-based defenses often fail to
balance model utility and defense performance, while BTI-based defenses
struggle to accurately reconstruct trigger patterns without prior knowledge. In
this paper, we propose REFINE, an inversion-free backdoor defense method based
on model reprogramming. REFINE consists of two key components: \textbf{(1)} an
input transformation module that disrupts both benign and backdoor patterns,
generating new benign features; and \textbf{(2)} an output remapping module
that redefines the model's output domain to guide the input transformations
effectively. By further integrating supervised contrastive loss, REFINE
enhances the defense capabilities while maintaining model utility. Extensive
experiments on various benchmark datasets demonstrate the effectiveness of our
REFINE and its resistance to potential adaptive attacks.


http://arxiv.org/abs/2502.16361
A Framework for Evaluating Vision-Language Model Safety: Building Trust in AI for Public Sector Applications. (80%)
Maisha Binte Rashid; Pablo Rivas
  Vision-Language Models (VLMs) are increasingly deployed in public sector
missions, necessitating robust evaluation of their safety and vulnerability to
adversarial attacks. This paper introduces a novel framework to quantify
adversarial risks in VLMs. We analyze model performance under Gaussian,
salt-and-pepper, and uniform noise, identifying misclassification thresholds
and deriving composite noise patches and saliency patterns that highlight
vulnerable regions. These patterns are compared against the Fast Gradient Sign
Method (FGSM) to assess their adversarial effectiveness. We propose a new
Vulnerability Score that combines the impact of random noise and adversarial
attacks, providing a comprehensive metric for evaluating model robustness.


http://arxiv.org/abs/2502.16167
PersGuard: Preventing Malicious Personalization via Backdoor Attacks on Pre-trained Text-to-Image Diffusion Models. (45%)
Xinwei Liu; Xiaojun Jia; Yuan Xun; Hua Zhang; Xiaochun Cao
  Diffusion models (DMs) have revolutionized data generation, particularly in
text-to-image (T2I) synthesis. However, the widespread use of personalized
generative models raises significant concerns regarding privacy violations and
copyright infringement. To address these issues, researchers have proposed
adversarial perturbation-based protection techniques. However, these methods
have notable limitations, including insufficient robustness against data
transformations and the inability to fully eliminate identifiable features of
protected objects in the generated output. In this paper, we introduce
PersGuard, a novel backdoor-based approach that prevents malicious
personalization of specific images. Unlike traditional adversarial perturbation
methods, PersGuard implant backdoor triggers into pre-trained T2I models,
preventing the generation of customized outputs for designated protected images
while allowing normal personalization for unprotected ones. Unfortunately,
existing backdoor methods for T2I diffusion models fail to be applied to
personalization scenarios due to the different backdoor objectives and the
potential backdoor elimination during downstream fine-tuning processes. To
address these, we propose three novel backdoor objectives specifically designed
for personalization scenarios, coupled with backdoor retention loss engineered
to resist downstream fine-tuning. These components are integrated into a
unified optimization framework. Extensive experimental evaluations demonstrate
PersGuard's effectiveness in preserving data privacy, even under challenging
conditions including gray-box settings, multi-object protection, and facial
identity scenarios. Our method significantly outperforms existing techniques,
offering a more robust solution for privacy and copyright protection.


http://arxiv.org/abs/2502.18511
ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Models. (33%)
Xuxu Liu; Siyuan Liang; Mengya Han; Yong Luo; Aishan Liu; Xiantao Cai; Zheng He; Dacheng Tao
  Generative large language models are crucial in natural language processing,
but they are vulnerable to backdoor attacks, where subtle triggers compromise
their behavior. Although backdoor attacks against LLMs are constantly emerging,
existing benchmarks remain limited in terms of sufficient coverage of attack,
metric system integrity, backdoor attack alignment. And existing pre-trained
backdoor attacks are idealized in practice due to resource access constraints.
Therefore we establish $\textit{ELBA-Bench}$, a comprehensive and unified
framework that allows attackers to inject backdoor through parameter efficient
fine-tuning ($\textit{e.g.,}$ LoRA) or without fine-tuning techniques
($\textit{e.g.,}$ In-context-learning). $\textit{ELBA-Bench}$ provides over
1300 experiments encompassing the implementations of 12 attack methods, 18
datasets, and 12 LLMs. Extensive experiments provide new invaluable findings
into the strengths and limitations of various attack strategies. For instance,
PEFT attack consistently outperform without fine-tuning approaches in
classification tasks while showing strong cross-dataset generalization with
optimized triggers boosting robustness; Task-relevant backdoor optimization
techniques or attack prompts along with clean and adversarial demonstrations
can enhance backdoor attack success while preserving model performance on clean
samples. Additionally, we introduce a universal toolbox designed for
standardized backdoor attack research, with the goal of propelling further
progress in this vital area.


http://arxiv.org/abs/2502.16396
FedNIA: Noise-Induced Activation Analysis for Mitigating Data Poisoning in FL. (31%)
Ehsan Hallaji; Roozbeh Razavi-Far; Mehrdad Saif
  Federated learning systems are increasingly threatened by data poisoning
attacks, where malicious clients compromise global models by contributing
tampered updates. Existing defenses often rely on impractical assumptions, such
as access to a central test dataset, or fail to generalize across diverse
attack types, particularly those involving multiple malicious clients working
collaboratively. To address this, we propose Federated Noise-Induced Activation
Analysis (FedNIA), a novel defense framework to identify and exclude
adversarial clients without relying on any central test dataset. FedNIA injects
random noise inputs to analyze the layerwise activation patterns in client
models leveraging an autoencoder that detects abnormal behaviors indicative of
data poisoning. FedNIA can defend against diverse attack types, including
sample poisoning, label flipping, and backdoors, even in scenarios with
multiple attacking nodes. Experimental results on non-iid federated datasets
demonstrate its effectiveness and robustness, underscoring its potential as a
foundational approach for enhancing the security of federated learning systems.


http://arxiv.org/abs/2502.16286
Verification of Bit-Flip Attacks against Quantized Neural Networks. (10%)
Yedi Zhang; Lei Huang; Pengfei Gao; Fu Song; Jun Sun; Jin Song Dong
  In the rapidly evolving landscape of neural network security, the resilience
of neural networks against bit-flip attacks (i.e., an attacker maliciously
flips an extremely small amount of bits within its parameter storage memory
system to induce harmful behavior), has emerged as a relevant area of research.
Existing studies suggest that quantization may serve as a viable defense
against such attacks. Recognizing the documented susceptibility of real-valued
neural networks to such attacks and the comparative robustness of quantized
neural networks (QNNs), in this work, we introduce BFAVerifier, the first
verification framework designed to formally verify the absence of bit-flip
attacks or to identify all vulnerable parameters in a sound and rigorous
manner. BFAVerifier comprises two integral components: an abstraction-based
method and an MILP-based method. Specifically, we first conduct a reachability
analysis with respect to symbolic parameters that represent the potential
bit-flip attacks, based on a novel abstract domain with a sound guarantee. If
the reachability analysis fails to prove the resilience of such attacks, then
we encode this verification problem into an equivalent MILP problem which can
be solved by off-the-shelf solvers. Therefore, BFAVerifier is sound, complete,
and reasonably efficient. We conduct extensive experiments, which demonstrate
its effectiveness and efficiency across various network architectures,
quantization bit-widths, and adversary capabilities.


http://arxiv.org/abs/2502.16423
Unified Prompt Attack Against Text-to-Image Generation Models. (8%)
Duo Peng; Qiuhong Ke; Mark He Huang; Ping Hu; Jun Liu
  Text-to-Image (T2I) models have advanced significantly, but their growing
popularity raises security concerns due to their potential to generate harmful
images. To address these issues, we propose UPAM, a novel framework to evaluate
the robustness of T2I models from an attack perspective. Unlike prior methods
that focus solely on textual defenses, UPAM unifies the attack on both textual
and visual defenses. Additionally, it enables gradient-based optimization,
overcoming reliance on enumeration for improved efficiency and effectiveness.
To handle cases where T2I models block image outputs due to defenses, we
introduce Sphere-Probing Learning (SPL) to enable optimization even without
image results. Following SPL, our model bypasses defenses, inducing the
generation of harmful content. To ensure semantic alignment with attacker
intent, we propose Semantic-Enhancing Learning (SEL) for precise semantic
control. UPAM also prioritizes the naturalness of adversarial prompts using
In-context Naturalness Enhancement (INE), making them harder for human
examiners to detect. Additionally, we address the issue of iterative
queries--common in prior methods and easily detectable by API defenders--by
introducing Transferable Attack Learning (TAL), allowing effective attacks with
minimal queries. Extensive experiments validate UPAM's superiority in
effectiveness, efficiency, naturalness, and low query detection rates.


http://arxiv.org/abs/2502.16115
Detecting OOD Samples via Optimal Transport Scoring Function. (1%)
Heng Gao; Zhuolin He; Jian Pu
  To deploy machine learning models in the real world, researchers have
proposed many OOD detection algorithms to help models identify unknown samples
during the inference phase and prevent them from making untrustworthy
predictions. Unlike methods that rely on extra data for outlier exposure
training, post hoc methods detect Out-of-Distribution (OOD) samples by
developing scoring functions, which are model agnostic and do not require
additional training. However, previous post hoc methods may fail to capture the
geometric cues embedded in network representations. Thus, in this study, we
propose a novel score function based on the optimal transport theory, named
OTOD, for OOD detection. We utilize information from features, logits, and the
softmax probability space to calculate the OOD score for each test sample. Our
experiments show that combining this information can boost the performance of
OTOD with a certain margin. Experiments on the CIFAR-10 and CIFAR-100
benchmarks demonstrate the superior performance of our method. Notably, OTOD
outperforms the state-of-the-art method GEN by 7.19% in the mean FPR@95 on the
CIFAR-10 benchmark using ResNet-18 as the backbone, and by 12.51% in the mean
FPR@95 using WideResNet-28 as the backbone. In addition, we provide theoretical
guarantees for OTOD. The code is available in
https://github.com/HengGao12/OTOD.


http://arxiv.org/abs/2502.16094
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging. (1%)
Lin Lu; Zhigang Zuo; Ziji Sheng; Pan Zhou
  Model merging has emerged as a promising approach for updating large language
models (LLMs) by integrating multiple domain-specific models into a
cross-domain merged model. Despite its utility and plug-and-play nature,
unmonitored mergers can introduce significant security vulnerabilities, such as
backdoor attacks and model merging abuse. In this paper, we identify a novel
and more realistic attack surface where a malicious merger can extract targeted
personally identifiable information (PII) from an aligned model with model
merging. Specifically, we propose \texttt{Merger-as-a-Stealer}, a two-stage
framework to achieve this attack: First, the attacker fine-tunes a malicious
model to force it to respond to any PII-related queries. The attacker then
uploads this malicious model to the model merging conductor and obtains the
merged model. Second, the attacker inputs direct PII-related queries to the
merged model to extract targeted PII. Extensive experiments demonstrate that
\texttt{Merger-as-a-Stealer} successfully executes attacks against various LLMs
and model merging methods across diverse settings, highlighting the
effectiveness of the proposed framework. Given that this attack enables
character-level extraction for targeted PII without requiring any additional
knowledge from the attacker, we stress the necessity for improved model
alignment and more robust defense mechanisms to mitigate such threats.


http://arxiv.org/abs/2502.16366
A generative approach to LLM harmfulness detection with special red flag tokens. (1%)
Sophie Xhonneux; David Dobre; Mehrnaz Mohfakhami; Leo Schwinn; Gauthier Gidel
  Most safety training methods for large language models (LLMs) based on
fine-tuning rely on dramatically changing the output distribution of the model
when faced with a harmful request, shifting it from an unsafe answer to a
refusal to respond. These methods inherently compromise model capabilities and
might make auto-regressive models vulnerable to attacks that make likely an
initial token of affirmative response. To avoid that, we propose to expand the
model's vocabulary with a special token we call red flag token (<rf>) and
propose to fine-tune the model to generate this token at any time harmful
content is generated or about to be generated. This novel safety training
method effectively augments LLMs into generative classifiers of harmfulness at
all times during the conversation. This method offers several advantages: it
enables the model to explicitly learn the concept of harmfulness while
marginally affecting the generated distribution, thus maintaining the model's
utility. It also evaluates each generated answer rather than just the input
prompt and provides a stronger defence against sampling-based attacks. In
addition, it simplifies the evaluation of the model's robustness and reduces
correlated failures when combined with a classifier. We further show an
increased robustness to long contexts, and supervised fine-tuning attacks.


http://arxiv.org/abs/2502.16044
A Multi-Scale Isolation Forest Approach for Real-Time Detection and Filtering of FGSM Adversarial Attacks in Video Streams of Autonomous Vehicles. (99%)
Richard Abhulimhen; Negash Begashaw; Gurcan Comert; Chunheng Zhao; Pierluigi Pisu
  Deep Neural Networks (DNNs) have demonstrated remarkable success across a
wide range of tasks, particularly in fields such as image classification.
However, DNNs are highly susceptible to adversarial attacks, where subtle
perturbations are introduced to input images, leading to erroneous model
outputs. In today's digital era, ensuring the security and integrity of images
processed by DNNs is of critical importance. One of the most prominent
adversarial attack methods is the Fast Gradient Sign Method (FGSM), which
perturbs images in the direction of the loss gradient to deceive the model.
  This paper presents a novel approach for detecting and filtering FGSM
adversarial attacks in image processing tasks. Our proposed method evaluates
10,000 images, each subjected to five different levels of perturbation,
characterized by $\epsilon$ values of 0.01, 0.02, 0.05, 0.1, and 0.2. These
perturbations are applied in the direction of the loss gradient. We demonstrate
that our approach effectively filters adversarially perturbed images,
mitigating the impact of FGSM attacks.
  The method is implemented in Python, and the source code is publicly
available on GitHub for reproducibility and further research.


http://arxiv.org/abs/2502.16012
Cross-Model Transferability of Adversarial Patches in Real-time Segmentation for Autonomous Driving. (98%)
Prashant Shekhar; Bidur Devkota; Dumindu Samaraweera; Laxima Niure Kandel; Manoj Babu
  Adversarial attacks pose a significant threat to deep learning models,
particularly in safety-critical applications like healthcare and autonomous
driving. Recently, patch based attacks have demonstrated effectiveness in
real-time inference scenarios owing to their 'drag and drop' nature. Following
this idea for Semantic Segmentation (SS), here we propose a novel Expectation
Over Transformation (EOT) based adversarial patch attack that is more realistic
for autonomous vehicles. To effectively train this attack we also propose a
'simplified' loss function that is easy to analyze and implement. Using this
attack as our basis, we investigate whether adversarial patches once optimized
on a specific SS model, can fool other models or architectures. We conduct a
comprehensive cross-model transferability analysis of adversarial patches
trained on SOTA Convolutional Neural Network (CNN) models such PIDNet-S,
PIDNet-M and PIDNet-L, among others. Additionally, we also include the
Segformer model to study transferability to Vision Transformers (ViTs). All of
our analysis is conducted on the widely used Cityscapes dataset. Our study
reveals key insights into how model architectures (CNN vs CNN or CNN vs.
Transformer-based) influence attack susceptibility. In particular, we conclude
that although the transferability (effectiveness) of attacks on unseen images
of any dimension is really high, the attacks trained against one particular
model are minimally effective on other models. And this was found to be true
for both ViT and CNN based models. Additionally our results also indicate that
for CNN-based models, the repercussions of patch attacks are local, unlike
ViTs. Per-class analysis reveals that simple-classes like 'sky' suffer less
misclassification than others. The code for the project is available at:
https://github.com/p-shekhar/adversarial-patch-transferability


http://arxiv.org/abs/2502.15561
A Defensive Framework Against Adversarial Attacks on Machine Learning-Based Network Intrusion Detection Systems. (92%)
Benyamin Tafreshian; Shengzhi Zhang
  As cyberattacks become increasingly sophisticated, advanced Network Intrusion
Detection Systems (NIDS) are critical for modern network security. Traditional
signature-based NIDS are inadequate against zero-day and evolving attacks. In
response, machine learning (ML)-based NIDS have emerged as promising solutions;
however, they are vulnerable to adversarial evasion attacks that subtly
manipulate network traffic to bypass detection. To address this vulnerability,
we propose a novel defensive framework that enhances the robustness of ML-based
NIDS by simultaneously integrating adversarial training, dataset balancing
techniques, advanced feature engineering, ensemble learning, and extensive
model fine-tuning. We validate our framework using the NSL-KDD and UNSW-NB15
datasets. Experimental results show, on average, a 35% increase in detection
accuracy and a 12.5% reduction in false positives compared to baseline models,
particularly under adversarial conditions. The proposed defense against
adversarial attacks significantly advances the practical deployment of robust
ML-based NIDS in real-world networks.


http://arxiv.org/abs/2502.15567
Model Privacy: A Unified Framework to Understand Model Stealing Attacks and Defenses. (78%)
Ganghua Wang; Yuhong Yang; Jie Ding
  The use of machine learning (ML) has become increasingly prevalent in various
domains, highlighting the importance of understanding and ensuring its safety.
One pressing concern is the vulnerability of ML applications to model stealing
attacks. These attacks involve adversaries attempting to recover a learned
model through limited query-response interactions, such as those found in
cloud-based services or on-chip artificial intelligence interfaces. While
existing literature proposes various attack and defense strategies, these often
lack a theoretical foundation and standardized evaluation criteria. In
response, this work presents a framework called ``Model Privacy'', providing a
foundation for comprehensively analyzing model stealing attacks and defenses.
We establish a rigorous formulation for the threat model and objectives,
propose methods to quantify the goodness of attack and defense strategies, and
analyze the fundamental tradeoffs between utility and privacy in ML models. Our
developed theory offers valuable insights into enhancing the security of ML
models, especially highlighting the importance of the attack-specific structure
of perturbations for effective defenses. We demonstrate the application of
model privacy from the defender's perspective through various learning
scenarios. Extensive experiments corroborate the insights and the effectiveness
of defense mechanisms developed under the proposed framework.


http://arxiv.org/abs/2502.15594
SafeInt: Shielding Large Language Models from Jailbreak Attacks via Safety-Aware Representation Intervention. (67%)
Jiaqi Wu; Chen Chen; Chunyan Hou; Xiaojie Yuan
  With the widespread real-world deployment of large language models (LLMs),
ensuring their behavior complies with safety standards has become crucial.
Jailbreak attacks exploit vulnerabilities in LLMs to induce undesirable
behavior, posing a significant threat to LLM safety. Previous defenses often
fail to achieve both effectiveness and efficiency simultaneously. Defenses from
a representation perspective offer new insights, but existing interventions
cannot dynamically adjust representations based on the harmfulness of the
queries. To address this limitation, we propose SafeIntervention (SafeInt), a
novel defense method that shields LLMs from jailbreak attacks through
safety-aware representation intervention. Built on our analysis of the
representations of jailbreak samples, the core idea of SafeInt is to relocate
jailbreak-related representations into the rejection region. This is achieved
by intervening in the representation distributions of jailbreak samples to
align them with those of unsafe samples. We conduct comprehensive experiments
covering six jailbreak attacks, two jailbreak datasets, and two utility
benchmarks. Experimental results demonstrate that SafeInt outperforms all
baselines in defending LLMs against jailbreak attacks while largely maintaining
utility. Additionally, we evaluate SafeInt against adaptive attacks and verify
its effectiveness in mitigating real-time attacks.


http://arxiv.org/abs/2502.18504
TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice. (13%)
Aman Goel; Xian Carrie Wu; Zhe Wang; Dmitriy Bespalov; Yanjun Qi
  Jailbreaking large-language models (LLMs) involves testing their robustness
against adversarial prompts and evaluating their ability to withstand prompt
attacks that could elicit unauthorized or malicious responses. In this paper,
we present TurboFuzzLLM, a mutation-based fuzzing technique for efficiently
finding a collection of effective jailbreaking templates that, when combined
with harmful questions, can lead a target LLM to produce harmful responses
through black-box access via user prompts. We describe the limitations of
directly applying existing template-based attacking techniques in practice, and
present functional and efficiency-focused upgrades we added to mutation-based
fuzzing to generate effective jailbreaking templates automatically.
TurboFuzzLLM achieves $\geq$ 95\% attack success rates (ASR) on public datasets
for leading LLMs (including GPT-4o \& GPT-4 Turbo), shows impressive
generalizability to unseen harmful questions, and helps in improving model
defenses to prompt attacks.


http://arxiv.org/abs/2502.16065
A Survey of Model Extraction Attacks and Defenses in Distributed Computing Environments. (10%)
Kaixiang Zhao; Lincan Li; Kaize Ding; Neil Zhenqiang Gong; Yue Zhao; Yushun Dong
  Model Extraction Attacks (MEAs) threaten modern machine learning systems by
enabling adversaries to steal models, exposing intellectual property and
training data. With the increasing deployment of machine learning models in
distributed computing environments, including cloud, edge, and federated
learning settings, each paradigm introduces distinct vulnerabilities and
challenges. Without a unified perspective on MEAs across these distributed
environments, organizations risk fragmented defenses, inadequate risk
assessments, and substantial economic and privacy losses. This survey is
motivated by the urgent need to understand how the unique characteristics of
cloud, edge, and federated deployments shape attack vectors and defense
requirements. We systematically examine the evolution of attack methodologies
and defense mechanisms across these environments, demonstrating how
environmental factors influence security strategies in critical sectors such as
autonomous vehicles, healthcare, and financial services. By synthesizing recent
advances in MEAs research and discussing the limitations of current evaluation
practices, this survey provides essential insights for developing robust and
adaptive defense strategies. Our comprehensive approach highlights the
importance of integrating protective measures across the entire distributed
computing landscape to ensure the secure deployment of machine learning models.


http://arxiv.org/abs/2502.15435
Single-pass Detection of Jailbreaking Input in Large Language Models. (2%)
Leyla Naz Candogan; Yongtao Wu; Elias Abad Rocamora; Grigorios G. Chrysos; Volkan Cevher
  Defending aligned Large Language Models (LLMs) against jailbreaking attacks
is a challenging problem, with existing approaches requiring multiple requests
or even queries to auxiliary LLMs, making them computationally heavy. Instead,
we focus on detecting jailbreaking input in a single forward pass. Our method,
called Single Pass Detection SPD, leverages the information carried by the
logits to predict whether the output sentence will be harmful. This allows us
to defend in just one forward pass. SPD can not only detect attacks effectively
on open-source models, but also minimizes the misclassification of harmless
inputs. Furthermore, we show that SPD remains effective even without complete
logit access in GPT-3.5 and GPT-4. We believe that our proposed method offers a
promising approach to efficiently safeguard LLMs against adversarial attacks.


http://arxiv.org/abs/2502.14976
EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models. (95%)
Nastaran Darabi; Devashri Naik; Sina Tayebati; Dinithi Jayasuriya; Ranganath Krishnan; Amit Ranjan Trivedi
  Vision-Language Models (VLMs) inherit adversarial vulnerabilities of Large
Language Models (LLMs), which are further exacerbated by their multimodal
nature. Existing defenses, including adversarial training, input
transformations, and heuristic detection, are computationally expensive,
architecture-dependent, and fragile against adaptive attacks. We introduce
EigenShield, an inference-time defense leveraging Random Matrix Theory to
quantify adversarial disruptions in high-dimensional VLM representations.
Unlike prior methods that rely on empirical heuristics, EigenShield employs the
spiked covariance model to detect structured spectral deviations. Using a
Robustness-based Nonconformity Score (RbNS) and quantile-based thresholding, it
separates causal eigenvectors, which encode semantic information, from
correlational eigenvectors that are susceptible to adversarial artifacts. By
projecting embeddings onto the causal subspace, EigenShield filters adversarial
noise without modifying model parameters or requiring adversarial training.
This architecture-independent, attack-agnostic approach significantly reduces
the attack success rate, establishing spectral analysis as a principled
alternative to conventional defenses. Our results demonstrate that EigenShield
consistently outperforms all existing defenses, including adversarial training,
UNIGUARD, and CIDER.


http://arxiv.org/abs/2502.14586
Moshi Moshi? A Model Selection Hijacking Adversarial Attack. (92%)
Riccardo Petrucci; Luca Pajola; Francesco Marchiori; Luca Pasa; Mauro conti
  Model selection is a fundamental task in Machine Learning~(ML), focusing on
selecting the most suitable model from a pool of candidates by evaluating their
performance on specific metrics. This process ensures optimal performance,
computational efficiency, and adaptability to diverse tasks and environments.
Despite its critical role, its security from the perspective of adversarial ML
remains unexplored. This risk is heightened in the
Machine-Learning-as-a-Service model, where users delegate the training phase
and the model selection process to third-party providers, supplying data and
training strategies. Therefore, attacks on model selection could harm both the
user and the provider, undermining model performance and driving up operational
costs.
  In this work, we present MOSHI (MOdel Selection HIjacking adversarial
attack), the first adversarial attack specifically targeting model selection.
Our novel approach manipulates model selection data to favor the adversary,
even without prior knowledge of the system. Utilizing a framework based on
Variational Auto Encoders, we provide evidence that an attacker can induce
inefficiencies in ML deployment. We test our attack on diverse computer vision
and speech recognition benchmark tasks and different settings, obtaining an
average attack success rate of 75.42%. In particular, our attack causes an
average 88.30% decrease in generalization capabilities, an 83.33% increase in
latency, and an increase of up to 105.85% in energy consumption. These results
highlight the significant vulnerabilities in model selection processes and
their potential impact on real-world applications.


http://arxiv.org/abs/2502.15017
Interpreting Adversarial Attacks and Defences using Architectures with Enhanced Interpretability. (81%)
Akshay G Rao; Chandrashekhar Lakshminarayanan; Arun Rajkumar
  Adversarial attacks in deep learning represent a significant threat to the
integrity and reliability of machine learning models. Adversarial training has
been a popular defence technique against these adversarial attacks. In this
work, we capitalize on a network architecture, namely Deep Linearly Gated
Networks (DLGN), which has better interpretation capabilities than regular deep
network architectures. Using this architecture, we interpret robust models
trained using PGD adversarial training and compare them with standard training.
Feature networks in DLGN act as feature extractors, making them the only medium
through which an adversary can attack the model. We analyze the feature network
of DLGN with fully connected layers with respect to properties like alignment
of the hyperplanes, hyperplane relation with PCA, and sub-network overlap among
classes and compare these properties between robust and standard models. We
also consider this architecture having CNN layers wherein we qualitatively
(using visualizations) and quantitatively contrast gating patterns between
robust and standard models. We uncover insights into hyperplanes resembling
principal components in PGD-AT and STD-TR models, with PGD-AT hyperplanes
aligned farther from the data points. We use path activity analysis to show
that PGD-AT models create diverse, non-overlapping active subnetworks across
classes, preventing attack-induced gating overlaps. Our visualization ideas
show the nature of representations learnt by PGD-AT and STD-TR models.


http://arxiv.org/abs/2502.14296
On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspective. (64%)
Yue Huang; Chujie Gao; Siyuan Wu; Haoran Wang; Xiangqi Wang; Yujun Zhou; Yanbo Wang; Jiayi Ye; Jiawen Shi; Qihui Zhang; Yuan Li; Han Bao; Zhaoyi Liu; Tianrui Guan; Dongping Chen; Ruoxi Chen; Kehan Guo; Andy Zou; Bryan Hooi Kuen-Yew; Caiming Xiong; Elias Stengel-Eskin; Hongyang Zhang; Hongzhi Yin; Huan Zhang; Huaxiu Yao; Jaehong Yoon; Jieyu Zhang; Kai Shu; Kaijie Zhu; Ranjay Krishna; Swabha Swayamdipta; Taiwei Shi; Weijia Shi; Xiang Li; Yiwei Li; Yuexing Hao; Yuexing Hao; Zhihao Jia; Zhize Li; Xiuying Chen; Zhengzhong Tu; Xiyang Hu; Tianyi Zhou; Jieyu Zhao; Lichao Sun; Furong Huang; Or Cohen Sasson; Prasanna Sattigeri; Anka Reuel; Max Lamparth; Yue Zhao; Nouha Dziri; Yu Su; Huan Sun; Heng Ji; Chaowei Xiao; Mohit Bansal; Nitesh V. Chawla; Jian Pei; Jianfeng Gao; Michael Backes; Philip S. Yu; Neil Zhenqiang Gong; Pin-Yu Chen; Bo Li; Dawn Song; Xiangliang Zhang
  Generative Foundation Models (GenFMs) have emerged as transformative tools.
However, their widespread adoption raises critical concerns regarding
trustworthiness across dimensions. This paper presents a comprehensive
framework to address these challenges through three key contributions. First,
we systematically review global AI governance laws and policies from
governments and regulatory bodies, as well as industry practices and standards.
Based on this analysis, we propose a set of guiding principles for GenFMs,
developed through extensive multidisciplinary collaboration that integrates
technical, ethical, legal, and societal perspectives. Second, we introduce
TrustGen, the first dynamic benchmarking platform designed to evaluate
trustworthiness across multiple dimensions and model types, including
text-to-image, large language, and vision-language models. TrustGen leverages
modular components--metadata curation, test case generation, and contextual
variation--to enable adaptive and iterative assessments, overcoming the
limitations of static evaluation methods. Using TrustGen, we reveal significant
progress in trustworthiness while identifying persistent challenges. Finally,
we provide an in-depth discussion of the challenges and future directions for
trustworthy GenFMs, which reveals the complex, evolving nature of
trustworthiness, highlighting the nuanced trade-offs between utility and
trustworthiness, and consideration for various downstream applications,
identifying persistent challenges and providing a strategic roadmap for future
research. This work establishes a holistic framework for advancing
trustworthiness in GenAI, paving the way for safer and more responsible
integration of GenFMs into critical applications. To facilitate advancement in
the community, we release the toolkit for dynamic evaluation.


http://arxiv.org/abs/2502.14298
Generalization Certificates for Adversarially Robust Bayesian Linear Regression. (22%)
Mahalakshmi Sabanayagam; Russell Tsuchida; Cheng Soon Ong; Debarghya Ghoshdastidar
  Adversarial robustness of machine learning models is critical to ensuring
reliable performance under data perturbations. Recent progress has been on
point estimators, and this paper considers distributional predictors. First,
using the link between exponential families and Bregman divergences, we
formulate an adversarial Bregman divergence loss as an adversarial negative
log-likelihood. Using the geometric properties of Bregman divergences, we
compute the adversarial perturbation for such models in closed-form. Second,
under such losses, we introduce \emph{adversarially robust posteriors}, by
exploiting the optimization-centric view of generalized Bayesian inference.
Third, we derive the \emph{first} rigorous generalization certificates in the
context of an adversarial extension of Bayesian linear regression by leveraging
the PAC-Bayesian framework. Finally, experiments on real and synthetic datasets
demonstrate the superior robustness of the derived adversarially robust
posterior over Bayes posterior, and also validate our theoretical guarantees.


http://arxiv.org/abs/2502.14572
Factor Graph-based Interpretable Neural Networks. (11%)
Yicong Li; Kuanjiu Zhou; Shuo Yu; Qiang Zhang; Renqiang Luo; Xiaodong Li; Feng Xia
  Comprehensible neural network explanations are foundations for a better
understanding of decisions, especially when the input data are infused with
malicious perturbations. Existing solutions generally mitigate the impact of
perturbations through adversarial training, yet they fail to generate
comprehensible explanations under unknown perturbations. To address this
challenge, we propose AGAIN, a fActor GrAph-based Interpretable neural Network,
which is capable of generating comprehensible explanations under unknown
perturbations. Instead of retraining like previous solutions, the proposed
AGAIN directly integrates logical rules by which logical errors in explanations
are identified and rectified during inference. Specifically, we construct the
factor graph to express logical rules between explanations and categories. By
treating logical rules as exogenous knowledge, AGAIN can identify
incomprehensible explanations that violate real-world logic. Furthermore, we
propose an interactive intervention switch strategy rectifying explanations
based on the logical guidance from the factor graph without learning
perturbations, which overcomes the inherent limitation of adversarial
training-based methods in defending only against known perturbations.
Additionally, we theoretically demonstrate the effectiveness of employing
factor graph by proving that the comprehensibility of explanations is strongly
correlated with factor graph. Extensive experiments are conducted on three
datasets and experimental results illustrate the superior performance of AGAIN
compared to state-of-the-art baselines.


http://arxiv.org/abs/2502.14370
PPO-MI: Efficient Black-Box Model Inversion via Proximal Policy Optimization. (10%)
Xinpeng Shou
  Model inversion attacks pose a significant privacy risk by attempting to
reconstruct private training data from trained models. Most of the existing
methods either depend on gradient estimation or require white-box access to
model parameters, which limits their applicability in practical scenarios. In
this paper, we propose PPO-MI, a novel reinforcement learning-based framework
for black-box model inversion attacks. Our approach formulates the inversion
task as a Markov Decision Process, where an agent navigates the latent space of
a generative model to reconstruct private training samples using only model
predictions. By employing Proximal Policy Optimization (PPO) with a
momentum-based state transition mechanism, along with a reward function
balancing prediction accuracy and exploration, PPO-MI ensures efficient latent
space exploration and high query efficiency. We conduct extensive experiments
illustrates that PPO-MI outperforms the existing methods while require less
attack knowledge, and it is robust across various model architectures and
datasets. These results underline its effectiveness and generalizability in
practical black-box scenarios, raising important considerations for the privacy
vulnerabilities of deployed machine learning models.


http://arxiv.org/abs/2502.14833
Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide. (10%)
Xingyu Zhao
  Deep learning (DL) has demonstrated significant potential across various
safety-critical applications, yet ensuring its robustness remains a key
challenge. While adversarial robustness has been extensively studied in
worst-case scenarios, probabilistic robustness (PR) offers a more practical
perspective by quantifying the likelihood of failures under stochastic
perturbations. This paper provides a concise yet comprehensive overview of PR,
covering its formal definitions, evaluation and enhancement methods. We
introduce a reformulated ''min-max'' optimisation framework for adversarial
training specifically designed to improve PR. Furthermore, we explore the
integration of PR verification evidence into system-level safety assurance,
addressing challenges in translating DL model-level robustness to system-level
claims. Finally, we highlight open research questions, including benchmarking
PR evaluation methods, extending PR to generative AI tasks, and developing
rigorous methodologies and case studies for system-level integration.


http://arxiv.org/abs/2502.15020
MACPruning: Dynamic Operation Pruning to Mitigate Side-Channel DNN Model Extraction. (4%)
Ruyi Ding; Cheng Gongye; Davis Ranney; Aidong Adam Ding; Yunsi Fei
  As deep learning gains popularity, edge IoT devices have seen proliferating
deployment of pre-trained Deep Neural Network (DNN) models. These DNNs
represent valuable intellectual property and face significant confidentiality
threats from side-channel analysis (SCA), particularly non-invasive
Differential Electromagnetic (EM) Analysis (DEMA), which retrieves individual
model parameters from EM traces collected during model inference. Traditional
SCA mitigation methods, such as masking and shuffling, can still be applied to
DNN inference, but will incur significant performance degradation due to the
large volume of operations and parameters. Based on the insight that DNN models
have high redundancy and are robust to input variation, we introduce
MACPruning, a novel lightweight defense against DEMA-based parameter extraction
attacks, exploiting specific characteristics of DNN execution. The design
principle of MACPruning is to randomly deactivate input pixels and prune the
operations (typically multiply-accumulate-MAC) on those pixels. The technique
removes certain leakages and overall redistributes weight-dependent EM leakages
temporally, and thus effectively mitigates DEMA. To maintain DNN performance,
we propose an importance-aware pixel map that preserves critical input pixels,
keeping randomness in the defense while minimizing its impact on DNN
performance due to operation pruning. We conduct a comprehensive security
analysis of MACPruning on various datasets for DNNs on edge devices. Our
evaluations demonstrate that MACPruning effectively reduces EM leakages with
minimal impact on the model accuracy and negligible computational overhead.


http://arxiv.org/abs/2502.14828
Fundamental Limitations in Defending LLM Finetuning APIs. (3%)
Xander Davies; Eric Winsor; Tomek Korbak; Alexandra Souly; Robert Kirk; Witt Christian Schroeder de; Yarin Gal
  LLM developers have imposed technical interventions to prevent fine-tuning
misuse attacks, attacks where adversaries evade safeguards by fine-tuning the
model using a public API. Previous work has established several successful
attacks against specific fine-tuning API defences. In this work, we show that
defences of fine-tuning APIs that seek to detect individual harmful training or
inference samples ('pointwise' detection) are fundamentally limited in their
ability to prevent fine-tuning attacks. We construct 'pointwise-undetectable'
attacks that repurpose entropy in benign model outputs (e.g. semantic or
syntactic variations) to covertly transmit dangerous knowledge. Our attacks are
composed solely of unsuspicious benign samples that can be collected from the
model before fine-tuning, meaning training and inference samples are all
individually benign and low-perplexity. We test our attacks against the OpenAI
fine-tuning API, finding they succeed in eliciting answers to harmful
multiple-choice questions, and that they evade an enhanced monitoring system we
design that successfully detects other fine-tuning attacks. We encourage the
community to develop defences that tackle the fundamental limitations we
uncover in pointwise fine-tuning API defences.


http://arxiv.org/abs/2502.14416
Reliable Explainability of Deep Learning Spatial-Spectral Classifiers for Improved Semantic Segmentation in Autonomous Driving. (1%)
Jon Gutiérrez-Zaballa; Koldo Basterretxea; Javier Echanobe
  Integrating hyperspectral imagery (HSI) with deep neural networks (DNNs) can
strengthen the accuracy of intelligent vision systems by combining spectral and
spatial information, which is useful for tasks like semantic segmentation in
autonomous driving. To advance research in such safety-critical systems,
determining the precise contribution of spectral information to complex DNNs'
output is needed. To address this, several saliency methods, such as class
activation maps (CAM), have been proposed primarily for image classification.
However, recent studies have raised concerns regarding their reliability. In
this paper, we address their limitations and propose an alternative approach by
leveraging the data provided by activations and weights from relevant DNN
layers to better capture the relationship between input features and
predictions. The study aims to assess the superior performance of HSI compared
to 3-channel and single-channel DNNs. We also address the influence of spectral
signature normalization for enhancing DNN robustness in real-world driving
conditions.


http://arxiv.org/abs/2502.15041
Benchmarking Android Malware Detection: Rethinking the Role of Traditional and Deep Learning Models. (1%)
Guojun Liu; Doina Caragea; Xinming Ou; Sankardas Roy
  Android malware detection has been extensively studied using both traditional
machine learning (ML) and deep learning (DL) approaches. While many
state-of-the-art detection models, particularly those based on DL, claim
superior performance, they often rely on limited comparisons, lacking
comprehensive benchmarking against traditional ML models across diverse
datasets. This raises concerns about the robustness of DL-based approaches'
performance and the potential oversight of simpler, more efficient ML models.
In this paper, we conduct a systematic evaluation of Android malware detection
models across four datasets: three recently published, publicly available
datasets and a large-scale dataset we systematically collected. We implement a
range of traditional ML models, including Random Forests (RF) and CatBoost,
alongside advanced DL models such as Capsule Graph Neural Networks (CapsGNN),
BERT-based models, and ExcelFormer based models. Our results reveal that while
advanced DL models can achieve strong performance, they are often compared
against an insufficient number of traditional ML baselines. In many cases,
simpler and more computationally efficient ML models achieve comparable or even
superior performance. These findings highlight the need for rigorous
benchmarking in Android malware detection research. We encourage future studies
to conduct more comprehensive benchmarking comparisons between traditional and
advanced models to ensure a more accurate assessment of detection capabilities.
To facilitate further research, we provide access to our dataset, including app
IDs, hash values, and labels.


http://arxiv.org/abs/2502.13527
Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking. (15%)
Yanzeng Li; Yunfan Xiong; Jialun Zhong; Jinchao Zhang; Jie Zhou; Lei Zou
  The rise of Large Language Models (LLMs) has led to significant applications
but also introduced serious security threats, particularly from jailbreak
attacks that manipulate output generation. These attacks utilize prompt
engineering and logit manipulation to steer models toward harmful content,
prompting LLM providers to implement filtering and safety alignment strategies.
We investigate LLMs' safety mechanisms and their recent applications, revealing
a new threat model targeting structured output interfaces, which enable
attackers to manipulate the inner logit during LLM generation, requiring only
API access permissions. To demonstrate this threat model, we introduce a
black-box attack framework called AttackPrefixTree (APT). APT exploits
structured output interfaces to dynamically construct attack patterns. By
leveraging prefixes of models' safety refusal response and latent harmful
outputs, APT effectively bypasses safety measures. Experiments on benchmark
datasets indicate that this approach achieves higher attack success rate than
existing methods. This work highlights the urgent need for LLM providers to
enhance security protocols to address vulnerabilities arising from the
interaction between safety patterns and structured outputs.


http://arxiv.org/abs/2502.13459
Poisoned Source Code Detection in Code Models. (15%)
Ehab Ghannoum; Mohammad Ghafari
  Deep learning models have gained popularity for conducting various tasks
involving source code. However, their black-box nature raises concerns about
potential risks. One such risk is a poisoning attack, where an attacker
intentionally contaminates the training set with malicious samples to mislead
the model's predictions in specific scenarios. To protect source code models
from poisoning attacks, we introduce CodeGarrison (CG), a hybrid deep-learning
model that relies on code embeddings to identify poisoned code samples. We
evaluated CG against the state-of-the-art technique ONION for detecting
poisoned samples generated by DAMP, MHM, ALERT, as well as a novel poisoning
technique named CodeFooler. Results showed that CG significantly outperformed
ONION with an accuracy of 93.5%. We also tested CG's robustness against unknown
attacks, achieving an average accuracy of 85.6% in identifying poisoned samples
across the four attacks mentioned above.


http://arxiv.org/abs/2502.13641
SLAMSpoof: Practical LiDAR Spoofing Attacks on Localization Systems Guided by Scan Matching Vulnerability Analysis. (8%)
Rokuto Nagata; Kenji Koide; Yuki Hayakawa; Ryo Suzuki; Kazuma Ikeda; Ozora Sako; Qi Alfred Chen; Takami Sato; Kentaro Yoshioka
  Accurate localization is essential for enabling modern full self-driving
services. These services heavily rely on map-based traffic information to
reduce uncertainties in recognizing lane shapes, traffic light locations, and
traffic signs. Achieving this level of reliance on map information requires
centimeter-level localization accuracy, which is currently only achievable with
LiDAR sensors. However, LiDAR is known to be vulnerable to spoofing attacks
that emit malicious lasers against LiDAR to overwrite its measurements. Once
localization is compromised, the attack could lead the victim off roads or make
them ignore traffic lights. Motivated by these serious safety implications, we
design SLAMSpoof, the first practical LiDAR spoofing attack on localization
systems for self-driving to assess the actual attack significance on autonomous
vehicles. SLAMSpoof can effectively find the effective attack location based on
our scan matching vulnerability score (SMVS), a point-wise metric representing
the potential vulnerability to spoofing attacks. To evaluate the effectiveness
of the attack, we conduct real-world experiments on ground vehicles and confirm
its high capability in real-world scenarios, inducing position errors of
$\geq$4.2 meters (more than typical lane width) for all 3 popular LiDAR-based
localization algorithms. We finally discuss the potential countermeasures of
this attack. Code is available at https://github.com/Keio-CSG/slamspoof


http://arxiv.org/abs/2502.14001
Towards a perturbation-based explanation for medical AI as differentiable programs. (2%)
Takeshi Abe; Yoshiyuki Asai
  Recent advancement in machine learning algorithms reaches a point where
medical devices can be equipped with artificial intelligence (AI) models for
diagnostic support and routine automation in clinical settings. In medicine and
healthcare, there is a particular demand for sufficient and objective
explainability of the outcome generated by AI models. However, AI models are
generally considered as black boxes due to their complexity, and the
computational process leading to their response is often opaque. Although
several methods have been proposed to explain the behavior of models by
evaluating the importance of each feature in discrimination and prediction,
they may suffer from biases and opacities arising from the scale and sampling
protocol of the dataset used for training or testing. To overcome the
shortcomings of existing methods, we explore an alternative approach to provide
an objective explanation of AI models that can be defined independently of the
learning process and does not require additional data. As a preliminary study
for this direction of research, this work examines a numerical availability of
the Jacobian matrix of deep learning models that measures how stably a model
responses against small perturbations added to the input. The indicator, if
available, are calculated from a trained AI model for a given target input.
This is a first step towards a perturbation-based explanation, which will
assist medical practitioners in understanding and interpreting the response of
the AI model in its clinical application.


http://arxiv.org/abs/2502.14146
Efficient and Optimal Policy Gradient Algorithm for Corrupted Multi-armed Bandits. (1%)
Jiayuan Liu; Siwei Wang; Zhixuan Fang
  In this paper, we consider the stochastic multi-armed bandits problem with
adversarial corruptions, where the random rewards of the arms are partially
modified by an adversary to fool the algorithm. We apply the policy gradient
algorithm SAMBA to this setting, and show that it is computationally efficient,
and achieves a state-of-the-art $O(K\log T/\Delta) + O(C/\Delta)$ regret upper
bound, where $K$ is the number of arms, $C$ is the unknown corruption level,
$\Delta$ is the minimum expected reward gap between the best arm and other
ones, and $T$ is the time horizon. Compared with the best existing efficient
algorithm (e.g., CBARBAR), whose regret upper bound is $O(K\log^2 T/\Delta) +
O(C)$, we show that SAMBA reduces one $\log T$ factor in the regret bound,
while maintaining the corruption-dependent term to be linear with $C$. This is
indeed asymptotically optimal. We also conduct simulations to demonstrate the
effectiveness of SAMBA, and the results show that SAMBA outperforms existing
baselines.


http://arxiv.org/abs/2502.13593
Toward Robust Non-Transferable Learning: A Survey and Benchmark. (1%)
Ziming Hong; Yongli Xiang; Tongliang Liu
  Over the past decades, researchers have primarily focused on improving the
generalization abilities of models, with limited attention given to regulating
such generalization. However, the ability of models to generalize to unintended
data (e.g., harmful or unauthorized data) can be exploited by malicious
adversaries in unforeseen ways, potentially resulting in violations of model
ethics. Non-transferable learning (NTL), a task aimed at reshaping the
generalization abilities of deep learning models, was proposed to address these
challenges. While numerous methods have been proposed in this field, a
comprehensive review of existing progress and a thorough analysis of current
limitations remain lacking. In this paper, we bridge this gap by presenting the
first comprehensive survey on NTL and introducing NTLBench, the first benchmark
to evaluate NTL performance and robustness within a unified framework.
Specifically, we first introduce the task settings, general framework, and
criteria of NTL, followed by a summary of NTL approaches. Furthermore, we
emphasize the often-overlooked issue of robustness against various attacks that
can destroy the non-transferable mechanism established by NTL. Experiments
conducted via NTLBench verify the limitations of existing NTL methods in
robustness. Finally, we discuss the practical applications of NTL, along with
its future directions and associated challenges.


http://arxiv.org/abs/2502.12734
Iron Sharpens Iron: Defending Against Attacks in Machine-Generated Text Detection with Adversarial Training. (99%)
Yuanfan Li; Zhaohan Zhang; Chengzhengxu Li; Chao Shen; Xiaoming Liu
  Machine-generated Text (MGT) detection is crucial for regulating and
attributing online texts. While the existing MGT detectors achieve strong
performance, they remain vulnerable to simple perturbations and adversarial
attacks. To build an effective defense against malicious perturbations, we view
MGT detection from a threat modeling perspective, that is, analyzing the
model's vulnerability from an adversary's point of view and exploring effective
mitigations. To this end, we introduce an adversarial framework for training a
robust MGT detector, named GREedy Adversary PromoTed DefendER (GREATER). The
GREATER consists of two key components: an adversary GREATER-A and a detector
GREATER-D. The GREATER-D learns to defend against the adversarial attack from
GREATER-A and generalizes the defense to other attacks. GREATER-A identifies
and perturbs the critical tokens in embedding space, along with greedy search
and pruning to generate stealthy and disruptive adversarial examples. Besides,
we update the GREATER-A and GREATER-D synchronously, encouraging the GREATER-D
to generalize its defense to different attacks and varying attack intensities.
Our experimental results across 10 text perturbation strategies and 6
adversarial attacks show that our GREATER-D reduces the Attack Success Rate
(ASR) by 0.67% compared with SOTA defense methods while our GREATER-A is
demonstrated to be more effective and efficient than SOTA attack approaches.
Codes and dataset are available in https://github.com/Liyuuuu111/GREATER.


http://arxiv.org/abs/2502.12958
Preventing the Popular Item Embedding Based Attack in Federated Recommendations. (73%)
Jun Zhang; Huan Li; Dazhong Rong; Yan Zhao; Ke Chen; Lidan Shou
  Privacy concerns have led to the rise of federated recommender systems (FRS),
which can create personalized models across distributed clients. However, FRS
is vulnerable to poisoning attacks, where malicious users manipulate gradients
to promote their target items intentionally. Existing attacks against FRS have
limitations, as they depend on specific models and prior knowledge, restricting
their real-world applicability. In our exploration of practical FRS
vulnerabilities, we devise a model-agnostic and prior-knowledge-free attack,
named PIECK (Popular Item Embedding based Attack). The core module of PIECK is
popular item mining, which leverages embedding changes during FRS training to
effectively identify the popular items. Built upon the core module, PIECK
branches into two diverse solutions: The PIECKIPE solution employs an item
popularity enhancement module, which aligns the embeddings of targeted items
with the mined popular items to increase item exposure. The PIECKUEA further
enhances the robustness of the attack by using a user embedding approximation
module, which approximates private user embeddings using mined popular items.
Upon identifying PIECK, we evaluate existing federated defense methods and find
them ineffective against PIECK, as poisonous gradients inevitably overwhelm the
cold target items. We then propose a novel defense method by introducing two
regularization terms during user training, which constrain item popularity
enhancement and user embedding approximation while preserving FRS performance.
We evaluate PIECK and its defense across two base models, three real datasets,
four top-tier attacks, and six general defense methods, affirming the efficacy
of both PIECK and its defense.


http://arxiv.org/abs/2502.13141
UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models. (10%)
Huawei Lin; Yingjie Lao; Tong Geng; Tan Yu; Weijie Zhao
  Large Language Models (LLMs) are vulnerable to attacks like prompt injection,
backdoor attacks, and adversarial attacks, which manipulate prompts or models
to generate harmful outputs. In this paper, departing from traditional deep
learning attack paradigms, we explore their intrinsic relationship and
collectively term them Prompt Trigger Attacks (PTA). This raises a key
question: Can we determine if a prompt is benign or poisoned? To address this,
we propose UniGuardian, the first unified defense mechanism designed to detect
prompt injection, backdoor attacks, and adversarial attacks in LLMs.
Additionally, we introduce a single-forward strategy to optimize the detection
pipeline, enabling simultaneous attack detection and text generation within a
single forward pass. Our experiments confirm that UniGuardian accurately and
efficiently identifies malicious prompts in LLMs.


http://arxiv.org/abs/2502.12575
DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent. (8%)
Pengyu Zhu; Zhenhong Zhou; Yuanhe Zhang; Shilinlu Yan; Kun Wang; Sen Su
  As LLM-based agents become increasingly prevalent, backdoors can be implanted
into agents through user queries or environment feedback, raising critical
concerns regarding safety vulnerabilities. However, backdoor attacks are
typically detectable by safety audits that analyze the reasoning process of
agents. To this end, we propose a novel backdoor implantation strategy called
\textbf{Dynamically Encrypted Multi-Backdoor Implantation Attack}.
Specifically, we introduce dynamic encryption, which maps the backdoor into
benign content, effectively circumventing safety audits. To enhance
stealthiness, we further decompose the backdoor into multiple sub-backdoor
fragments. Based on these advancements, backdoors are allowed to bypass safety
audits significantly. Additionally, we present AgentBackdoorEval, a dataset
designed for the comprehensive evaluation of agent backdoor attacks.
Experimental results across multiple datasets demonstrate that our method
achieves an attack success rate nearing 100\% while maintaining a detection
rate of 0\%, illustrating its effectiveness in evading safety audits. Our
findings highlight the limitations of existing safety mechanisms in detecting
advanced attacks, underscoring the urgent need for more robust defenses against
backdoor threats. Code and data are available at
https://github.com/whfeLingYu/DemonAgent.


http://arxiv.org/abs/2502.12659
The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1. (8%)
Kaiwen Zhou; Chengzhi Liu; Xuandong Zhao; Shreedhar Jangam; Jayanth Srinivasa; Gaowen Liu; Dawn Song; Xin Eric Wang
  The rapid development of large reasoning models, such as OpenAI-o3 and
DeepSeek-R1, has led to significant improvements in complex reasoning over
non-reasoning large language models~(LLMs). However, their enhanced
capabilities, combined with the open-source access of models like DeepSeek-R1,
raise serious safety concerns, particularly regarding their potential for
misuse. In this work, we present a comprehensive safety assessment of these
reasoning models, leveraging established safety benchmarks to evaluate their
compliance with safety regulations. Furthermore, we investigate their
susceptibility to adversarial attacks, such as jailbreaking and prompt
injection, to assess their robustness in real-world applications. Through our
multi-faceted analysis, we uncover four key findings: (1) There is a
significant safety gap between the open-source R1 models and the o3-mini model,
on both safety benchmark and attack, suggesting more safety effort on R1 is
needed. (2) The distilled reasoning model shows poorer safety performance
compared to its safety-aligned base models. (3) The stronger the model's
reasoning ability, the greater the potential harm it may cause when answering
unsafe questions. (4) The thinking process in R1 models pose greater safety
concerns than their final answers. Our study provides insights into the
security implications of reasoning models and highlights the need for further
advancements in R1 models' safety to close the gap.


http://arxiv.org/abs/2502.12970
Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking. (3%)
Junda Zhu; Lingyong Yan; Shuaiqiang Wang; Dawei Yin; Lei Sha
  Large Reasoning Models (LRMs) have demonstrated impressive performances
across diverse domains. However, how safety of Large Language Models (LLMs)
benefits from enhanced reasoning capabilities against jailbreak queries remains
unexplored. To bridge this gap, in this paper, we propose Reasoning-to-Defend
(R2D), a novel training paradigm that integrates a safety-aware reasoning
mechanism into LLMs' generation. This enables self-evaluation at each step of
the reasoning process, forming safety pivot tokens as indicators of the safety
status of responses. Furthermore, in order to improve the accuracy of
predicting pivot tokens, we propose Contrastive Pivot Optimization (CPO), which
enhances the model's perception of the safety status of given dialogues. LLMs
dynamically adjust their response strategies during reasoning, significantly
enhancing their safety capabilities defending jailbreak attacks. Extensive
experiments demonstrate that R2D effectively mitigates various attacks and
improves overall safety, while maintaining the original performances. This
highlights the substantial potential of safety-aware reasoning in improving
robustness of LRMs and LLMs against various jailbreaks.


http://arxiv.org/abs/2502.12576
A Fuzzy Evaluation of Sentence Encoders on Grooming Risk Classification. (2%)
Geetanjali Bihani; Julia Rayz
  With the advent of social media, children are becoming increasingly
vulnerable to the risk of grooming in online settings. Detecting grooming
instances in an online conversation poses a significant challenge as the
interactions are not necessarily sexually explicit, since the predators take
time to build trust and a relationship with their victim. Moreover, predators
evade detection using indirect and coded language. While previous studies have
fine-tuned Transformers to automatically identify grooming in chat
conversations, they overlook the impact of coded and indirect language on model
predictions, and how these align with human perceptions of grooming. In this
paper, we address this gap and evaluate bi-encoders on the task of classifying
different degrees of grooming risk in chat contexts, for three different
participant groups, i.e. law enforcement officers, real victims, and decoys.
Using a fuzzy-theoretic framework, we map human assessments of grooming
behaviors to estimate the actual degree of grooming risk. Our analysis reveals
that fine-tuned models fail to tag instances where the predator uses indirect
speech pathways and coded language to evade detection. Further, we find that
such instances are characterized by a higher presence of out-of-vocabulary
(OOV) words in samples, causing the model to misclassify. Our findings
highlight the need for more robust models to identify coded language from noisy
chat inputs in grooming contexts.


http://arxiv.org/abs/2502.12893
H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking. (1%)
Martin Kuo; Jianyi Zhang; Aolin Ding; Qinsi Wang; Louis DiValentin; Yujia Bao; Wei Wei; Hai Li; Yiran Chen
  Large Reasoning Models (LRMs) have recently extended their powerful reasoning
capabilities to safety checks-using chain-of-thought reasoning to decide
whether a request should be answered. While this new approach offers a
promising route for balancing model utility and safety, its robustness remains
underexplored. To address this gap, we introduce Malicious-Educator, a
benchmark that disguises extremely dangerous or malicious requests beneath
seemingly legitimate educational prompts. Our experiments reveal severe
security flaws in popular commercial-grade LRMs, including OpenAI o1/o3,
DeepSeek-R1, and Gemini 2.0 Flash Thinking. For instance, although OpenAI's o1
model initially maintains a high refusal rate of about 98%, subsequent model
updates significantly compromise its safety; and attackers can easily extract
criminal strategies from DeepSeek-R1 and Gemini 2.0 Flash Thinking without any
additional tricks. To further highlight these vulnerabilities, we propose
Hijacking Chain-of-Thought (H-CoT), a universal and transferable attack method
that leverages the model's own displayed intermediate reasoning to jailbreak
its safety reasoning mechanism. Under H-CoT, refusal rates sharply
decline-dropping from 98% to below 2%-and, in some instances, even transform
initially cautious tones into ones that are willing to provide harmful content.
We hope these findings underscore the urgent need for more robust safety
mechanisms to preserve the benefits of advanced reasoning capabilities without
compromising ethical standards.


http://arxiv.org/abs/2502.13024
Fragility-aware Classification for Understanding Risk and Improving Generalization. (1%)
Chen Yang; Zheng Cui; Daniel Zhuoyu Long; Jin Qi; Ruohan Zhan
  Classification models play a critical role in data-driven decision-making
applications such as medical diagnosis, user profiling, recommendation systems,
and default detection. Traditional performance metrics, such as accuracy, focus
on overall error rates but fail to account for the confidence of incorrect
predictions, thereby overlooking the risk of confident misjudgments. This risk
is particularly significant in cost-sensitive and safety-critical domains like
medical diagnosis and autonomous driving, where overconfident false predictions
may cause severe consequences. To address this issue, we introduce the
Fragility Index (FI), a novel metric that evaluates classification performance
from a risk-averse perspective by explicitly capturing the tail risk of
confident misjudgments. To enhance generalizability, we define FI within the
robust satisficing (RS) framework, incorporating data uncertainty. We further
develop a model training approach that optimizes FI while maintaining
tractability for common loss functions. Specifically, we derive exact
reformulations for cross-entropy loss, hinge-type loss, and Lipschitz loss, and
extend the approach to deep learning models. Through synthetic experiments and
real-world medical diagnosis tasks, we demonstrate that FI effectively
identifies misjudgment risk and FI-based training improves model robustness and
generalizability. Finally, we extend our framework to deep neural network
training, further validating its effectiveness in enhancing deep learning
models.


http://arxiv.org/abs/2502.13191
On the Privacy Risks of Spiking Neural Networks: A Membership Inference Analysis. (1%)
Junyi Guan; Abhijith Sharma; Chong Tian; Salem Lahlou
  Spiking Neural Networks (SNNs) are increasingly explored for their energy
efficiency and robustness in real-world applications, yet their privacy risks
remain largely unexamined. In this work, we investigate the susceptibility of
SNNs to Membership Inference Attacks (MIAs) -- a major privacy threat where an
adversary attempts to determine whether a given sample was part of the training
dataset. While prior work suggests that SNNs may offer inherent robustness due
to their discrete, event-driven nature, we find that its resilience diminishes
as latency (T) increases. Furthermore, we introduce an input dropout strategy
under black box setting, that significantly enhances membership inference in
SNNs. Our findings challenge the assumption that SNNs are inherently more
secure, and even though they are expected to be better, our results reveal that
SNNs exhibit privacy vulnerabilities that are equally comparable to Artificial
Neural Networks (ANNs). Our code is available at
https://anonymous.4open.science/r/MIA_SNN-3610.


http://arxiv.org/abs/2502.11858
Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives. (96%)
Zeliang Zhang; Susan Liang; Daiki Shimada; Chenliang Xu
  While audio-visual learning equips models with a richer understanding of the
real world by leveraging multiple sensory modalities, this integration also
introduces new vulnerabilities to adversarial attacks.
  In this paper, we present a comprehensive study of the adversarial robustness
of audio-visual models, considering both temporal and modality-specific
vulnerabilities. We propose two powerful adversarial attacks: 1) a temporal
invariance attack that exploits the inherent temporal redundancy across
consecutive time segments and 2) a modality misalignment attack that introduces
incongruence between the audio and visual modalities. These attacks are
designed to thoroughly assess the robustness of audio-visual models against
diverse threats. Furthermore, to defend against such attacks, we introduce a
novel audio-visual adversarial training framework. This framework addresses key
challenges in vanilla adversarial training by incorporating efficient
adversarial perturbation crafting tailored to multi-modal data and an
adversarial curriculum strategy. Extensive experiments in the Kinetics-Sounds
dataset demonstrate that our proposed temporal and modality-based attacks in
degrading model performance can achieve state-of-the-art performance, while our
adversarial training defense largely improves the adversarial robustness as
well as the adversarial training efficiency.


http://arxiv.org/abs/2502.13175
Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks. (78%)
Wenpeng Xing; Minghao Li; Mohan Li; Meng Han
  Embodied AI systems, including robots and autonomous vehicles, are
increasingly integrated into real-world applications, where they encounter a
range of vulnerabilities stemming from both environmental and system-level
factors. These vulnerabilities manifest through sensor spoofing, adversarial
attacks, and failures in task and motion planning, posing significant
challenges to robustness and safety. Despite the growing body of research,
existing reviews rarely focus specifically on the unique safety and security
challenges of embodied AI systems. Most prior work either addresses general AI
vulnerabilities or focuses on isolated aspects, lacking a dedicated and unified
framework tailored to embodied AI. This survey fills this critical gap by: (1)
categorizing vulnerabilities specific to embodied AI into exogenous (e.g.,
physical attacks, cybersecurity threats) and endogenous (e.g., sensor failures,
software flaws) origins; (2) systematically analyzing adversarial attack
paradigms unique to embodied AI, with a focus on their impact on perception,
decision-making, and embodied interaction; (3) investigating attack vectors
targeting large vision-language models (LVLMs) and large language models (LLMs)
within embodied systems, such as jailbreak attacks and instruction
misinterpretation; (4) evaluating robustness challenges in algorithms for
embodied perception, decision-making, and task planning; and (5) proposing
targeted strategies to enhance the safety and reliability of embodied AI
systems. By integrating these dimensions, we provide a comprehensive framework
for understanding the interplay between vulnerabilities and safety in embodied
AI.


http://arxiv.org/abs/2502.11455
Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training. (67%)
Fenghua Weng; Jian Lou; Jun Feng; Minlie Huang; Wenjie Wang
  Safety alignment is critical in pre-training large language models (LLMs) to
generate responses aligned with human values and refuse harmful queries. Unlike
LLM, the current safety alignment of VLMs is often achieved with post-hoc
safety fine-tuning. However, these methods are less effective to white-box
attacks. To address this, we propose $\textit{Adversary-aware DPO (ADPO)}$, a
novel training framework that explicitly considers adversarial.
$\textit{Adversary-aware DPO (ADPO)}$ integrates adversarial training into DPO
to enhance the safety alignment of VLMs under worst-case adversarial
perturbations. $\textit{ADPO}$ introduces two key components: (1) an
adversarial-trained reference model that generates human-preferred responses
under worst-case perturbations, and (2) an adversarial-aware DPO loss that
generates winner-loser pairs accounting for adversarial distortions. By
combining these innovations, $\textit{ADPO}$ ensures that VLMs remain robust
and reliable even in the presence of sophisticated jailbreak attacks. Extensive
experiments demonstrate that $\textit{ADPO}$ outperforms baselines in the
safety alignment and general utility of VLMs.


http://arxiv.org/abs/2502.11725
Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics. (50%)
Francesco Croce; Christian Schlarmann; Naman Deep Singh; Matthias Hein
  Measuring perceptual similarity is a key tool in computer vision. In recent
years perceptual metrics based on features extracted from neural networks with
large and diverse training sets, e.g. CLIP, have become popular. At the same
time, the metrics extracted from features of neural networks are not
adversarially robust. In this paper we show that adversarially robust CLIP
models, called R-CLIP$_\textrm{F}$, obtained by unsupervised adversarial
fine-tuning induce a better and adversarially robust perceptual metric that
outperforms existing metrics in a zero-shot setting, and further matches the
performance of state-of-the-art metrics while being robust after fine-tuning.
Moreover, our perceptual metric achieves strong performance on related tasks
such as robust image-to-image retrieval, which becomes especially relevant when
applied to "Not Safe for Work" (NSFW) content detection and dataset filtering.
While standard perceptual metrics can be easily attacked by a small
perturbation completely degrading NSFW detection, our robust perceptual metric
maintains high accuracy under an attack while having similar performance for
unperturbed images. Finally, perceptual metrics induced by robust CLIP models
have higher interpretability: feature inversion can show which images are
considered similar, while text inversion can find what images are associated to
a given prompt. This also allows us to visualize the very rich visual concepts
learned by a CLIP model, including memorized persons, paintings and complex
queries.


http://arxiv.org/abs/2502.12292
Independence Tests for Language Models. (13%)
Sally Zhu; Ahmed Ahmed; Rohith Kuditipudi; Percy Liang
  We consider the following problem: given the weights of two models, can we
test whether they were trained independently -- i.e., from independent random
initializations? We consider two settings: constrained and unconstrained. In
the constrained setting, we make assumptions about model architecture and
training and propose a family of statistical tests that yield exact p-values
with respect to the null hypothesis that the models are trained from
independent random initializations. These p-values are valid regardless of the
composition of either model's training data; we compute them by simulating
exchangeable copies of each model under our assumptions and comparing various
similarity measures of weights and activations between the original two models
versus these copies. We report the p-values from these tests on pairs of 21
open-weight models (210 total pairs) and correctly identify all pairs of
non-independent models. Our tests remain effective even if one model was
fine-tuned for many tokens. In the unconstrained setting, where we make no
assumptions about training procedures, can change model architecture, and allow
for adversarial evasion attacks, the previous tests no longer work. Instead, we
propose a new test which matches hidden activations between two models, and
which is robust to adversarial transformations and to changes in model
architecture. The test can also do localized testing: identifying specific
non-independent components of models. Though we no longer obtain exact p-values
from this, empirically we find it behaves as one and reliably identifies
non-independent models. Notably, we can use the test to identify specific parts
of one model that are derived from another (e.g., how Llama 3.1-8B was pruned
to initialize Llama 3.2-3B, or shared layers between Mistral-7B and
StripedHyena-7B), and it is even robust to retraining individual layers of
either model from scratch.


http://arxiv.org/abs/2502.11687
ReVeil: Unconstrained Concealed Backdoor Attack on Deep Neural Networks using Machine Unlearning. (11%)
Manaar Alam; Hithem Lamri; Michail Maniatakos
  Backdoor attacks embed hidden functionalities in deep neural networks (DNN),
triggering malicious behavior with specific inputs. Advanced defenses monitor
anomalous DNN inferences to detect such attacks. However, concealed backdoors
evade detection by maintaining a low pre-deployment attack success rate (ASR)
and restoring high ASR post-deployment via machine unlearning. Existing
concealed backdoors are often constrained by requiring white-box or black-box
access or auxiliary data, limiting their practicality when such access or data
is unavailable. This paper introduces ReVeil, a concealed backdoor attack
targeting the data collection phase of the DNN training pipeline, requiring no
model access or auxiliary data. ReVeil maintains low pre-deployment ASR across
four datasets and four trigger patterns, successfully evades three popular
backdoor detection methods, and restores high ASR post-deployment through
machine unlearning.


http://arxiv.org/abs/2502.11910
Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives. (10%)
Leo Schwinn; Yan Scholten; Tom Wollschläger; Sophie Xhonneux; Stephen Casper; Stephan Günnemann; Gauthier Gidel
  Misaligned research objectives have considerably hindered progress in
adversarial robustness research over the past decade. For instance, an
extensive focus on optimizing target metrics, while neglecting rigorous
standardized evaluation, has led researchers to pursue ad-hoc heuristic
defenses that were seemingly effective. Yet, most of these were exposed as
flawed by subsequent evaluations, ultimately contributing little measurable
progress to the field. In this position paper, we illustrate that current
research on the robustness of large language models (LLMs) risks repeating past
patterns with potentially worsened real-world implications. To address this, we
argue that realigned objectives are necessary for meaningful progress in
adversarial alignment. To this end, we build on established cybersecurity
taxonomy to formally define differences between past and emerging threat models
that apply to LLMs. Using this framework, we illustrate that progress requires
disentangling adversarial alignment into addressable sub-problems and returning
to core academic principles, such as measureability, reproducibility, and
comparability. Although the field presents significant challenges, the fresh
start on adversarial robustness offers the unique opportunity to build on past
experience while avoiding previous mistakes.


http://arxiv.org/abs/2502.11598
Can LLM Watermarks Robustly Prevent Unauthorized Knowledge Distillation? (10%)
Leyi Pan; Aiwei Liu; Shiyu Huang; Yijian Lu; Xuming Hu; Lijie Wen; Irwin King; Philip S. Yu
  The radioactive nature of Large Language Model (LLM) watermarking enables the
detection of watermarks inherited by student models when trained on the outputs
of watermarked teacher models, making it a promising tool for preventing
unauthorized knowledge distillation. However, the robustness of watermark
radioactivity against adversarial actors remains largely unexplored. In this
paper, we investigate whether student models can acquire the capabilities of
teacher models through knowledge distillation while avoiding watermark
inheritance. We propose two categories of watermark removal approaches:
pre-distillation removal through untargeted and targeted training data
paraphrasing (UP and TP), and post-distillation removal through inference-time
watermark neutralization (WN). Extensive experiments across multiple model
pairs, watermarking schemes and hyper-parameter settings demonstrate that both
TP and WN thoroughly eliminate inherited watermarks, with WN achieving this
while maintaining knowledge transfer efficiency and low computational overhead.
Given the ongoing deployment of watermarking techniques in production LLMs,
these findings emphasize the urgent need for more robust defense strategies.
Our code is available at
https://github.com/THU-BPM/Watermark-Radioactivity-Attack.


http://arxiv.org/abs/2502.11743
Robust Partial-Label Learning by Leveraging Class Activation Values. (8%)
Tobias Fuchs; Florian Kalinke
  Real-world training data is often noisy; for example, human annotators assign
conflicting class labels to the same instances. Partial-label learning (PLL) is
a weakly supervised learning paradigm that allows training classifiers in this
context without manual data cleaning. While state-of-the-art methods have good
predictive performance, their predictions are sensitive to high noise levels,
out-of-distribution data, and adversarial perturbations. We propose a novel PLL
method based on subjective logic, which explicitly represents uncertainty by
leveraging the magnitudes of the underlying neural network's class activation
values. Thereby, we effectively incorporate prior knowledge about the class
labels by using a novel label weight re-distribution strategy that we prove to
be optimal. We empirically show that our method yields more robust predictions
in terms of predictive performance under high PLL noise levels, handling
out-of-distribution examples, and handling adversarial perturbations on the
test instances.


http://arxiv.org/abs/2502.11647
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing. (2%)
Yi Wang; Fenghua Weng; Sibei Yang; Zhan Qin; Minlie Huang; Wenjie Wang
  Large Language Models (LLMs) are widely applied in decision making, but their
deployment is threatened by jailbreak attacks, where adversarial users
manipulate model behavior to bypass safety measures. Existing defense
mechanisms, such as safety fine-tuning and model editing, either require
extensive parameter modifications or lack precision, leading to performance
degradation on general tasks, which is unsuitable to post-deployment safety
alignment. To address these challenges, we propose DELMAN (Dynamic Editing for
LLMs JAilbreak DefeNse), a novel approach leveraging direct model editing for
precise, dynamic protection against jailbreak attacks. DELMAN directly updates
a minimal set of relevant parameters to neutralize harmful behaviors while
preserving the model's utility. To avoid triggering a safe response in benign
context, we incorporate KL-divergence regularization to ensure the updated
model remains consistent with the original model when processing benign
queries. Experimental results demonstrate that DELMAN outperforms baseline
methods in mitigating jailbreak attacks while preserving the model's utility,
and adapts seamlessly to new attack instances, providing a practical and
efficient solution for post-deployment model protection.


http://arxiv.org/abs/2502.14896
A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Models. (2%)
Changhoon Kim; Yanjun Qi
  Text-to-Image (T2I) models have made remarkable progress in generating
high-quality, diverse visual content from natural language prompts. However,
their ability to reproduce copyrighted styles, sensitive imagery, and harmful
content raises significant ethical and legal concerns. Concept erasure offers a
proactive alternative to external filtering by modifying T2I models to prevent
the generation of undesired content. In this survey, we provide a structured
overview of concept erasure, categorizing existing methods based on their
optimization strategies and the architectural components they modify. We
categorize concept erasure methods into fine-tuning for parameter updates,
closed-form solutions for efficient edits, and inference-time interventions for
content restriction without weight modification. Additionally, we explore
adversarial attacks that bypass erasure techniques and discuss emerging
defenses. To support further research, we consolidate key datasets, evaluation
metrics, and benchmarks for assessing erasure effectiveness and model
robustness. This survey serves as a comprehensive resource, offering insights
into the evolving landscape of concept erasure, its challenges, and future
directions.


http://arxiv.org/abs/2502.11448
AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection. (1%)
Weidi Luo; Shenghong Dai; Xiaogeng Liu; Suman Banerjee; Huan Sun; Muhao Chen; Chaowei Xiao
  The rapid advancements in Large Language Models (LLMs) have enabled their
deployment as autonomous agents for handling complex tasks in dynamic
environments. These LLMs demonstrate strong problem-solving capabilities and
adaptability to multifaceted scenarios. However, their use as agents also
introduces significant risks, including task-specific risks, which are
identified by the agent administrator based on the specific task requirements
and constraints, and systemic risks, which stem from vulnerabilities in their
design or interactions, potentially compromising confidentiality, integrity, or
availability (CIA) of information and triggering security risks. Existing
defense agencies fail to adaptively and effectively mitigate these risks. In
this paper, we propose AGrail, a lifelong agent guardrail to enhance LLM agent
safety, which features adaptive safety check generation, effective safety check
optimization, and tool compatibility and flexibility. Extensive experiments
demonstrate that AGrail not only achieves strong performance against
task-specific and system risks but also exhibits transferability across
different LLM agents' tasks.


http://arxiv.org/abs/2502.12202
To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Models. (93%)
Zihao Zhu; Hongbao Zhang; Ruotong Wang; Ke Xu; Siwei Lyu; Baoyuan Wu
  Large Reasoning Models (LRMs) are designed to solve complex tasks by
generating explicit reasoning traces before producing final answers. However,
we reveal a critical vulnerability in LRMs -- termed Unthinking Vulnerability
-- wherein the thinking process can be bypassed by manipulating special
delimiter tokens. It is empirically demonstrated to be widespread across
mainstream LRMs, posing both a significant risk and potential utility,
depending on how it is exploited. In this paper, we systematically investigate
this vulnerability from both malicious and beneficial perspectives. On the
malicious side, we introduce Breaking of Thought (BoT), a novel attack that
enables adversaries to bypass the thinking process of LRMs, thereby
compromising their reliability and availability. We present two variants of
BoT: a training-based version that injects backdoor during the fine-tuning
stage, and a training-free version based on adversarial attack during the
inference stage. As a potential defense, we propose thinking recovery alignment
to partially mitigate the vulnerability. On the beneficial side, we introduce
Monitoring of Thought (MoT), a plug-and-play framework that allows model owners
to enhance efficiency and safety. It is implemented by leveraging the same
vulnerability to dynamically terminate redundant or risky reasoning through
external monitoring. Extensive experiments show that BoT poses a significant
threat to reasoning reliability, while MoT provides a practical solution for
preventing overthinking and jailbreaking. Our findings expose an inherent flaw
in current LRM architectures and underscore the need for more robust reasoning
systems in the future.


http://arxiv.org/abs/2502.13162
ShieldLearner: A New Paradigm for Jailbreak Attack Defense in LLMs. (69%)
Ziyi Ni; Hao Wang; Huacan Wang
  Large Language Models (LLMs) have achieved remarkable success in various
domains but remain vulnerable to adversarial jailbreak attacks. Existing
prompt-defense strategies, including parameter-modifying and parameter-free
approaches, face limitations in adaptability, interpretability, and
customization, constraining their effectiveness against evolving threats. To
address these challenges, we propose ShieldLearner, a novel paradigm that
mimics human learning in defense. Through trial and error, it autonomously
distills attack signatures into a Pattern Atlas and synthesizes defense
heuristics into a Meta-analysis Framework, enabling systematic and
interpretable threat detection. Furthermore, we introduce Adaptive Adversarial
Augmentation to generate adversarial variations of successfully defended
prompts, enabling continuous self-improvement without model retraining. In
addition to standard benchmarks, we create a hard test set by curating
adversarial prompts from the Wildjailbreak dataset, emphasizing more concealed
malicious intent. Experimental results show that ShieldLearner achieves a
significantly higher defense success rate than existing baselines on both
conventional and hard test sets, while also operating with lower computational
overhead, making it a practical and efficient solution for real-world
adversarial defense.


http://arxiv.org/abs/2502.14888
Multi-Faceted Multimodal Monosemanticity. (41%)
Hanqi Yan; Xiangxiang Cui; Lu Yin; Paul Pu Liang; Yulan He; Yifei Wang
  Humans experience the world through multiple modalities, such as, vision,
language, and speech, making it natural to explore the commonality and
distinctions among them. In this work, we take a data-driven approach to
address this question by analyzing interpretable, monosemantic features
extracted from deep multimodal models. Specifically, we investigate CLIP, a
prominent visual-language representation model trained on massive image-text
pairs. Building on prior research in single-modal interpretability, we develop
a set of multi-modal interpretability tools and measures designed to
disentangle and analyze features learned from CLIP. Specifically, we introduce
the Modality Dominance Score (MDS) to attribute each CLIP feature to a specific
modality. We then map CLIP features into a more interpretable space, enabling
us to categorize them into three distinct classes: vision features
(single-modal), language features (single-modal), and visual-language features
(cross-modal). Interestingly, this data-driven categorization closely aligns
with human intuitive understandings of different modalities. We further show
that this modality decomposition can benefit multiple downstream tasks,
including reducing bias in gender detection, generating cross-modal adversarial
examples, and enabling modal-specific feature control in text-to-image
generation. These results indicate that large-scale multimodal models, when
equipped with task-agnostic interpretability tools, can offer valuable insights
into the relationships between different data modalities.


http://arxiv.org/abs/2502.11308
ALGEN: Few-shot Inversion Attacks on Textual Embeddings using Alignment and Generation. (11%)
Yiyi Chen; Qiongkai Xu; Johannes Bjerva
  With the growing popularity of Large Language Models (LLMs) and vector
databases, private textual data is increasingly processed and stored as
numerical embeddings. However, recent studies have proven that such embeddings
are vulnerable to inversion attacks, where original text is reconstructed to
reveal sensitive information. Previous research has largely assumed access to
millions of sentences to train attack models, e.g., through data leakage or
nearly unrestricted API access. With our method, a single data point is
sufficient for a partially successful inversion attack. With as little as 1k
data samples, performance reaches an optimum across a range of black-box
encoders, without training on leaked data. We present a Few-shot Textual
Embedding Inversion Attack using ALignment and GENeration (ALGEN), by aligning
victim embeddings to the attack space and using a generative model to
reconstruct text. We find that ALGEN attacks can be effectively transferred
across domains and languages, revealing key information. We further examine a
variety of defense mechanisms against ALGEN, and find that none are effective,
highlighting the vulnerabilities posed by inversion attacks. By significantly
lowering the cost of inversion and proving that embedding spaces can be aligned
through one-step optimization, we establish a new textual embedding inversion
paradigm with broader applications for embedding alignment in NLP.


http://arxiv.org/abs/2502.11358
Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning System. (8%)
Ziyou Jiang; Mingyang Li; Guowei Yang; Junjie Wang; Yuekai Huang; Zhiyuan Chang; Qing Wang
  Information theft attacks pose a significant risk to Large Language Model
(LLM) tool-learning systems. Adversaries can inject malicious commands through
compromised tools, manipulating LLMs to send sensitive information to these
tools, which leads to potential privacy breaches. However, existing attack
approaches are black-box oriented and rely on static commands that cannot adapt
flexibly to the changes in user queries and the invocation chain of tools. It
makes malicious commands more likely to be detected by LLM and leads to attack
failure. In this paper, we propose AutoCMD, a dynamic attack comment generation
approach for information theft attacks in LLM tool-learning systems. Inspired
by the concept of mimicking the familiar, AutoCMD is capable of inferring the
information utilized by upstream tools in the toolchain through learning on
open-source systems and reinforcement with target system examples, thereby
generating more targeted commands for information theft. The evaluation results
show that AutoCMD outperforms the baselines with +13.2% $ASR_{Theft}$, and can
be generalized to new tool-learning systems to expose their information leakage
risks. We also design four defense methods to effectively protect tool-learning
systems from the attack.


http://arxiv.org/abs/2502.11379
CCJA: Context-Coherent Jailbreak Attack for Aligned Large Language Models. (8%)
Guanghao Zhou; Panjia Qiu; Mingyuan Fan; Cen Chen; Mingyuan Chu; Xin Zhang; Jun Zhou
  Despite explicit alignment efforts for large language models (LLMs), they can
still be exploited to trigger unintended behaviors, a phenomenon known as
"jailbreaking." Current jailbreak attack methods mainly focus on discrete
prompt manipulations targeting closed-source LLMs, relying on manually crafted
prompt templates and persuasion rules. However, as the capabilities of
open-source LLMs improve, ensuring their safety becomes increasingly crucial.
In such an environment, the accessibility of model parameters and gradient
information by potential attackers exacerbates the severity of jailbreak
threats. To address this research gap, we propose a novel
\underline{C}ontext-\underline{C}oherent \underline{J}ailbreak
\underline{A}ttack (CCJA). We define jailbreak attacks as an optimization
problem within the embedding space of masked language models. Through
combinatorial optimization, we effectively balance the jailbreak attack success
rate with semantic coherence. Extensive evaluations show that our method not
only maintains semantic consistency but also surpasses state-of-the-art
baselines in attack effectiveness. Additionally, by integrating semantically
coherent jailbreak prompts generated by our method into widely used black-box
methodologies, we observe a notable enhancement in their success rates when
targeting closed-source commercial LLMs. This highlights the security threat
posed by open-source LLMs to commercial counterparts. We will open-source our
code if the paper is accepted.


http://arxiv.org/abs/2502.11127
G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems. (1%)
Shilong Wang; Guibin Zhang; Miao Yu; Guancheng Wan; Fanci Meng; Chongye Guo; Kun Wang; Yang Wang
  Large Language Model (LLM)-based Multi-agent Systems (MAS) have demonstrated
remarkable capabilities in various complex tasks, ranging from collaborative
problem-solving to autonomous decision-making. However, as these systems become
increasingly integrated into critical applications, their vulnerability to
adversarial attacks, misinformation propagation, and unintended behaviors have
raised significant concerns. To address this challenge, we introduce
G-Safeguard, a topology-guided security lens and treatment for robust LLM-MAS,
which leverages graph neural networks to detect anomalies on the multi-agent
utterance graph and employ topological intervention for attack remediation.
Extensive experiments demonstrate that G-Safeguard: (I) exhibits significant
effectiveness under various attack strategies, recovering over 40% of the
performance for prompt injection; (II) is highly adaptable to diverse LLM
backbones and large-scale MAS; (III) can seamlessly combine with mainstream MAS
with security guarantees. The code is available at
https://github.com/wslong20/G-safeguard.


http://arxiv.org/abs/2502.11084
Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction. (1%)
Yuting Huang; Chengyuan Liu; Yifeng Feng; Yiquan Wu; Chao Wu; Fei Wu; Kun Kuang
  As Large Language Models (LLMs) are widely applied in various domains, the
safety of LLMs is increasingly attracting attention to avoid their powerful
capabilities being misused. Existing jailbreak methods create a forced
instruction-following scenario, or search adversarial prompts with prefix or
suffix tokens to achieve a specific representation manually or automatically.
However, they suffer from low efficiency and explicit jailbreak patterns, far
from the real deployment of mass attacks to LLMs. In this paper, we point out
that simply rewriting the original instruction can achieve a jailbreak, and we
find that this rewriting approach is learnable and transferable. We propose the
Rewrite to Jailbreak (R2J) approach, a transferable black-box jailbreak method
to attack LLMs by iteratively exploring the weakness of the LLMs and
automatically improving the attacking strategy. The jailbreak is more efficient
and hard to identify since no additional features are introduced. Extensive
experiments and analysis demonstrate the effectiveness of R2J, and we find that
the jailbreak is also transferable to multiple datasets and various types of
models with only a few queries. We hope our work motivates further
investigation of LLM safety. The code can be found at
https://github.com/ythuang02/R2J/.


http://arxiv.org/abs/2502.10801
FaceSwapGuard: Safeguarding Facial Privacy from DeepFake Threats through Identity Obfuscation. (83%)
Li Wang; Zheng Li; Xuhong Zhang; Shouling Ji; Shanqing Guo
  DeepFakes pose a significant threat to our society. One representative
DeepFake application is face-swapping, which replaces the identity in a facial
image with that of a victim. Although existing methods partially mitigate these
risks by degrading the quality of swapped images, they often fail to disrupt
the identity transformation effectively. To fill this gap, we propose
FaceSwapGuard (FSG), a novel black-box defense mechanism against deepfake
face-swapping threats. Specifically, FSG introduces imperceptible perturbations
to a user's facial image, disrupting the features extracted by identity
encoders. When shared online, these perturbed images mislead face-swapping
techniques, causing them to generate facial images with identities
significantly different from the original user. Extensive experiments
demonstrate the effectiveness of FSG against multiple face-swapping techniques,
reducing the face match rate from 90\% (without defense) to below 10\%. Both
qualitative and quantitative studies further confirm its ability to confuse
human perception, highlighting its practical utility. Additionally, we
investigate key factors that may influence FSG and evaluate its robustness
against various adaptive adversaries.


http://arxiv.org/abs/2502.10682
CAE-Net: Generalized Deepfake Image Detection using Convolution and Attention Mechanisms with Spatial and Frequency Domain Features. (8%)
Kafi Anan; Anindya Bhattacharjee; Ashir Intesher; Kaidul Islam; Abrar Assaeem Fuad; Utsab Saha; Hafiz Imtiaz
  Effective deepfake detection tools are becoming increasingly essential to the
growing usage of deepfakes in unethical practices. There exists a wide range of
deepfake generation techniques, which makes it challenging to develop an
accurate universal detection mechanism. The 2025 IEEE Signal Processing Cup
(\textit{DFWild-Cup} competition) provided a diverse dataset of deepfake images
containing significant class imbalance. The images in the dataset are generated
from multiple deepfake image generators, for training machine learning model(s)
to emphasize the generalization of deepfake detection. To this end, we proposed
a disjoint set-based multistage training method to address the class imbalance
and devised an ensemble-based architecture \emph{CAE-Net}. Our architecture
consists of a convolution- and attention-based ensemble network, and employs
three different neural network architectures: EfficientNet, Data-Efficient
Image Transformer (DeiT), and ConvNeXt with wavelet transform to capture both
local and global features of deepfakes. We visualize the specific regions that
these models focus on for classification using Grad-CAM, and empirically
demonstrate the effectiveness of these models in grouping real and fake images
into cohesive clusters using t-SNE plots. Individually, the EfficientNet B0
architecture has achieved 90.79\% accuracy, whereas the ConvNeXt and the DeiT
architecture have achieved 89.49\% and 89.32\% accuracy, respectively. With
these networks, our weighted ensemble model achieves an excellent accuracy of
94.63\% on the validation dataset of the SP Cup 2025 competition. The equal
error rate of 4.72\% and the Area Under the ROC curve of 97.37\% further
confirm the stability of our proposed method. Finally, the robustness of our
proposed model against adversarial perturbation attacks is tested as well,
showing the inherent defensive properties of the ensemble approach.


http://arxiv.org/abs/2502.10329
VocalCrypt: Novel Active Defense Against Deepfake Voice Based on Masking Effect. (54%)
Qingyuan Fei; Wenjie Hou; Xuan Hai; Xin Liu
  The rapid advancements in AI voice cloning, fueled by machine learning, have
significantly impacted text-to-speech (TTS) and voice conversion (VC) fields.
While these developments have led to notable progress, they have also raised
concerns about the misuse of AI VC technology, causing economic losses and
negative public perceptions. To address this challenge, this study focuses on
creating active defense mechanisms against AI VC systems.
  We propose a novel active defense method, VocalCrypt, which embeds
pseudo-timbre (jamming information) based on SFS into audio segments that are
imperceptible to the human ear, thereby forming systematic fragments to prevent
voice cloning. This approach protects the voice without compromising its
quality. In comparison to existing methods, such as adversarial noise
incorporation, VocalCrypt significantly enhances robustness and real-time
performance, achieving a 500\% increase in generation speed while maintaining
interference effectiveness.
  Unlike audio watermarking techniques, which focus on post-detection, our
method offers preemptive defense, reducing implementation costs and enhancing
feasibility. Extensive experiments using the Zhvoice and VCTK Corpus datasets
show that our AI-cloned speech defense system performs excellently in automatic
speaker verification (ASV) tests while preserving the integrity of the
protected audio.


http://arxiv.org/abs/2502.10487
Fast Proxies for LLM Robustness Evaluation. (15%)
Tim Beyer; Jan Schuchardt; Leo Schwinn; Stephan Günnemann
  Evaluating the robustness of LLMs to adversarial attacks is crucial for safe
deployment, yet current red-teaming methods are often prohibitively expensive.
We compare the ability of fast proxy metrics to predict the real-world
robustness of an LLM against a simulated attacker ensemble. This allows us to
estimate a model's robustness to computationally expensive attacks without
requiring runs of the attacks themselves. Specifically, we consider
gradient-descent-based embedding-space attacks, prefilling attacks, and direct
prompting. Even though direct prompting in particular does not achieve high
ASR, we find that it and embedding-space attacks can predict attack success
rates well, achieving $r_p=0.87$ (linear) and $r_s=0.94$ (Spearman rank)
correlations with the full attack ensemble while reducing computational cost by
three orders of magnitude.


http://arxiv.org/abs/2502.09990
X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usability. (3%)
Xiaoya Lu; Dongrui Liu; Yi Yu; Luxin Xu; Jing Shao
  Despite the rapid development of safety alignment techniques for LLMs,
defending against multi-turn jailbreaks is still a challenging task. In this
paper, we conduct a comprehensive comparison, revealing that some existing
defense methods can improve the robustness of LLMs against multi-turn
jailbreaks but compromise usability, i.e., reducing general capabilities or
causing the over-refusal problem. From the perspective of mechanism
interpretability of LLMs, we discover that these methods fail to establish a
boundary that exactly distinguishes safe and harmful feature representations.
Therefore, boundary-safe representations close to harmful representations are
inevitably disrupted, leading to a decline in usability. To address this issue,
we propose X-Boundary to push harmful representations away from boundary-safe
representations and obtain an exact distinction boundary. In this way, harmful
representations can be precisely erased without disrupting safe ones.
Experimental results show that X-Boundary achieves state-of-the-art defense
performance against multi-turn jailbreaks, while reducing the over-refusal rate
by about 20% and maintaining nearly complete general capability. Furthermore,
we theoretically prove and empirically verify that X-Boundary can accelerate
the convergence process during training. Please see our code at:
https://github.com/AI45Lab/X-Boundary.


http://arxiv.org/abs/2502.09110
Pulling Back the Curtain: Unsupervised Adversarial Detection via Contrastive Auxiliary Networks. (99%)
Eylon Mizrahi; Raz Lapid; Moshe Sipper
  Deep learning models are widely employed in safety-critical applications yet
remain susceptible to adversarial attacks -- imperceptible perturbations that
can significantly degrade model performance. Conventional defense mechanisms
predominantly focus on either enhancing model robustness or detecting
adversarial inputs independently. In this work, we propose an Unsupervised
adversarial detection via Contrastive Auxiliary Networks (U-CAN) to uncover
adversarial behavior within auxiliary feature representations, without the need
for adversarial examples. U-CAN is embedded within selected intermediate layers
of the target model. These auxiliary networks, comprising projection layers and
ArcFace-based linear layers, refine feature representations to more effectively
distinguish between benign and adversarial inputs. Comprehensive experiments
across multiple datasets (CIFAR-10, Mammals, and a subset of ImageNet) and
architectures (ResNet-50, VGG-16, and ViT) demonstrate that our method
surpasses existing unsupervised adversarial detection techniques, achieving
superior F1 scores against four distinct attack methods. The proposed framework
provides a scalable and effective solution for enhancing the security and
reliability of deep learning systems.


http://arxiv.org/abs/2502.09553
SyntheticPop: Attacking Speaker Verification Systems With Synthetic VoicePops. (67%)
Eshaq Jamdar; Amith Kamath Belman
  Voice Authentication (VA), also known as Automatic Speaker Verification
(ASV), is a widely adopted authentication method, particularly in automated
systems like banking services, where it serves as a secondary layer of user
authentication. Despite its popularity, VA systems are vulnerable to various
attacks, including replay, impersonation, and the emerging threat of deepfake
audio that mimics the voice of legitimate users. To mitigate these risks,
several defense mechanisms have been proposed. One such solution, Voice Pops,
aims to distinguish an individual's unique phoneme pronunciations during the
enrollment process. While promising, the effectiveness of VA+VoicePop against a
broader range of attacks, particularly logical or adversarial attacks, remains
insufficiently explored. We propose a novel attack method, which we refer to as
SyntheticPop, designed to target the phoneme recognition capabilities of the
VA+VoicePop system. The SyntheticPop attack involves embedding synthetic "pop"
noises into spoofed audio samples, significantly degrading the model's
performance. We achieve an attack success rate of over 95% while poisoning 20%
of the training dataset. Our experiments demonstrate that VA+VoicePop achieves
69% accuracy under normal conditions, 37% accuracy when subjected to a baseline
label flipping attack, and just 14% accuracy under our proposed SyntheticPop
attack, emphasizing the effectiveness of our method.


http://arxiv.org/abs/2502.09352
Wasserstein distributional adversarial training for deep neural networks. (54%)
Xingjian Bai; Guangyi He; Yifan Jiang; Jan Obloj
  Design of adversarial attacks for deep neural networks, as well as methods of
adversarial training against them, are subject of intense research. In this
paper, we propose methods to train against distributional attack threats,
extending the TRADES method used for pointwise attacks. Our approach leverages
recent contributions and relies on sensitivity analysis for Wasserstein
distributionally robust optimization problems. We introduce an efficient
fine-tuning method which can be deployed on a previously trained model. We test
our methods on a range of pre-trained models on RobustBench. These experimental
results demonstrate the additional training enhances Wasserstein distributional
robustness, while maintaining original levels of pointwise robustness, even for
already very successful networks. The improvements are less marked for models
pre-trained using huge synthetic datasets of 20-100M images. However,
remarkably, sometimes our methods are still able to improve their performance
even when trained using only the original training dataset (50k images).


http://arxiv.org/abs/2502.09723
Making Them a Malicious Database: Exploiting Query Code to Jailbreak Aligned Large Language Models. (31%)
Qingsong Zou; Jingyu Xiao; Qing Li; Zhi Yan; Yuhang Wang; Li Xu; Wenxuan Wang; Kuofeng Gao; Ruoyu Li; Yong Jiang
  Recent advances in large language models (LLMs) have demonstrated remarkable
potential in the field of natural language processing. Unfortunately, LLMs face
significant security and ethical risks. Although techniques such as safety
alignment are developed for defense, prior researches reveal the possibility of
bypassing such defenses through well-designed jailbreak attacks. In this paper,
we propose QueryAttack, a novel framework to examine the generalizability of
safety alignment. By treating LLMs as knowledge databases, we translate
malicious queries in natural language into structured non-natural query
language to bypass the safety alignment mechanisms of LLMs. We conduct
extensive experiments on mainstream LLMs, and the results show that QueryAttack
not only can achieve high attack success rates (ASRs), but also can jailbreak
various defense methods. Furthermore, we tailor a defense method against
QueryAttack, which can reduce ASR by up to 64% on GPT-4-1106. Our code is
available at https://github.com/horizonsinzqs/QueryAttack.


http://arxiv.org/abs/2502.09175
FLAME: Flexible LLM-Assisted Moderation Engine. (11%)
Ivan AIRI Moscow Institute of Physics and Technology Bakulin; Ilia AIRI Moscow Institute of Physics and Technology Kopanichuk; Iaroslav AIRI Bespalov; Nikita SberHealth Radchenko; Vladimir AIRI Skolkovo Institute of Science and Technology Shaposhnikov; Dmitry AIRI Skolkovo Institute of Science and Technology Dylov; Ivan AIRI Skolkovo Institute of Science and Technology Oseledets
  The rapid advancement of Large Language Models (LLMs) has introduced
significant challenges in moderating user-model interactions. While LLMs
demonstrate remarkable capabilities, they remain vulnerable to adversarial
attacks, particularly ``jailbreaking'' techniques that bypass content safety
measures. Current content moderation systems, which primarily rely on input
prompt filtering, have proven insufficient, with techniques like Best-of-N
(BoN) jailbreaking achieving success rates of 80% or more against popular LLMs.
In this paper, we introduce Flexible LLM-Assisted Moderation Engine (FLAME): a
new approach that shifts the focus from input filtering to output moderation.
Unlike traditional circuit-breaking methods that analyze user queries, FLAME
evaluates model responses, offering several key advantages: (1) computational
efficiency in both training and inference, (2) enhanced resistance to BoN
jailbreaking attacks, and (3) flexibility in defining and updating safety
criteria through customizable topic filtering. Our experiments demonstrate that
FLAME significantly outperforms current moderation systems. For example, FLAME
reduces attack success rate in GPT-4o-mini and DeepSeek-v3 by a factor of ~9,
while maintaining low computational overhead. We provide comprehensive
evaluation on various LLMs and analyze the engine's efficiency against the
state-of-the-art jailbreaking. This work contributes to the development of more
robust and adaptable content moderation systems for LLMs.


http://arxiv.org/abs/2502.09271
LiSA: Leveraging Link Recommender to Attack Graph Neural Networks via Subgraph Injection. (10%)
Wenlun Zhang; Enyan Dai; Kentaro Yoshioka
  Graph Neural Networks (GNNs) have demonstrated remarkable proficiency in
modeling data with graph structures, yet recent research reveals their
susceptibility to adversarial attacks. Traditional attack methodologies, which
rely on manipulating the original graph or adding links to artificially created
nodes, often prove impractical in real-world settings. This paper introduces a
novel adversarial scenario involving the injection of an isolated subgraph to
deceive both the link recommender and the node classifier within a GNN system.
Specifically, the link recommender is mislead to propose links between targeted
victim nodes and the subgraph, encouraging users to unintentionally establish
connections and that would degrade the node classification accuracy, thereby
facilitating a successful attack. To address this, we present the LiSA
framework, which employs a dual surrogate model and bi-level optimization to
simultaneously meet two adversarial objectives. Extensive experiments on
real-world datasets demonstrate the effectiveness of our method.


http://arxiv.org/abs/2502.09837
SoK: State of the time: On Trustworthiness of Digital Clocks. (3%)
Adeel Nasrullah; Fatima M. Anwar
  Despite the critical role of timing infrastructure in enabling essential
services, from public key infrastructure and smart grids to autonomous
navigation and high-frequency trading, modern timing stacks remain highly
vulnerable to malicious attacks. These threats emerge due to several reasons,
including inadequate security mechanisms, the timing architectures unique
vulnerability to delays, and implementation issues. In this paper, we aim to
obtain a holistic understanding of the issues that make the timing stacks
vulnerable to adversarial manipulations, what the challenges are in securing
them, and what solutions can be borrowed from the research community to address
them. To this end, we perform a systematic analysis of the security
vulnerabilities of the timing stack. In doing so, we discover new attack
surfaces, i.e., physical timing components and on-device timekeeping, which are
often overlooked by existing research that predominantly studies the security
of time synchronization protocols. We also show that the emerging trusted
timing architectures are flawed and risk compromising wider system security,
and propose an alternative design using hardware-software co-design.


http://arxiv.org/abs/2502.09150
Shortcut Learning Susceptibility in Vision Classifiers. (3%)
Pirzada Suhail; Amit Sethi
  Shortcut learning, where machine learning models exploit spurious
correlations in data instead of capturing meaningful features, poses a
significant challenge to building robust and generalizable models. This
phenomenon is prevalent across various machine learning applications, including
vision, natural language processing, and speech recognition, where models may
find unintended cues that minimize training loss but fail to capture the
underlying structure of the data. Vision classifiers such as Convolutional
Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), and Vision Transformers
(ViTs) leverage distinct architectural principles to process spatial and
structural information, making them differently susceptible to shortcut
learning. In this study, we systematically evaluate these architectures by
introducing deliberate shortcuts into the dataset that are positionally
correlated with class labels, creating a controlled setup to assess whether
models rely on these artificial cues or learn actual distinguishing features.
We perform both quantitative evaluation by training on the shortcut-modified
dataset and testing them on two different test sets -- one containing the same
shortcuts and another without them -- to determine the extent of reliance on
shortcuts. Additionally, qualitative evaluation is performed by using network
inversion-based reconstruction techniques to analyze what the models
internalize in their weights, aiming to reconstruct the training data as
perceived by the classifiers. We evaluate shortcut learning behavior across
multiple benchmark datasets, including MNIST, Fashion-MNIST, SVHN, and
CIFAR-10, to compare the susceptibility of different vision classifier
architectures to shortcut reliance and assess their varying degrees of
sensitivity to spurious correlations.


http://arxiv.org/abs/2502.08989
RLSA-PFL: Robust Lightweight Secure Aggregation with Model Inconsistency Detection in Privacy-Preserving Federated Learning. (1%)
Nazatul H. Sultan; Yan Bo; Yansong Gao; Seyit Camtepe; Arash Mahboubi; Hang Thanh Bui; Aufeef Chauhan; Hamed Aboutorab; Michael Bewong; Praveen Gauravaram; Rafiqul Islam; Sharif Abuadbba
  Federated Learning (FL) allows users to collaboratively train a global
machine learning model by sharing local model only, without exposing their
private data to a central server. This distributed learning is particularly
appealing in scenarios where data privacy is crucial, and it has garnered
substantial attention from both industry and academia. However, studies have
revealed privacy vulnerabilities in FL, where adversaries can potentially infer
sensitive information from the shared model parameters. In this paper, we
present an efficient masking-based secure aggregation scheme utilizing
lightweight cryptographic primitives to mitigate privacy risks. Our scheme
offers several advantages over existing methods. First, it requires only a
single setup phase for the entire FL training session, significantly reducing
communication overhead. Second, it minimizes user-side overhead by eliminating
the need for user-to-user interactions, utilizing an intermediate server layer
and a lightweight key negotiation method. Third, the scheme is highly resilient
to user dropouts, and the users can join at any FL round. Fourth, it can detect
and defend against malicious server activities, including recently discovered
model inconsistency attacks. Finally, our scheme ensures security in both
semi-honest and malicious settings. We provide security analysis to formally
prove the robustness of our approach. Furthermore, we implemented an end-to-end
prototype of our scheme. We conducted comprehensive experiments and
comparisons, which show that it outperforms existing solutions in terms of
communication and computation overhead, functionality, and security.


http://arxiv.org/abs/2502.08374
AdvSwap: Covert Adversarial Perturbation with High Frequency Info-swapping for Autonomous Driving Perception. (99%)
Yuanhao Huang; Qinfan Zhang; Jiandong Xing; Mengyue Cheng; Haiyang Yu; Yilong Ren; Xiao Xiong
  Perception module of Autonomous vehicles (AVs) are increasingly susceptible
to be attacked, which exploit vulnerabilities in neural networks through
adversarial inputs, thereby compromising the AI safety. Some researches focus
on creating covert adversarial samples, but existing global noise techniques
are detectable and difficult to deceive the human visual system. This paper
introduces a novel adversarial attack method, AdvSwap, which creatively
utilizes wavelet-based high-frequency information swapping to generate covert
adversarial samples and fool the camera. AdvSwap employs invertible neural
network for selective high-frequency information swapping, preserving both
forward propagation and data integrity. The scheme effectively removes the
original label data and incorporates the guidance image data, producing
concealed and robust adversarial samples. Experimental evaluations and
comparisons on the GTSRB and nuScenes datasets demonstrate that AdvSwap can
make concealed attacks on common traffic targets. The generates adversarial
samples are also difficult to perceive by humans and algorithms. Meanwhile, the
method has strong attacking robustness and attacking transferability.


http://arxiv.org/abs/2502.08151
Local Differential Privacy is Not Enough: A Sample Reconstruction Attack against Federated Learning with Local Differential Privacy. (54%)
Zhichao You; Xuewen Dong; Shujun Li; Ximeng Liu; Siqi Ma; Yulong Shen
  Reconstruction attacks against federated learning (FL) aim to reconstruct
users' samples through users' uploaded gradients. Local differential privacy
(LDP) is regarded as an effective defense against various attacks, including
sample reconstruction in FL, where gradients are clipped and perturbed.
Existing attacks are ineffective in FL with LDP since clipped and perturbed
gradients obliterate most sample information for reconstruction. Besides,
existing attacks embed additional sample information into gradients to improve
the attack effect and cause gradient expansion, leading to a more severe
gradient clipping in FL with LDP. In this paper, we propose a sample
reconstruction attack against LDP-based FL with any target models to
reconstruct victims' sensitive samples to illustrate that FL with LDP is not
flawless. Considering gradient expansion in reconstruction attacks and noise in
LDP, the core of the proposed attack is gradient compression and reconstructed
sample denoising. For gradient compression, an inference structure based on
sample characteristics is presented to reduce redundant gradients against LDP.
For reconstructed sample denoising, we artificially introduce zero gradients to
observe noise distribution and scale confidence interval to filter the noise.
Theoretical proof guarantees the effectiveness of the proposed attack.
Evaluations show that the proposed attack is the only attack that reconstructs
victims' training samples in LDP-based FL and has little impact on the target
model's accuracy. We conclude that LDP-based FL needs further improvements to
defend against sample reconstruction attacks effectively.


http://arxiv.org/abs/2502.08638
Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples. (50%)
Andrianos Michail; Simon Clematide; Rico Sennrich
  The evaluation of cross-lingual semantic search capabilities of models is
often limited to existing datasets from tasks such as information retrieval and
semantic textual similarity. To allow for domain-specific evaluation, we
introduce Cross Lingual Semantic Discrimination (CLSD), a novel cross-lingual
semantic search task that requires only a set of parallel sentence pairs of the
language pair of interest within the target domain. This task focuses on the
ability of a model to cross-lingually rank the true parallel sentence higher
than hard negatives generated by a large language model. We create four
instances of our introduced CLSD task for the language pair German-French
within the domain of news. Within this case study, we find that models that are
also fine-tuned for retrieval tasks (e.g., multilingual E5) benefit from using
English as the pivot language, while bitext mining models such as LaBSE perform
best directly cross-lingually. We also show a fine-grained similarity analysis
enabled by our distractor generation strategy, indicating that different
embedding models are sensitive to different types of perturbations.


http://arxiv.org/abs/2502.08193
Typographic Attacks in a Multi-Image Setting. (41%)
Xiaomeng Wang; Zhengyu Zhao; Martha Larson
  Large Vision-Language Models (LVLMs) are susceptible to typographic attacks,
which are misclassifications caused by an attack text that is added to an
image. In this paper, we introduce a multi-image setting for studying
typographic attacks, broadening the current emphasis of the literature on
attacking individual images. Specifically, our focus is on attacking image sets
without repeating the attack query. Such non-repeating attacks are stealthier,
as they are more likely to evade a gatekeeper than attacks that repeat the same
attack text. We introduce two attack strategies for the multi-image setting,
leveraging the difficulty of the target image, the strength of the attack text,
and text-image similarity. Our text-image similarity approach improves attack
success rates by 21% over random, non-specific methods on the CLIP model using
ImageNet while maintaining stealth in a multi-image scenario. An additional
experiment demonstrates transferability, i.e., text-image similarity calculated
using CLIP transfers when attacking InstructBLIP.


http://arxiv.org/abs/2502.08586
Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks. (12%)
Ang Li; Yin Zhou; Vethavikashini Chithrra Raghuram; Tom Goldstein; Micah Goldblum
  A high volume of recent ML security literature focuses on attacks against
aligned large language models (LLMs). These attacks may extract private
information or coerce the model into producing harmful outputs. In real-world
deployments, LLMs are often part of a larger agentic pipeline including memory
systems, retrieval, web access, and API calling. Such additional components
introduce vulnerabilities that make these LLM-powered agents much easier to
attack than isolated LLMs, yet relatively little work focuses on the security
of LLM agents. In this paper, we analyze security and privacy vulnerabilities
that are unique to LLM agents. We first provide a taxonomy of attacks
categorized by threat actors, objectives, entry points, attacker observability,
attack strategies, and inherent vulnerabilities of agent pipelines. We then
conduct a series of illustrative attacks on popular open-source and commercial
agents, demonstrating the immediate practical implications of their
vulnerabilities. Notably, our attacks are trivial to implement and require no
understanding of machine learning.


http://arxiv.org/abs/2502.08927
Dynamic watermarks in images generated by diffusion models. (3%)
Yunzhuo Chen; Naveed Akhtar; Nur Al Hasan Haldar; Ajmal Mian
  High-fidelity text-to-image diffusion models have revolutionized visual
content generation, but their widespread use raises significant ethical
concerns, including intellectual property protection and the misuse of
synthetic media. To address these challenges, we propose a novel multi-stage
watermarking framework for diffusion models, designed to establish copyright
and trace generated images back to their source. Our multi-stage watermarking
technique involves embedding: (i) a fixed watermark that is localized in the
diffusion model's learned noise distribution and, (ii) a human-imperceptible,
dynamic watermark in generates images, leveraging a fine-tuned decoder. By
leveraging the Structural Similarity Index Measure (SSIM) and cosine
similarity, we adapt the watermark's shape and color to the generated content
while maintaining robustness. We demonstrate that our method enables reliable
source verification through watermark classification, even when the dynamic
watermark is adjusted for content-specific variations. Source model
verification is enabled through watermark classification. o support further
research, we generate a dataset of watermarked images and introduce a
methodology to evaluate the statistical impact of watermarking on generated
content.Additionally, we rigorously test our framework against various attack
scenarios, demonstrating its robustness and minimal impact on image quality.
Our work advances the field of AI-generated content security by providing a
scalable solution for model ownership verification and misuse prevention.


http://arxiv.org/abs/2502.08448
Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry. (1%)
Albert Kjøller Jacobsen; Georgios Arvanitidis
  Recent studies on deep neural networks show that flat minima of the loss
landscape correlate with improved generalization. Sharpness-aware minimization
(SAM) efficiently finds flat regions by updating the parameters according to
the gradient at an adversarial perturbation. The perturbation depends on the
Euclidean metric, making SAM non-invariant under reparametrizations, which
blurs sharpness and generalization. We propose Monge SAM (M-SAM), a
reparametrization invariant version of SAM by considering a Riemannian metric
in the parameter space induced naturally by the loss surface. Compared to
previous approaches, M-SAM works under any modeling choice, relies only on mild
assumptions while being as computationally efficient as SAM. We theoretically
argue that M-SAM varies between SAM and gradient descent (GD), which increases
robustness to hyperparameter selection and reduces attraction to suboptimal
equilibria like saddle points. We demonstrate this behavior both theoretically
and empirically on a multi-modal representation alignment task.


http://arxiv.org/abs/2502.08123
Provably Robust Federated Reinforcement Learning. (1%)
Minghong Fang; Xilong Wang; Neil Zhenqiang Gong
  Federated reinforcement learning (FRL) allows agents to jointly learn a
global decision-making policy under the guidance of a central server. While FRL
has advantages, its decentralized design makes it prone to poisoning attacks.
To mitigate this, Byzantine-robust aggregation techniques tailored for FRL have
been introduced. Yet, in our work, we reveal that these current
Byzantine-robust techniques are not immune to our newly introduced Normalized
attack. Distinct from previous attacks that targeted enlarging the distance of
policy updates before and after an attack, our Normalized attack emphasizes on
maximizing the angle of deviation between these updates. To counter these
threats, we develop an ensemble FRL approach that is provably secure against
both known and our newly proposed attacks. Our ensemble method involves
training multiple global policies, where each is learnt by a group of agents
using any foundational aggregation rule. These well-trained global policies
then individually predict the action for a specific test state. The ultimate
action is chosen based on a majority vote for discrete action systems or the
geometric median for continuous ones. Our experimental results across different
settings show that the Normalized attack can greatly disrupt non-ensemble
Byzantine-robust methods, and our ensemble approach offers substantial
resistance against poisoning attacks.


http://arxiv.org/abs/2502.07492
RoMA: Robust Malware Attribution via Byte-level Adversarial Training with Global Perturbations and Adversarial Consistency Regularization. (99%)
Yuxia Sun; Huihong Chen; Jingcai Guo; Aoxiang Sun; Zhetao Li; Haolin Liu
  Attributing APT (Advanced Persistent Threat) malware to their respective
groups is crucial for threat intelligence and cybersecurity. However, APT
adversaries often conceal their identities, rendering attribution inherently
adversarial. Existing machine learning-based attribution models, while
effective, remain highly vulnerable to adversarial attacks. For example, the
state-of-the-art byte-level model MalConv sees its accuracy drop from over 90%
to below 2% under PGD (projected gradient descent) attacks. Existing
gradient-based adversarial training techniques for malware detection or image
processing were applied to malware attribution in this study, revealing that
both robustness and training efficiency require significant improvement. To
address this, we propose RoMA, a novel single-step adversarial training
approach that integrates global perturbations to generate enhanced adversarial
samples and employs adversarial consistency regularization to improve
representation quality and resilience. A novel APT malware dataset named AMG18,
with diverse samples and realistic class imbalances, is introduced for
evaluation. Extensive experiments show that RoMA significantly outperforms
seven competing methods in both adversarial robustness (e.g., achieving over
80% robust accuracy-more than twice that of the next-best method under PGD
attacks) and training efficiency (e.g., more than twice as fast as the
second-best method in terms of accuracy), while maintaining superior standard
accuracy in non-adversarial scenarios.


http://arxiv.org/abs/2502.07987
Universal Adversarial Attack on Aligned Multimodal LLMs. (98%)
Temurbek Rahmatullaev; Polina Druzhinina; Matvey Mikhalchuk; Andrey Kuznetsov; Anton Razzhigaev
  We propose a universal adversarial attack on multimodal Large Language Models
(LLMs) that leverages a single optimized image to override alignment safeguards
across diverse queries and even multiple models. By backpropagating through the
vision encoder and language head, we craft a synthetic image that forces the
model to respond with a targeted phrase (e.g., ''Sure, here it is'') or
otherwise unsafe content-even for harmful prompts. In experiments on the
SafeBench benchmark, our method achieves significantly higher attack success
rates than existing baselines, including text-only universal prompts (e.g., up
to 93% on certain models). We further demonstrate cross-model transferability
by training on several multimodal LLMs simultaneously and testing on unseen
architectures. Additionally, a multi-answer variant of our approach produces
more natural-sounding (yet still malicious) responses. These findings
underscore critical vulnerabilities in current multimodal alignment and call
for more robust adversarial defenses. We will release code and datasets under
the Apache-2.0 license. Warning: some content generated by Multimodal LLMs in
this paper may be offensive to some readers.


http://arxiv.org/abs/2502.08079
MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models. (96%)
Peng-Fei Zhang; Guangdong Bai; Zi Huang
  Current adversarial attacks for evaluating the robustness of vision-language
pre-trained (VLP) models in multi-modal tasks suffer from limited
transferability, where attacks crafted for a specific model often struggle to
generalize effectively across different models, limiting their utility in
assessing robustness more broadly. This is mainly attributed to the
over-reliance on model-specific features and regions, particularly in the image
modality. In this paper, we propose an elegant yet highly effective method
termed Meticulous Adversarial Attack (MAA) to fully exploit model-independent
characteristics and vulnerabilities of individual samples, achieving enhanced
generalizability and reduced model dependence. MAA emphasizes fine-grained
optimization of adversarial images by developing a novel resizing and sliding
crop (RScrop) technique, incorporating a multi-granularity similarity
disruption (MGSD) strategy. Extensive experiments across diverse VLP models,
multiple benchmark datasets, and a variety of downstream tasks demonstrate that
MAA significantly enhances the effectiveness and transferability of adversarial
attacks. A large cohort of performance studies is conducted to generate
insights into the effectiveness of various model configurations, guiding future
advancements in this domain.


http://arxiv.org/abs/2502.07753
Direct Ascent Synthesis: Revealing Hidden Generative Capabilities in Discriminative Models. (68%)
Stanislav Fort; Jonathan Whitaker
  We demonstrate that discriminative models inherently contain powerful
generative capabilities, challenging the fundamental distinction between
discriminative and generative architectures. Our method, Direct Ascent
Synthesis (DAS), reveals these latent capabilities through multi-resolution
optimization of CLIP model representations. While traditional inversion
attempts produce adversarial patterns, DAS achieves high-quality image
synthesis by decomposing optimization across multiple spatial scales (1x1 to
224x224), requiring no additional training. This approach not only enables
diverse applications -- from text-to-image generation to style transfer -- but
maintains natural image statistics ($1/f^2$ spectrum) and guides the generation
away from non-robust adversarial patterns. Our results demonstrate that
standard discriminative models encode substantially richer generative knowledge
than previously recognized, providing new perspectives on model
interpretability and the relationship between adversarial examples and natural
image synthesis.


http://arxiv.org/abs/2502.07557
JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation. (41%)
Shenyi Zhang; Yuchen Zhai; Keyan Guo; Hongxin Hu; Shengnan Guo; Zheng Fang; Lingchen Zhao; Chao Shen; Cong Wang; Qian Wang
  Despite the implementation of safety alignment strategies, large language
models (LLMs) remain vulnerable to jailbreak attacks, which undermine these
safety guardrails and pose significant security threats. Some defenses have
been proposed to detect or mitigate jailbreaks, but they are unable to
withstand the test of time due to an insufficient understanding of jailbreak
mechanisms. In this work, we investigate the mechanisms behind jailbreaks based
on the Linear Representation Hypothesis (LRH), which states that neural
networks encode high-level concepts as subspaces in their hidden
representations. We define the toxic semantics in harmful and jailbreak prompts
as toxic concepts and describe the semantics in jailbreak prompts that
manipulate LLMs to comply with unsafe requests as jailbreak concepts. Through
concept extraction and analysis, we reveal that LLMs can recognize the toxic
concepts in both harmful and jailbreak prompts. However, unlike harmful
prompts, jailbreak prompts activate the jailbreak concepts and alter the LLM
output from rejection to compliance. Building on our analysis, we propose a
comprehensive jailbreak defense framework, JBShield, consisting of two key
components: jailbreak detection JBShield-D and mitigation JBShield-M.
JBShield-D identifies jailbreak prompts by determining whether the input
activates both toxic and jailbreak concepts. When a jailbreak prompt is
detected, JBShield-M adjusts the hidden representations of the target LLM by
enhancing the toxic concept and weakening the jailbreak concept, ensuring LLMs
produce safe content. Extensive experiments demonstrate the superior
performance of JBShield, achieving an average detection accuracy of 0.95 and
reducing the average attack success rate of various jailbreak attacks to 2%
from 61% across distinct LLMs.


http://arxiv.org/abs/2502.07783
Curvature Tuning: Provable Training-free Model Steering From a Single Parameter. (1%)
Leyang Hu; Randall Balestriero
  The scaling of model size and data size has reshaped the paradigm of AI. As a
result, the common protocol to leverage the latest models is to steer them
towards a specific downstream task of interest through {\em fine-tuning}.
Despite its importance, the main methods for fine-tuning remain limited to full
or low-rank adapters--containing countless hyper-parameters and lacking
interpretability. In this paper, we take a step back and demonstrate how novel
and explainable post-training steering solutions can be derived theoretically
from {\em spline operators}, a rich mathematical framing of Deep Networks that
was recently developed. Our method--coined \textbf{Curvature Tuning (CT)}--has
a single parameter that provably modulates the curvature of the model's
decision boundary henceforth allowing training-free steering. This makes CT
both more efficient and interpretable than conventional fine-tuning methods. We
empirically validate its effectiveness in improving generalization and
robustness of pretrained models. For example, CT improves out-of-distribution
transfer performances of ResNet-18/50 by 2.57\%/1.74\% across seventeen
downstream datasets, and improves RobustBench robust accuracy by
11.76\%/348.44\%. Additionally, we apply CT to ReLU-based Swin-T/S, improving
their generalization on nine downstream datasets by 2.43\%/3.33\%. Our code is
available at
\href{https://github.com/Leon-Leyang/curvature-tuning}{https://github.com/Leon-Leyang/curvature-tuning}.


http://arxiv.org/abs/2502.07845
Spread them Apart: Towards Robust Watermarking of Generated Content. (1%)
Mikhail Pautov; Danil Ivanov; Andrey V. Galichin; Oleg Rogov; Ivan Oseledets
  Generative models that can produce realistic images have improved
significantly in recent years. The quality of the generated content has
increased drastically, so sometimes it is very difficult to distinguish between
the real images and the generated ones. Such an improvement comes at a price of
ethical concerns about the usage of the generative models: the users of
generative models can improperly claim ownership of the generated content
protected by a license. In this paper, we propose an approach to embed
watermarks into the generated content to allow future detection of the
generated content and identification of the user who generated it. The
watermark is embedded during the inference of the model, so the proposed
approach does not require the retraining of the latter. We prove that
watermarks embedded are guaranteed to be robust against additive perturbations
of a bounded magnitude. We apply our method to watermark diffusion models and
show that it matches state-of-the-art watermarking schemes in terms of
robustness to different types of synthetic watermark removal attacks.


http://arxiv.org/abs/2502.08055
SLVR: Securely Leveraging Client Validation for Robust Federated Learning. (1%)
Jihye Choi; Sai Rahul Rachuri; Ke Wang; Somesh Jha; Yizhen Wang
  Federated Learning (FL) enables collaborative model training while keeping
client data private. However, exposing individual client updates makes FL
vulnerable to reconstruction attacks. Secure aggregation mitigates such privacy
risks but prevents the server from verifying the validity of each client
update, creating a privacy-robustness tradeoff. Recent efforts attempt to
address this tradeoff by enforcing checks on client updates using
zero-knowledge proofs, but they support limited predicates and often depend on
public validation data. We propose SLVR, a general framework that securely
leverages clients' private data through secure multi-party computation. By
utilizing clients' data, SLVR not only eliminates the need for public
validation data, but also enables a wider range of checks for robustness,
including cross-client accuracy validation. It also adapts naturally to
distribution shifts in client data as it can securely refresh its validation
data up-to-date. Our empirical evaluations show that SLVR improves robustness
against model poisoning attacks, particularly outperforming existing methods by
up to 50% under adaptive attacks. Additionally, SLVR demonstrates effective
adaptability and stable convergence under various distribution shift scenarios.


http://arxiv.org/abs/2502.07821
Amnesia as a Catalyst for Enhancing Black Box Pixel Attacks in Image Classification and Object Detection. (98%)
Dongsu Song; Daehwa Ko; Jay Hoon Jung
  It is well known that query-based attacks tend to have relatively higher
success rates in adversarial black-box attacks. While research on black-box
attacks is actively being conducted, relatively few studies have focused on
pixel attacks that target only a limited number of pixels. In image
classification, query-based pixel attacks often rely on patches, which heavily
depend on randomness and neglect the fact that scattered pixels are more
suitable for adversarial attacks. Moreover, to the best of our knowledge,
query-based pixel attacks have not been explored in the field of object
detection. To address these issues, we propose a novel pixel-based black-box
attack called Remember and Forget Pixel Attack using Reinforcement
Learning(RFPAR), consisting of two main components: the Remember and Forget
processes. RFPAR mitigates randomness and avoids patch dependency by leveraging
rewards generated through a one-step RL algorithm to perturb pixels. RFPAR
effectively creates perturbed images that minimize the confidence scores while
adhering to limited pixel constraints. Furthermore, we advance our proposed
attack beyond image classification to object detection, where RFPAR reduces the
confidence scores of detected objects to avoid detection. Experiments on the
ImageNet-1K dataset for classification show that RFPAR outperformed
state-of-the-art query-based pixel attacks. For object detection, using the
MSCOCO dataset with YOLOv8 and DDQ, RFPAR demonstrates comparable mAP reduction
to state-of-the-art query-based attack while requiring fewer query. Further
experiments on the Argoverse dataset using YOLOv8 confirm that RFPAR
effectively removed objects on a larger scale dataset. Our code is available at
https://github.com/KAU-QuantumAILab/RFPAR.


http://arxiv.org/abs/2502.07225
CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models. (93%)
Sen Peng; Mingyue Wang; Jianfei He; Jijia Yang; Xiaohua Jia
  Latent diffusion models have recently demonstrated superior capabilities in
many downstream image synthesis tasks. However, customization of latent
diffusion models using unauthorized data can severely compromise the privacy
and intellectual property rights of data owners. Adversarial examples as
protective perturbations have been developed to defend against unauthorized
data usage by introducing imperceptible noise to customization samples,
preventing diffusion models from effectively learning them. In this paper, we
first reveal that the primary reason adversarial examples are effective as
protective perturbations in latent diffusion models is the distortion of their
latent representations, as demonstrated through qualitative and quantitative
experiments. We then propose the Contrastive Adversarial Training (CAT)
utilizing lightweight adapters as an adaptive attack against these protection
methods, highlighting their lack of robustness. Extensive experiments
demonstrate that our CAT method significantly reduces the effectiveness of
protective perturbations in customization, urging the community to reconsider
and improve the robustness of existing protective perturbations. The code is
available at https://github.com/senp98/CAT.


http://arxiv.org/abs/2502.07011
DROP: Poison Dilution via Knowledge Distillation for Federated Learning. (92%)
Georgios Syros; Anshuman Suri; Farinaz Koushanfar; Cristina Nita-Rotaru; Alina Oprea
  Federated Learning is vulnerable to adversarial manipulation, where malicious
clients can inject poisoned updates to influence the global model's behavior.
While existing defense mechanisms have made notable progress, they fail to
protect against adversaries that aim to induce targeted backdoors under
different learning and attack configurations. To address this limitation, we
introduce DROP (Distillation-based Reduction Of Poisoning), a novel defense
mechanism that combines clustering and activity-tracking techniques with
extraction of benign behavior from clients via knowledge distillation to tackle
stealthy adversaries that manipulate low data poisoning rates and diverse
malicious client ratios within the federation. Through extensive
experimentation, our approach demonstrates superior robustness compared to
existing defenses across a wide range of learning configurations. Finally, we
evaluate existing defenses and our method under the challenging setting of
non-IID client data distribution and highlight the challenges of designing a
resilient FL defense in this setting.


http://arxiv.org/abs/2502.07101
SMAB: MAB based word Sensitivity Estimation Framework and its Applications in Adversarial Text Generation. (83%)
Saurabh Kumar Pandey; Sachin Vashistha; Debrup Das; Somak Aditya; Monojit Choudhury
  To understand the complexity of sequence classification tasks, Hahn et al.
(2021) proposed sensitivity as the number of disjoint subsets of the input
sequence that can each be individually changed to change the output. Though
effective, calculating sensitivity at scale using this framework is costly
because of exponential time complexity. Therefore, we introduce a
Sensitivity-based Multi-Armed Bandit framework (SMAB), which provides a
scalable approach for calculating word-level local (sentence-level) and global
(aggregated) sensitivities concerning an underlying text classifier for any
dataset. We establish the effectiveness of our approach through various
applications. We perform a case study on CHECKLIST generated sentiment analysis
dataset where we show that our algorithm indeed captures intuitively high and
low-sensitive words. Through experiments on multiple tasks and languages, we
show that sensitivity can serve as a proxy for accuracy in the absence of gold
data. Lastly, we show that guiding perturbation prompts using sensitivity
values in adversarial example generation improves attack success rate by
15.58%, whereas using sensitivity as an additional reward in adversarial
paraphrase generation gives a 12.00% improvement over SOTA approaches. Warning:
Contains potentially offensive content.


http://arxiv.org/abs/2502.06917
Krum Federated Chain (KFC): Using blockchain to defend against adversarial attacks in Federated Learning. (68%)
Mario García-Márquez; Nuria Rodríguez-Barroso; M. Victoria Luzón; Francisco Herrera
  Federated Learning presents a nascent approach to machine learning, enabling
collaborative model training across decentralized devices while safeguarding
data privacy. However, its distributed nature renders it susceptible to
adversarial attacks. Integrating blockchain technology with Federated Learning
offers a promising avenue to enhance security and integrity. In this paper, we
tackle the potential of blockchain in defending Federated Learning against
adversarial attacks. First, we test Proof of Federated Learning, a well known
consensus mechanism designed ad-hoc to federated contexts, as a defense
mechanism demonstrating its efficacy against Byzantine and backdoor attacks
when at least one miner remains uncompromised. Second, we propose Krum
Federated Chain, a novel defense strategy combining Krum and Proof of Federated
Learning, valid to defend against any configuration of Byzantine or backdoor
attacks, even when all miners are compromised. Our experiments conducted on
image classification datasets validate the effectiveness of our proposed
approaches.


http://arxiv.org/abs/2502.06390
When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs. (8%)
Aobotao Dai; Xinyu Ma; Lei Chen; Songze Li; Lin Wang
  Vision-Language Models (VLMs) have gained considerable prominence in recent
years due to their remarkable capability to effectively integrate and process
both textual and visual information. This integration has significantly
enhanced performance across a diverse spectrum of applications, such as scene
perception and robotics. However, the deployment of VLMs has also given rise to
critical safety and security concerns, necessitating extensive research to
assess the potential vulnerabilities these VLM systems may harbor. In this
work, we present an in-depth survey of the attack strategies tailored for VLMs.
We categorize these attacks based on their underlying objectives - namely
jailbreak, camouflage, and exploitation - while also detailing the various
methodologies employed for data manipulation of VLMs. Meanwhile, we outline
corresponding defense mechanisms that have been proposed to mitigate these
vulnerabilities. By discerning key connections and distinctions among the
diverse types of attacks, we propose a compelling taxonomy for VLM attacks.
Moreover, we summarize the evaluation metrics that comprehensively describe the
characteristics and impact of different attacks on VLMs. Finally, we conclude
with a discussion of promising future research directions that could further
enhance the robustness and safety of VLMs, emphasizing the importance of
ongoing exploration in this critical area of study. To facilitate community
engagement, we maintain an up-to-date project page, accessible at:
https://github.com/AobtDai/VLM_Attack_Paper_List.


http://arxiv.org/abs/2502.06418
Robust Watermarks Leak: Channel-Aware Feature Extraction Enables Adversarial Watermark Manipulation. (3%)
Zhongjie Ba; Yitao Zhang; Peng Cheng; Bin Gong; Xinyu Zhang; Qinglong Wang; Kui Ren
  Watermarking plays a key role in the provenance and detection of AI-generated
content. While existing methods prioritize robustness against real-world
distortions (e.g., JPEG compression and noise addition), we reveal a
fundamental tradeoff: such robust watermarks inherently improve the redundancy
of detectable patterns encoded into images, creating exploitable information
leakage. To leverage this, we propose an attack framework that extracts leakage
of watermark patterns through multi-channel feature learning using a
pre-trained vision model. Unlike prior works requiring massive data or detector
access, our method achieves both forgery and detection evasion with a single
watermarked image. Extensive experiments demonstrate that our method achieves a
60\% success rate gain in detection evasion and 51\% improvement in forgery
accuracy compared to state-of-the-art methods while maintaining visual
fidelity. Our work exposes the robustness-stealthiness paradox: current
"robust" watermarks sacrifice security for distortion resistance, providing
insights for future watermark design.


http://arxiv.org/abs/2502.06892
Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attacks. (76%)
Bowei He; Lihao Yin; Hui-Ling Zhen; Jianping Zhang; Lanqing Hong; Mingxuan Yuan; Chen Ma
  The widespread deployment of pre-trained language models (PLMs) has exposed
them to textual backdoor attacks, particularly those planted during the
pre-training stage. These attacks pose significant risks to high-reliability
applications, as they can stealthily affect multiple downstream tasks. While
certifying robustness against such threats is crucial, existing defenses
struggle with the high-dimensional, interdependent nature of textual data and
the lack of access to original poisoned pre-training data. To address these
challenges, we introduce \textbf{F}uzzed \textbf{R}andomized \textbf{S}moothing
(\textbf{FRS}), a novel approach for efficiently certifying language model
robustness against backdoor attacks. FRS integrates software robustness
certification techniques with biphased model parameter smoothing, employing
Monte Carlo tree search for proactive fuzzing to identify vulnerable textual
segments within the Damerau-Levenshtein space. This allows for targeted and
efficient text randomization, while eliminating the need for access to poisoned
training data during model smoothing. Our theoretical analysis demonstrates
that FRS achieves a broader certified robustness radius compared to existing
methods. Extensive experiments across various datasets, model configurations,
and attack strategies validate FRS's superiority in terms of defense
efficiency, accuracy, and robustness.


http://arxiv.org/abs/2502.05966
Detection of Physiological Data Tampering Attacks with Quantum Machine Learning. (41%)
Md. Saif Hassan Onim; Himanshu Thapliyal
  The widespread use of cloud-based medical devices and wearable sensors has
made physiological data susceptible to tampering. These attacks can compromise
the reliability of healthcare systems which can be critical and
life-threatening. Detection of such data tampering is of immediate need.
Machine learning has been used to detect anomalies in datasets but the
performance of Quantum Machine Learning (QML) is still yet to be evaluated for
physiological sensor data. Thus, our study compares the effectiveness of QML
for detecting physiological data tampering, focusing on two types of white-box
attacks: data poisoning and adversarial perturbation. The results show that QML
models are better at identifying label-flipping attacks, achieving accuracy
rates of 75%-95% depending on the data and attack severity. This superior
performance is due to the ability of quantum algorithms to handle complex and
high-dimensional data. However, both QML and classical models struggle to
detect more sophisticated adversarial perturbation attacks, which subtly alter
data without changing its statistical properties. Although QML performed poorly
against this attack with around 45%-65% accuracy, it still outperformed
classical algorithms in some cases.


http://arxiv.org/abs/2502.05931
Protecting Intellectual Property of EEG-based Neural Networks with Watermarking. (12%)
Ahmed Abdelaziz; Ahmed Fathi; Ahmed Fares
  EEG-based neural networks, pivotal in medical diagnosis and brain-computer
interfaces, face significant intellectual property (IP) risks due to their
reliance on sensitive neurophysiological data and resource-intensive
development. Current watermarking methods, particularly those using abstract
trigger sets, lack robust authentication and fail to address the unique
challenges of EEG models. This paper introduces a cryptographic wonder
filter-based watermarking framework tailored for EEG-based neural networks.
Leveraging collision-resistant hashing and public-key encryption, the wonder
filter embeds the watermark during training, ensuring minimal distortion ($\leq
5\%$ drop in EEG task accuracy) and high reliability (100\% watermark
detection). The framework is rigorously evaluated against adversarial attacks,
including fine-tuning, transfer learning, and neuron pruning. Results
demonstrate persistent watermark retention, with classification accuracy for
watermarked states remaining above 90\% even after aggressive pruning, while
primary task performance degrades faster, deterring removal attempts. Piracy
resistance is validated by the inability to embed secondary watermarks without
severe accuracy loss ( $>10\%$ in EEGNet and CCNN models). Cryptographic
hashing ensures authentication, reducing brute-force attack success
probabilities. Evaluated on the DEAP dataset across models (CCNN, EEGNet,
TSception), the method achieves $>99.4\%$ null-embedding accuracy, effectively
eliminating false positives. By integrating wonder filters with EEG-specific
adaptations, this work bridges a critical gap in IP protection for
neurophysiological models, offering a secure, tamper-proof solution for
healthcare and biometric applications. The framework's robustness against
adversarial modifications underscores its potential to safeguard sensitive EEG
models while maintaining diagnostic utility.


http://arxiv.org/abs/2502.05954
Optimization under Attack: Resilience, Vulnerability, and the Path to Collapse. (1%)
Amal Aldawsari; Evangelos Pournaras
  Optimization is instrumental for improving operations of large-scale
socio-technical infrastructures of Smart Cities, for instance, energy and
traffic systems. In particular, understanding the performance of multi-agent
discrete-choice combinatorial optimization under distributed adversary attacks
is a compelling and underexplored problem, since multi-agent systems exhibit a
large number of remote control variables that can influence in an unprecedented
way the cost-effectiveness of distributed optimization heuristics. This paper
unravels for the first time the trajectories of distributed optimization from
resilience to vulnerability, and finally to collapse under varying adversary
influence. Using real-world data to emulate over 28 billion multi-agent
optimization scenarios, we exhaustively assess how the number of agents with
different adversarial severity and network positioning influences optimization
performance, including the influence on Pareto optimal points. With this novel
large-scale dataset, made openly available as a benchmark, we disentangle how
optimization remains resilient to adversaries and which adversary conditions
are required to make optimization vulnerable or collapsed. These new findings
can provide new insights for designing self-healing strategies for
fault-tolerance and fault-correction in adversarial distributed optimization
that have been missing so far.


http://arxiv.org/abs/2502.05542
Democratic Training Against Universal Adversarial Perturbations. (99%)
Bing Sun; Jun Sun; Wei Zhao
  Despite their advances and success, real-world deep neural networks are known
to be vulnerable to adversarial attacks. Universal adversarial perturbation, an
input-agnostic attack, poses a serious threat for them to be deployed in
security-sensitive systems. In this case, a single universal adversarial
perturbation deceives the model on a range of clean inputs without requiring
input-specific optimization, which makes it particularly threatening. In this
work, we observe that universal adversarial perturbations usually lead to
abnormal entropy spectrum in hidden layers, which suggests that the prediction
is dominated by a small number of ``feature'' in such cases (rather than
democratically by many features). Inspired by this, we propose an efficient yet
effective defense method for mitigating UAPs called \emph{Democratic Training}
by performing entropy-based model enhancement to suppress the effect of the
universal adversarial perturbations in a given model. \emph{Democratic
Training} is evaluated with 7 neural networks trained on 5 benchmark datasets
and 5 types of state-of-the-art universal adversarial attack methods. The
results show that it effectively reduces the attack success rate, improves
model robustness and preserves the model accuracy on clean samples.


http://arxiv.org/abs/2502.05509
Do Spikes Protect Privacy? Investigating Black-Box Model Inversion Attacks in Spiking Neural Networks. (89%)
Hamed Poursiami; Ayana Moshruba; Maryam Parsa
  As machine learning models become integral to security-sensitive
applications, concerns over data leakage from adversarial attacks continue to
rise. Model Inversion (MI) attacks pose a significant privacy threat by
enabling adversaries to reconstruct training data from model outputs. While MI
attacks on Artificial Neural Networks (ANNs) have been widely studied, Spiking
Neural Networks (SNNs) remain largely unexplored in this context. Due to their
event-driven and discrete computations, SNNs introduce fundamental differences
in information processing that may offer inherent resistance to such attacks. A
critical yet underexplored aspect of this threat lies in black-box settings,
where attackers operate through queries without direct access to model
parameters or gradients-representing a more realistic adversarial scenario in
deployed systems. This work presents the first study of black-box MI attacks on
SNNs. We adapt a generative adversarial MI framework to the spiking domain by
incorporating rate-based encoding for input transformation and decoding
mechanisms for output interpretation. Our results show that SNNs exhibit
significantly greater resistance to MI attacks than ANNs, as demonstrated by
degraded reconstructions, increased instability in attack convergence, and
overall reduced attack effectiveness across multiple evaluation metrics.
Further analysis suggests that the discrete and temporally distributed nature
of SNN decision boundaries disrupts surrogate modeling, limiting the attacker's
ability to approximate the target model.


http://arxiv.org/abs/2502.05669
Rigid Body Adversarial Attacks. (81%)
Aravind Ramakrishnan; David I. W. Levin; Alec Jacobson
  Due to their performance and simplicity, rigid body simulators are often used
in applications where the objects of interest can considered very stiff.
However, no material has infinite stiffness, which means there are potentially
cases where the non-zero compliance of the seemingly rigid object can cause a
significant difference between its trajectories when simulated in a rigid body
or deformable simulator.
  Similarly to how adversarial attacks are developed against image classifiers,
we propose an adversarial attack against rigid body simulators. In this
adversarial attack, we solve an optimization problem to construct perceptually
rigid adversarial objects that have the same collision geometry and moments of
mass to a reference object, so that they behave identically in rigid body
simulations but maximally different in more accurate deformable simulations. We
demonstrate the validity of our method by comparing simulations of several
examples in commercially available simulators.


http://arxiv.org/abs/2502.05772
Effective Black-Box Multi-Faceted Attacks Breach Vision Large Language Model Guardrails. (64%)
Yijun Yang; Lichao Wang; Xiao Yang; Lanqing Hong; Jun Zhu
  Vision Large Language Models (VLLMs) integrate visual data processing,
expanding their real-world applications, but also increasing the risk of
generating unsafe responses. In response, leading companies have implemented
Multi-Layered safety defenses, including alignment training, safety system
prompts, and content moderation. However, their effectiveness against
sophisticated adversarial attacks remains largely unexplored. In this paper, we
propose MultiFaceted Attack, a novel attack framework designed to
systematically bypass Multi-Layered Defenses in VLLMs. It comprises three
complementary attack facets: Visual Attack that exploits the multimodal nature
of VLLMs to inject toxic system prompts through images; Alignment Breaking
Attack that manipulates the model's alignment mechanism to prioritize the
generation of contrasting responses; and Adversarial Signature that deceives
content moderators by strategically placing misleading information at the end
of the response. Extensive evaluations on eight commercial VLLMs in a black-box
setting demonstrate that MultiFaceted Attack achieves a 61.56% attack success
rate, surpassing state-of-the-art methods by at least 42.18%.


http://arxiv.org/abs/2502.05637
Adversarial Machine Learning: Attacks, Defenses, and Open Challenges. (61%)
Pranav K Jha
  Adversarial Machine Learning (AML) addresses vulnerabilities in AI systems
where adversaries manipulate inputs or training data to degrade performance.
This article provides a comprehensive analysis of evasion and poisoning
attacks, formalizes defense mechanisms with mathematical rigor, and discusses
the challenges of implementing robust solutions in adaptive threat models.
Additionally, it highlights open challenges in certified robustness,
scalability, and real-world deployment.


http://arxiv.org/abs/2502.05755
Filter, Obstruct and Dilute: Defending Against Backdoor Attacks on Semi-Supervised Learning. (41%)
Xinrui Wang; Chuanxing Geng; Wenhai Wan; Shao-yuan Li; Songcan Chen
  Recent studies have verified that semi-supervised learning (SSL) is
vulnerable to data poisoning backdoor attacks. Even a tiny fraction of
contaminated training data is sufficient for adversaries to manipulate up to
90\% of the test outputs in existing SSL methods. Given the emerging threat of
backdoor attacks designed for SSL, this work aims to protect SSL against such
risks, marking it as one of the few known efforts in this area. Specifically,
we begin by identifying that the spurious correlations between the backdoor
triggers and the target class implanted by adversaries are the primary cause of
manipulated model predictions during the test phase. To disrupt these
correlations, we utilize three key techniques: Gaussian Filter, complementary
learning and trigger mix-up, which collectively filter, obstruct and dilute the
influence of backdoor attacks in both data pre-processing and feature learning.
Experimental results demonstrate that our proposed method, Backdoor Invalidator
(BI), significantly reduces the average attack success rate from 84.7\% to
1.8\% across different state-of-the-art backdoor attacks. It is also worth
mentioning that BI does not sacrifice accuracy on clean data and is supported
by a theoretical guarantee of its generalization capability.


http://arxiv.org/abs/2502.05547
Dual Defense: Enhancing Privacy and Mitigating Poisoning Attacks in Federated Learning. (4%)
Runhua Xu; Shiqi Gao; Chao Li; James Joshi; Jianxin Li
  Federated learning (FL) is inherently susceptible to privacy breaches and
poisoning attacks. To tackle these challenges, researchers have separately
devised secure aggregation mechanisms to protect data privacy and robust
aggregation methods that withstand poisoning attacks. However, simultaneously
addressing both concerns is challenging; secure aggregation facilitates
poisoning attacks as most anomaly detection techniques require access to
unencrypted local model updates, which are obscured by secure aggregation. Few
recent efforts to simultaneously tackle both challenges offen depend on
impractical assumption of non-colluding two-server setups that disrupt FL's
topology, or three-party computation which introduces scalability issues,
complicating deployment and application. To overcome this dilemma, this paper
introduce a Dual Defense Federated learning (DDFed) framework. DDFed
simultaneously boosts privacy protection and mitigates poisoning attacks,
without introducing new participant roles or disrupting the existing FL
topology. DDFed initially leverages cutting-edge fully homomorphic encryption
(FHE) to securely aggregate model updates, without the impractical requirement
for non-colluding two-server setups and ensures strong privacy protection.
Additionally, we proposes a unique two-phase anomaly detection mechanism for
encrypted model updates, featuring secure similarity computation and
feedback-driven collaborative selection, with additional measures to prevent
potential privacy breaches from Byzantine clients incorporated into the
detection process. We conducted extensive experiments on various model
poisoning attacks and FL scenarios, including both cross-device and cross-silo
FL. Experiments on publicly available datasets demonstrate that DDFed
successfully protects model privacy and effectively defends against model
poisoning threats.


http://arxiv.org/abs/2502.05727
Impact of Data Poisoning Attacks on Feasibility and Optimality of Neural Power System Optimizers. (2%)
Nora Agah; Meiyi Li; Javad Mohammadi
  The increased integration of clean yet stochastic energy resources and the
growing number of extreme weather events are narrowing the decision-making
window of power grid operators. This time constraint is fueling a plethora of
research on Machine Learning-, or ML-, based optimization proxies. While
finding a fast solution is appealing, the inherent vulnerabilities of the
learning-based methods are hindering their adoption. One of these
vulnerabilities is data poisoning attacks, which adds perturbations to ML
training data, leading to incorrect decisions. The impact of poisoning attacks
on learning-based power system optimizers have not been thoroughly studied,
which creates a critical vulnerability. In this paper, we examine the impact of
data poisoning attacks on ML-based optimization proxies that are used to solve
the DC Optimal Power Flow problem. Specifically, we compare the resilience of
three different methods-a penalty-based method, a post-repair approach, and a
direct mapping approach-against the adverse effects of poisoning attacks. We
will use the optimality and feasibility of these proxies as performance
metrics. The insights of this work will establish a foundation for enhancing
the resilience of neural power system optimizers.


http://arxiv.org/abs/2502.06872
Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey. (1%)
Bo Ni; Zheyuan Liu; Leyao Wang; Yongjia Lei; Yuying Zhao; Xueqi Cheng; Qingkai Zeng; Luna Dong; Yinglong Xia; Krishnaram Kenthapadi; Ryan Rossi; Franck Dernoncourt; Md Mehrab Tanjim; Nesreen Ahmed; Xiaorui Liu; Wenqi Fan; Erik Blasch; Yu Wang; Meng Jiang; Tyler Derr
  Retrieval-Augmented Generation (RAG) is an advanced technique designed to
address the challenges of Artificial Intelligence-Generated Content (AIGC). By
integrating context retrieval into content generation, RAG provides reliable
and up-to-date external knowledge, reduces hallucinations, and ensures relevant
context across a wide range of tasks. However, despite RAG's success and
potential, recent studies have shown that the RAG paradigm also introduces new
risks, including robustness issues, privacy concerns, adversarial attacks, and
accountability issues. Addressing these risks is critical for future
applications of RAG systems, as they directly impact their trustworthiness.
Although various methods have been developed to improve the trustworthiness of
RAG methods, there is a lack of a unified perspective and framework for
research in this topic. Thus, in this paper, we aim to address this gap by
providing a comprehensive roadmap for developing trustworthy RAG systems. We
place our discussion around five key perspectives: reliability, privacy,
safety, fairness, explainability, and accountability. For each perspective, we
present a general framework and taxonomy, offering a structured approach to
understanding the current challenges, evaluating existing solutions, and
identifying promising future research directions. To encourage broader adoption
and innovation, we also highlight the downstream applications where trustworthy
RAG systems have a significant impact.


http://arxiv.org/abs/2502.04679
Mechanistic Understandings of Representation Vulnerabilities and Engineering Robust Vision Transformers. (99%)
Chashi Mahiul Islam; Samuel Jacob Chacko; Mao Nishino; Xiuwen Liu
  While transformer-based models dominate NLP and vision applications, their
underlying mechanisms to map the input space to the label space semantically
are not well understood. In this paper, we study the sources of known
representation vulnerabilities of vision transformers (ViT), where perceptually
identical images can have very different representations and semantically
unrelated images can have the same representation. Our analysis indicates that
imperceptible changes to the input can result in significant representation
changes, particularly in later layers, suggesting potential instabilities in
the performance of ViTs. Our comprehensive study reveals that adversarial
effects, while subtle in early layers, propagate and amplify through the
network, becoming most pronounced in middle to late layers. This insight
motivates the development of NeuroShield-ViT, a novel defense mechanism that
strategically neutralizes vulnerable neurons in earlier layers to prevent the
cascade of adversarial effects. We demonstrate NeuroShield-ViT's effectiveness
across various attacks, particularly excelling against strong iterative
attacks, and showcase its remarkable zero-shot generalization capabilities.
Without fine-tuning, our method achieves a competitive accuracy of 77.8% on
adversarial examples, surpassing conventional robustness methods. Our results
shed new light on how adversarial effects propagate through ViT layers, while
providing a promising approach to enhance the robustness of vision transformers
against adversarial attacks. Additionally, they provide a promising approach to
enhance the robustness of vision transformers against adversarial attacks.


http://arxiv.org/abs/2502.05041
Federated Learning for Anomaly Detection in Energy Consumption Data: Assessing the Vulnerability to Adversarial Attacks. (99%)
Yohannis Kifle Telila; Damitha Senevirathne; Dumindu Tissera; Apurva Narayan; Miriam A. M. Capretz; Katarina Grolinger
  Anomaly detection is crucial in the energy sector to identify irregular
patterns indicating equipment failures, energy theft, or other issues. Machine
learning techniques for anomaly detection have achieved great success, but are
typically centralized, involving sharing local data with a central server which
raises privacy and security concerns. Federated Learning (FL) has been gaining
popularity as it enables distributed learning without sharing local data.
However, FL depends on neural networks, which are vulnerable to adversarial
attacks that manipulate data, leading models to make erroneous predictions.
While adversarial attacks have been explored in the image domain, they remain
largely unexplored in time series problems, especially in the energy domain.
Moreover, the effect of adversarial attacks in the FL setting is also mostly
unknown. This paper assesses the vulnerability of FL-based anomaly detection in
energy data to adversarial attacks. Specifically, two state-of-the-art models,
Long Short Term Memory (LSTM) and Transformers, are used to detect anomalies in
an FL setting, and two white-box attack methods, Fast Gradient Sign Method
(FGSM) and Projected Gradient Descent (PGD), are employed to perturb the data.
The results show that FL is more sensitive to PGD attacks than to FGSM attacks,
attributed to PGD's iterative nature, resulting in an accuracy drop of over 10%
even with naive, weaker attacks. Moreover, FL is more affected by these attacks
than centralized learning, highlighting the need for defense mechanisms in FL.


http://arxiv.org/abs/2502.05374
Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond. (75%)
Chongyu Fan; Jinghan Jia; Yihua Zhang; Anil Ramakrishna; Mingyi Hong; Sijia Liu
  The LLM unlearning technique has recently been introduced to comply with data
regulations and address the safety and ethical concerns of LLMs by removing the
undesired data-model influence. However, state-of-the-art unlearning methods
face a critical vulnerability: they are susceptible to ``relearning'' the
removed information from a small number of forget data points, known as
relearning attacks. In this paper, we systematically investigate how to make
unlearned models robust against such attacks. For the first time, we establish
a connection between robust unlearning and sharpness-aware minimization (SAM)
through a unified robust optimization framework, in an analogy to adversarial
training designed to defend against adversarial attacks. Our analysis for SAM
reveals that smoothness optimization plays a pivotal role in mitigating
relearning attacks. Thus, we further explore diverse smoothing strategies to
enhance unlearning robustness. Extensive experiments on benchmark datasets,
including WMDP and MUSE, demonstrate that SAM and other smoothness optimization
approaches consistently improve the resistance of LLM unlearning to relearning
attacks. Notably, smoothness-enhanced unlearning also helps defend against
(input-level) jailbreaking attacks, broadening our proposal's impact in
robustifying LLM unlearning. Codes are available at
https://github.com/OPTML-Group/Unlearn-Smooth.


http://arxiv.org/abs/2502.05000
Robust Graph Learning Against Adversarial Evasion Attacks via Prior-Free Diffusion-Based Structure Purification. (74%)
Jiayi Luo; Qingyun Sun; Haonan Yuan; Xingcheng Fu; Jianxin Li
  Adversarial evasion attacks pose significant threats to graph learning, with
lines of studies that have improved the robustness of Graph Neural Networks
(GNNs). However, existing works rely on priors about clean graphs or attacking
strategies, which are often heuristic and inconsistent. To achieve robust graph
learning over different types of evasion attacks and diverse datasets, we
investigate this problem from a prior-free structure purification perspective.
Specifically, we propose a novel Diffusion-based Structure Purification
framework named DiffSP, which creatively incorporates the graph diffusion model
to learn intrinsic distributions of clean graphs and purify the perturbed
structures by removing adversaries under the direction of the captured
predictive patterns without relying on priors. DiffSP is divided into the
forward diffusion process and the reverse denoising process, during which
structure purification is achieved. To avoid valuable information loss during
the forward process, we propose an LID-driven nonisotropic diffusion mechanism
to selectively inject noise anisotropically. To promote semantic alignment
between the clean graph and the purified graph generated during the reverse
process, we reduce the generation uncertainty by the proposed graph transfer
entropy guided denoising mechanism. Extensive experiments demonstrate the
superior robustness of DiffSP against evasion attacks.


http://arxiv.org/abs/2502.05174
MELON: Indirect Prompt Injection Defense via Masked Re-execution and Tool Comparison. (50%)
Kaijie Zhu; Xianjun Yang; Jindong Wang; Wenbo Guo; William Yang Wang
  Recent research has explored that LLM agents are vulnerable to indirect
prompt injection (IPI) attacks, where malicious tasks embedded in
tool-retrieved information can redirect the agent to take unauthorized actions.
Existing defenses against IPI have significant limitations: either require
essential model training resources, lack effectiveness against sophisticated
attacks, or harm the normal utilities. We present MELON (Masked re-Execution
and TooL comparisON), a novel IPI defense. Our approach builds on the
observation that under a successful attack, the agent's next action becomes
less dependent on user tasks and more on malicious tasks. Following this, we
design MELON to detect attacks by re-executing the agent's trajectory with a
masked user prompt modified through a masking function. We identify an attack
if the actions generated in the original and masked executions are similar. We
also include three key designs to reduce the potential false positives and
false negatives. Extensive evaluation on the IPI benchmark AgentDojo
demonstrates that MELON outperforms SOTA defenses in both attack prevention and
utility preservation. Moreover, we show that combining MELON with a SOTA prompt
augmentation defense (denoted as MELON-Aug) further improves its performance.
We also conduct a detailed ablation study to validate our key designs.


http://arxiv.org/abs/2502.07807
CP-Guard+: A New Paradigm for Malicious Agent Detection and Defense in Collaborative Perception. (11%)
Senkang Hu; Yihang Tao; Zihan Fang; Guowen Xu; Yiqin Deng; Sam Kwong; Yuguang Fang
  Collaborative perception (CP) is a promising method for safe connected and
autonomous driving, which enables multiple vehicles to share sensing
information to enhance perception performance. However, compared with
single-vehicle perception, the openness of a CP system makes it more vulnerable
to malicious attacks that can inject malicious information to mislead the
perception of an ego vehicle, resulting in severe risks for safe driving. To
mitigate such vulnerability, we first propose a new paradigm for malicious
agent detection that effectively identifies malicious agents at the feature
level without requiring verification of final perception results, significantly
reducing computational overhead. Building on this paradigm, we introduce
CP-GuardBench, the first comprehensive dataset provided to train and evaluate
various malicious agent detection methods for CP systems. Furthermore, we
develop a robust defense method called CP-Guard+, which enhances the margin
between the representations of benign and malicious features through a
carefully designed Dual-Centered Contrastive Loss (DCCLoss). Finally, we
conduct extensive experiments on both CP-GuardBench and V2X-Sim, and
demonstrate the superiority of CP-Guard+.


http://arxiv.org/abs/2502.04771
DMPA: Model Poisoning Attacks on Decentralized Federated Learning for Model Differences. (10%)
Chao Feng; Yunlong Li; Yuanzhe Gao; Alberto Huertas Celdrán; der Assen Jan von; Gérôme Bovet; Burkhard Stiller
  Federated learning (FL) has garnered significant attention as a prominent
privacy-preserving Machine Learning (ML) paradigm. Decentralized FL (DFL)
eschews traditional FL's centralized server architecture, enhancing the
system's robustness and scalability. However, these advantages of DFL also
create new vulnerabilities for malicious participants to execute adversarial
attacks, especially model poisoning attacks. In model poisoning attacks,
malicious participants aim to diminish the performance of benign models by
creating and disseminating the compromised model. Existing research on model
poisoning attacks has predominantly concentrated on undermining global models
within the Centralized FL (CFL) paradigm, while there needs to be more research
in DFL. To fill the research gap, this paper proposes an innovative model
poisoning attack called DMPA. This attack calculates the differential
characteristics of multiple malicious client models and obtains the most
effective poisoning strategy, thereby orchestrating a collusive attack by
multiple participants. The effectiveness of this attack is validated across
multiple datasets, with results indicating that the DMPA approach consistently
surpasses existing state-of-the-art FL model poisoning attack strategies.


http://arxiv.org/abs/2502.04662
Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits. (4%)
Sreejeet Maity; Aritra Mitra
  One of the most basic problems in reinforcement learning (RL) is policy
evaluation: estimating the long-term return, i.e., value function,
corresponding to a given fixed policy. The celebrated Temporal Difference (TD)
learning algorithm addresses this problem, and recent work has investigated
finite-time convergence guarantees for this algorithm and variants thereof.
However, these guarantees hinge on the reward observations being always
generated from a well-behaved (e.g., sub-Gaussian) true reward distribution.
Motivated by harsh, real-world environments where such an idealistic assumption
may no longer hold, we revisit the policy evaluation problem from the
perspective of adversarial robustness. In particular, we consider a
Huber-contaminated reward model where an adversary can arbitrarily corrupt each
reward sample with a small probability $\epsilon$. Under this observation
model, we first show that the adversary can cause the vanilla TD algorithm to
converge to any arbitrary value function. We then develop a novel algorithm
called Robust-TD and prove that its finite-time guarantees match that of
vanilla TD with linear function approximation up to a small $O(\epsilon)$ term
that captures the effect of corruption. We complement this result with a
minimax lower bound, revealing that such an additive corruption-induced term is
unavoidable. To our knowledge, these results are the first of their kind in the
context of adversarial robustness of stochastic approximation schemes driven by
Markov noise. The key new technical tool that enables our results is an
analysis of the Median-of-Means estimator with corrupted, time-correlated data
that might be of independent interest to the literature on robust statistics.


http://arxiv.org/abs/2502.04951
The Rising Threat to Emerging AI-Powered Search Engines. (1%)
Zeren Luo; Zifan Peng; Yule Liu; Zhen Sun; Mingchen Li; Jingyi Zheng; Xinlei He
  Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of AI-Powered Search Engines (AIPSEs), offering
precise and efficient responses by integrating external databases with
pre-existing knowledge. However, we observe that these AIPSEs raise risks such
as quoting malicious content or citing malicious websites, leading to harmful
or unverified information dissemination. In this study, we conduct the first
safety risk quantification on seven production AIPSEs by systematically
defining the threat model, risk level, and evaluating responses to various
query types. With data collected from PhishTank, ThreatBook, and LevelBlue, our
findings reveal that AIPSEs frequently generate harmful content that contains
malicious URLs even with benign queries (e.g., with benign keywords). We also
observe that directly query URL will increase the risk level while query with
natural language will mitigate such risk. We further perform two case studies
on online document spoofing and phishing to show the ease of deceiving AIPSEs
in the real-world setting. To mitigate these risks, we develop an agent-based
defense with a GPT-4o-based content refinement tool and an XGBoost-based URL
detector. Our evaluation shows that our defense can effectively reduce the risk
but with the cost of reducing available information. Our research highlights
the urgent need for robust safety measures in AIPSEs.


http://arxiv.org/abs/2502.04204
Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence. (99%)
Shaopeng Fu; Liang Ding; Jingfeng Zhang; Di Wang
  Jailbreak attacks against large language models (LLMs) aim to induce harmful
behaviors in LLMs through carefully crafted adversarial prompts. To mitigate
attacks, one way is to perform adversarial training (AT)-based alignment, i.e.,
training LLMs on some of the most adversarial prompts to help them learn how to
behave safely under attacks. During AT, the length of adversarial prompts plays
a critical role in the robustness of aligned LLMs. While long-length
adversarial prompts during AT might lead to strong LLM robustness, their
synthesis however is very resource-consuming, which may limit the application
of LLM AT. This paper focuses on adversarial suffix jailbreak attacks and
unveils that to defend against a jailbreak attack with an adversarial suffix of
length $\Theta(M)$, it is enough to align LLMs on prompts with adversarial
suffixes of length $\Theta(\sqrt{M})$. Theoretically, we analyze the
adversarial in-context learning of linear transformers on linear regression
tasks and prove a robust generalization bound for trained transformers. The
bound depends on the term $\Theta(\sqrt{M_{\text{test}}}/M_{\text{train}})$,
where $M_{\text{train}}$ and $M_{\text{test}}$ are the numbers of adversarially
perturbed in-context samples during training and testing. Empirically, we
conduct AT on popular open-source LLMs and evaluate their robustness against
jailbreak attacks of different adversarial suffix lengths. Results confirm a
positive correlation between the attack success rate and the ratio of the
square root of the adversarial suffix length during jailbreaking to the length
during AT. Our findings show that it is practical to defend against
``long-length'' jailbreak attacks via efficient ``short-length'' AT. The code
is available at https://github.com/fshp971/adv-icl.


http://arxiv.org/abs/2502.04643
Confidence Elicitation: A New Attack Vector for Large Language Models. (96%)
Brian Formento; Chuan Sheng Foo; See-Kiong Ng
  A fundamental issue in deep learning has been adversarial robustness. As
these systems have scaled, such issues have persisted. Currently, large
language models (LLMs) with billions of parameters suffer from adversarial
attacks just like their earlier, smaller counterparts. However, the threat
models have changed. Previously, having gray-box access, where input embeddings
or output logits/probabilities were visible to the user, might have been
reasonable. However, with the introduction of closed-source models, no
information about the model is available apart from the generated output. This
means that current black-box attacks can only utilize the final prediction to
detect if an attack is successful. In this work, we investigate and demonstrate
the potential of attack guidance, akin to using output probabilities, while
having only black-box access in a classification setting. This is achieved
through the ability to elicit confidence from the model. We empirically show
that the elicited confidence is calibrated and not hallucinated for current
LLMs. By minimizing the elicited confidence, we can therefore increase the
likelihood of misclassification. Our new proposed paradigm demonstrates
promising state-of-the-art results on three datasets across two models
(LLaMA-3-8B-Instruct and Mistral-7B-Instruct-V0.3) when comparing our technique
to existing hard-label black-box attack methods that introduce word-level
substitutions.


http://arxiv.org/abs/2502.04248
Adapting to Evolving Adversaries with Regularized Continual Robust Training. (92%)
Sihui Dai; Christian Cianfarani; Arjun Bhagoji; Vikash Sehwag; Prateek Mittal
  Robust training methods typically defend against specific attack types, such
as Lp attacks with fixed budgets, and rarely account for the fact that
defenders may encounter new attacks over time. A natural solution is to adapt
the defended model to new adversaries as they arise via fine-tuning, a method
which we call continual robust training (CRT). However, when implemented
naively, fine-tuning on new attacks degrades robustness on previous attacks.
This raises the question: how can we improve the initial training and
fine-tuning of the model to simultaneously achieve robustness against previous
and new attacks? We present theoretical results which show that the gap in a
model's robustness against different attacks is bounded by how far each attack
perturbs a sample in the model's logit space, suggesting that regularizing with
respect to this logit space distance can help maintain robustness against
previous attacks. Extensive experiments on 3 datasets (CIFAR-10, CIFAR-100, and
ImageNette) and over 100 attack combinations demonstrate that the proposed
regularization improves robust accuracy with little overhead in training time.
Our findings and open-source code lay the groundwork for the deployment of
models robust to evolving attacks.


http://arxiv.org/abs/2502.05225
BitAbuse: A Dataset of Visually Perturbed Texts for Defending Phishing Attacks. (88%)
Hanyong Lee; Chaelyn Lee; Yongjae Lee; Jaesung Lee
  Phishing often targets victims through visually perturbed texts to bypass
security systems. The noise contained in these texts functions as an
adversarial attack, designed to deceive language models and hinder their
ability to accurately interpret the content. However, since it is difficult to
obtain sufficient phishing cases, previous studies have used synthetic datasets
that do not contain real-world cases. In this study, we propose the BitAbuse
dataset, which includes real-world phishing cases, to address the limitations
of previous research. Our dataset comprises a total of 325,580 visually
perturbed texts. The dataset inputs are drawn from the raw corpus, consisting
of visually perturbed sentences and sentences generated through an artificial
perturbation process. Each input sentence is labeled with its corresponding
ground truth, representing the restored, non-perturbed version. Language models
trained on our proposed dataset demonstrated significantly better performance
compared to previous methods, achieving an accuracy of approximately 96%. Our
analysis revealed a significant gap between real-world and synthetic examples,
underscoring the value of our dataset for building reliable pre-trained models
for restoration tasks. We release the BitAbuse dataset, which includes
real-world phishing cases annotated with visual perturbations, to support
future research in adversarial attack defense.


http://arxiv.org/abs/2502.04121
Optimizing Perturbations for Improved Training of Machine Learning Models. (69%)
Sagi Meir; Tommer D. Keidar; Shlomi Reuveni; Barak Hirshberg
  Machine learning models have become indispensable tools in applications
across the physical sciences. Their training is often time-consuming, vastly
exceeding the inference timescales. Several protocols have been developed to
perturb the learning process and improve the training, such as shrink and
perturb, warm restarts, and stochastic resetting. For classifiers, these
perturbations have been shown to result in enhanced speedups or improved
generalization. However, the design of such perturbations is usually done
\textit{ad hoc} by intuition and trial and error. To rationally optimize
training protocols, we frame them as first-passage processes and consider their
response to perturbations. We show that if the unperturbed learning process
reaches a quasi-steady state, the response at a single perturbation frequency
can predict the behavior at a wide range of frequencies. We demonstrate that
this is the case when training a CIFAR-10 classifier using the ResNet-18 model
and use this approach to identify an optimal perturbation and frequency. Our
work allows optimization of training protocols of machine learning models using
a statistical mechanical approach.


http://arxiv.org/abs/2502.03801
SoK: Benchmarking Poisoning Attacks and Defenses in Federated Learning. (15%)
Heyi Zhang; Yule Liu; Xinlei He; Jun Wu; Tianshuo Cong; Xinyi Huang
  Federated learning (FL) enables collaborative model training while preserving
data privacy, but its decentralized nature exposes it to client-side data
poisoning attacks (DPAs) and model poisoning attacks (MPAs) that degrade global
model performance. While numerous proposed defenses claim substantial
effectiveness, their evaluation is typically done in isolation with limited
attack strategies, raising concerns about their validity. Additionally,
existing studies overlook the mutual effectiveness of defenses against both
DPAs and MPAs, causing fragmentation in this field. This paper aims to provide
a unified benchmark and analysis of defenses against DPAs and MPAs, clarifying
the distinction between these two similar but slightly distinct domains. We
present a systematic taxonomy of poisoning attacks and defense strategies,
outlining their design, strengths, and limitations. Then, a unified comparative
evaluation across FL algorithms and data heterogeneity is conducted to validate
their individual and mutual effectiveness and derive key insights for design
principles and future research. Along with the analysis, we frame our work to a
unified benchmark, FLPoison, with high modularity and scalability to evaluate
15 representative poisoning attacks and 17 defense strategies, facilitating
future research in this domain. Code is available at
https://github.com/vio1etus/FLPoison.


http://arxiv.org/abs/2502.04224
Provably Robust Explainable Graph Neural Networks against Graph Perturbation Attacks. (12%)
Jiate Li; Meng Pang; Yun Dong; Jinyuan Jia; Binghui Wang
  Explaining Graph Neural Network (XGNN) has gained growing attention to
facilitate the trust of using GNNs, which is the mainstream method to learn
graph data. Despite their growing attention, Existing XGNNs focus on improving
the explanation performance, and its robustness under attacks is largely
unexplored. We noticed that an adversary can slightly perturb the graph
structure such that the explanation result of XGNNs is largely changed. Such
vulnerability of XGNNs could cause serious issues particularly in
safety/security-critical applications. In this paper, we take the first step to
study the robustness of XGNN against graph perturbation attacks, and propose
XGNNCert, the first provably robust XGNN. Particularly, our XGNNCert can
provably ensure the explanation result for a graph under the worst-case graph
perturbation attack is close to that without the attack, while not affecting
the GNN prediction, when the number of perturbed edges is bounded. Evaluation
results on multiple graph datasets and GNN explainers show the effectiveness of
XGNNCert.


http://arxiv.org/abs/2502.04229
Dark Distillation: Backdooring Distilled Datasets without Accessing Raw Data. (11%)
Ziyuan Yang; Ming Yan; Yi Zhang; Joey Tianyi Zhou
  Dataset distillation (DD) enhances training efficiency and reduces bandwidth
by condensing large datasets into smaller synthetic ones. It enables models to
achieve performance comparable to those trained on the raw full dataset and has
become a widely adopted method for data sharing. However, security concerns in
DD remain underexplored. Existing studies typically assume that malicious
behavior originates from dataset owners during the initial distillation
process, where backdoors are injected into raw datasets. In contrast, this work
is the first to address a more realistic and concerning threat: attackers may
intercept the dataset distribution process, inject backdoors into the distilled
datasets, and redistribute them to users. While distilled datasets were
previously considered resistant to backdoor attacks, we demonstrate that they
remain vulnerable to such attacks. Furthermore, we show that attackers do not
even require access to any raw data to inject the backdoors successfully.
Specifically, our approach reconstructs conceptual archetypes for each class
from the model trained on the distilled dataset. Backdoors are then injected
into these archetypes to update the distilled dataset. Moreover, we ensure the
updated dataset not only retains the backdoor but also preserves the original
optimization trajectory, thus maintaining the knowledge of the raw dataset. To
achieve this, a hybrid loss is designed to integrate backdoor information along
the benign optimization trajectory, ensuring that previously learned
information is not forgotten. Extensive experiments demonstrate that distilled
datasets are highly vulnerable to backdoor attacks, with risks pervasive across
various raw datasets, distillation methods, and downstream training strategies.
Moreover, our attack method is efficient, capable of synthesizing a malicious
distilled dataset in under one minute in certain cases.


http://arxiv.org/abs/2502.04230
XAttnMark: Learning Robust Audio Watermarking with Cross-Attention. (1%)
Yixin Liu; Lie Lu; Jihui Jin; Lichao Sun; Andrea Fanelli
  The rapid proliferation of generative audio synthesis and editing
technologies has raised significant concerns about copyright infringement, data
provenance, and the spread of misinformation through deepfake audio.
Watermarking offers a proactive solution by embedding imperceptible,
identifiable, and traceable marks into audio content. While recent neural
network-based watermarking methods like WavMark and AudioSeal have improved
robustness and quality, they struggle to achieve both robust detection and
accurate attribution simultaneously. This paper introduces Cross-Attention
Robust Audio Watermark (XAttnMark), which bridges this gap by leveraging
partial parameter sharing between the generator and the detector, a
cross-attention mechanism for efficient message retrieval, and a temporal
conditioning module for improved message distribution. Additionally, we propose
a psychoacoustic-aligned temporal-frequency masking loss that captures
fine-grained auditory masking effects, enhancing watermark imperceptibility.
Our approach achieves state-of-the-art performance in both detection and
attribution, demonstrating superior robustness against a wide range of audio
transformations, including challenging generative editing with strong editing
strength. The project webpage is available at
https://liuyixin-louis.github.io/xattnmark/.


http://arxiv.org/abs/2502.03950
LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Models. (1%)
Priyank Pathak; Shyam Marjit; Shruti Vyas; Yogesh S Rawat
  Visual-language foundation Models (FMs) exhibit remarkable zero-shot
generalization across diverse tasks, largely attributed to extensive
pre-training on largescale datasets. However, their robustness on
low-resolution/pixelated (LR) images, a common challenge in real-world
scenarios, remains underexplored. We introduce LR0.FM, a comprehensive
benchmark evaluating the impact of low resolution on the zero-shot
classification performance of 10 FM(s) across 66 backbones and 15 datasets. We
propose a novel metric, Weighted Aggregated Robustness, to address the
limitations of existing metrics and better evaluate model performance across
resolutions and datasets. Our key findings show that: (i) model size positively
correlates with robustness to resolution degradation, (ii) pre-training dataset
quality is more important than its size, and (iii) fine-tuned and higher
resolution models are less robust against LR. Our analysis further reveals that
the model makes semantically reasonable predictions at LR, and the lack of
fine-grained details in input adversely impacts the model's initial layers more
than the deeper layers. We use these insights and introduce a simple strategy,
LR-TK0, to enhance the robustness of models without compromising their
pre-trained weights. We demonstrate the effectiveness of LR-TK0 for robustness
against low-resolution across several datasets and its generalization
capability across backbones and other approaches. Code is available at
https://github.com/shyammarjit/LR0.FM


http://arxiv.org/abs/2502.03698
How vulnerable is my policy? Adversarial attacks on modern behavior cloning policies. (99%)
Basavasagar Patil; Akansha Kalra; Guanhong Tao; Daniel S. Brown
  Learning from Demonstration (LfD) algorithms have shown promising results in
robotic manipulation tasks, but their vulnerability to adversarial attacks
remains underexplored. This paper presents a comprehensive study of adversarial
attacks on both classic and recently proposed algorithms, including Behavior
Cloning (BC), LSTM-GMM, Implicit Behavior Cloning (IBC), Diffusion Policy (DP),
and VQ-Behavior Transformer (VQ-BET). We study the vulnerability of these
methods to untargeted, targeted and universal adversarial perturbations. While
explicit policies, such as BC, LSTM-GMM and VQ-BET can be attacked in the same
manner as standard computer vision models, we find that attacks for implicit
and denoising policy models are nuanced and require developing novel attack
methods. Our experiments on several simulated robotic manipulation tasks reveal
that most of the current methods are highly vulnerable to adversarial
perturbations. We also show that these attacks are transferable across
algorithms, architectures, and tasks, raising concerning security
vulnerabilities with potentially a white-box threat model. In addition, we test
the efficacy of a randomized smoothing, a widely used adversarial defense
technique, and highlight its limitation in defending against attacks on complex
and multi-modal action distribution common in complex control tasks. In
summary, our findings highlight the vulnerabilities of modern BC algorithms,
paving way for future work in addressing such limitations.


http://arxiv.org/abs/2502.06832
Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach. (87%)
Xu Zhang; Kaidi Xu; Ziqing Hu; Ren Wang
  Mixture of Experts (MoE) have shown remarkable success in leveraging
specialized expert networks for complex machine learning tasks. However, their
susceptibility to adversarial attacks presents a critical challenge for
deployment in robust applications. This paper addresses the critical question
of how to incorporate robustness into MoEs while maintaining high natural
accuracy. We begin by analyzing the vulnerability of MoE components, finding
that expert networks are notably more susceptible to adversarial attacks than
the router. Based on this insight, we propose a targeted robust training
technique that integrates a novel loss function to enhance the adversarial
robustness of MoE, requiring only the robustification of one additional expert
without compromising training or inference efficiency. Building on this, we
introduce a dual-model strategy that linearly combines a standard MoE model
with our robustified MoE model using a smoothing parameter. This approach
allows for flexible control over the robustness-accuracy trade-off. We further
provide theoretical foundations by deriving certified robustness bounds for
both the single MoE and the dual-model. To push the boundaries of robustness
and accuracy, we propose a novel joint training strategy JTDMoE for the
dual-model. This joint training enhances both robustness and accuracy beyond
what is achievable with separate models. Experimental results on CIFAR-10 and
TinyImageNet datasets using ResNet18 and Vision Transformer (ViT) architectures
demonstrate the effectiveness of our proposed methods.


http://arxiv.org/abs/2502.03758
Improving Adversarial Robustness via Phase and Amplitude-aware Prompting. (80%)
Yibo Xu; Dawei Zhou; Decheng Liu; Nannan Wang
  Deep neural networks are found to be vulnerable to adversarial perturbations.
The prompt-based defense has been increasingly studied due to its high
efficiency. However, existing prompt-based defenses mainly exploited mixed
prompt patterns, where critical patterns closely related to object semantics
lack sufficient focus. The phase and amplitude spectra have been proven to be
highly related to specific semantic patterns and crucial for robustness. To
this end, in this paper, we propose a Phase and Amplitude-aware Prompting (PAP)
defense. Specifically, we construct phase-level and amplitude-level prompts for
each class, and adjust weights for prompting according to the model's robust
performance under these prompts during training. During testing, we select
prompts for each image using its predicted label to obtain the prompted image,
which is inputted to the model to get the final prediction. Experimental
results demonstrate the effectiveness of our method.


http://arxiv.org/abs/2502.02913
Real-Time Privacy Risk Measurement with Privacy Tokens for Gradient Leakage. (70%)
Jiayang Meng; Tao Huang; Hong Chen; Xin Shi; Qingyu Huang; Chen Hou
  The widespread deployment of deep learning models in privacy-sensitive
domains has amplified concerns regarding privacy risks, particularly those
stemming from gradient leakage during training. Current privacy assessments
primarily rely on post-training attack simulations. However, these methods are
inherently reactive, unable to encompass all potential attack scenarios, and
often based on idealized adversarial assumptions. These limitations underscore
the need for proactive approaches to privacy risk assessment during the
training process. To address this gap, we propose the concept of privacy
tokens, which are derived directly from private gradients during training.
Privacy tokens encapsulate gradient features and, when combined with data
features, offer valuable insights into the extent of private information
leakage from training data, enabling real-time measurement of privacy risks
without relying on adversarial attack simulations. Additionally, we employ
Mutual Information (MI) as a robust metric to quantify the relationship between
training data and gradients, providing precise and continuous assessments of
privacy leakage throughout the training process. Extensive experiments validate
our framework, demonstrating the effectiveness of privacy tokens and MI in
identifying and quantifying privacy risks. This proactive approach marks a
significant advancement in privacy monitoring, promoting the safer deployment
of deep learning models in sensitive applications.


http://arxiv.org/abs/2502.05224
A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations. (67%)
Yihe Zhou; Tao Ni; Wei-Bin Lee; Qingchuan Zhao
  Large Language Models (LLMs) have achieved significantly advanced
capabilities in understanding and generating human language text, which have
gained increasing popularity over recent years. Apart from their
state-of-the-art natural language processing (NLP) performance, considering
their widespread usage in many industries, including medicine, finance,
education, etc., security concerns over their usage grow simultaneously. In
recent years, the evolution of backdoor attacks has progressed with the
advancement of defense mechanisms against them and more well-developed features
in the LLMs. In this paper, we adapt the general taxonomy for classifying
machine learning attacks on one of the subdivisions - training-time white-box
backdoor attacks. Besides systematically classifying attack methods, we also
consider the corresponding defense methods against backdoor attacks. By
providing an extensive summary of existing works, we hope this survey can serve
as a guideline for inspiring future research that further extends the attack
scenarios and creates a stronger defense against them for more robust LLMs.


http://arxiv.org/abs/2502.02960
Large Language Model Adversarial Landscape Through the Lens of Attack Objectives. (56%)
Nan Wang; Kane Walter; Yansong Gao; Alsharif Abuadbba
  Large Language Models (LLMs) represent a transformative leap in artificial
intelligence, enabling the comprehension, generation, and nuanced interaction
with human language on an unparalleled scale. However, LLMs are increasingly
vulnerable to a range of adversarial attacks that threaten their privacy,
reliability, security, and trustworthiness. These attacks can distort outputs,
inject biases, leak sensitive information, or disrupt the normal functioning of
LLMs, posing significant challenges across various applications.
  In this paper, we provide a novel comprehensive analysis of the adversarial
landscape of LLMs, framed through the lens of attack objectives. By
concentrating on the core goals of adversarial actors, we offer a fresh
perspective that examines threats from the angles of privacy, integrity,
availability, and misuse, moving beyond conventional taxonomies that focus
solely on attack techniques. This objective-driven adversarial landscape not
only highlights the strategic intent behind different adversarial approaches
but also sheds light on the evolving nature of these threats and the
effectiveness of current defenses. Our analysis aims to guide researchers and
practitioners in better understanding, anticipating, and mitigating these
attacks, ultimately contributing to the development of more resilient and
robust LLM systems.


http://arxiv.org/abs/2502.03721
Detecting Backdoor Attacks via Similarity in Semantic Communication Systems. (54%)
Ziyang Wei; Yili Jiang; Jiaqi Huang; Fangtian Zhong; Sohan Gyawali
  Semantic communication systems, which leverage Generative AI (GAI) to
transmit semantic meaning rather than raw data, are poised to revolutionize
modern communications. However, they are vulnerable to backdoor attacks, a type
of poisoning manipulation that embeds malicious triggers into training
datasets. As a result, Backdoor attacks mislead the inference for poisoned
samples while clean samples remain unaffected. The existing defenses may alter
the model structure (such as neuron pruning that potentially degrades inference
performance on clean inputs, or impose strict requirements on data formats
(such as ``Semantic Shield" that requires image-text pairs). To address these
limitations, this work proposes a defense mechanism that leverages semantic
similarity to detect backdoor attacks without modifying the model structure or
imposing data format constraints. By analyzing deviations in semantic feature
space and establishing a threshold-based detection framework, the proposed
approach effectively identifies poisoned samples. The experimental results
demonstrate high detection accuracy and recall across varying poisoning ratios,
underlining the significant effectiveness of our proposed solution.


http://arxiv.org/abs/2502.03692
DocMIA: Document-Level Membership Inference Attacks against DocVQA Models. (41%)
Khanh Nguyen; Raouf Kerkouche; Mario Fritz; Dimosthenis Karatzas
  Document Visual Question Answering (DocVQA) has introduced a new paradigm for
end-to-end document understanding, and quickly became one of the standard
benchmarks for multimodal LLMs. Automating document processing workflows,
driven by DocVQA models, presents significant potential for many business
sectors. However, documents tend to contain highly sensitive information,
raising concerns about privacy risks associated with training such DocVQA
models. One significant privacy vulnerability, exploited by the membership
inference attack, is the possibility for an adversary to determine if a
particular record was part of the model's training data. In this paper, we
introduce two novel membership inference attacks tailored specifically to
DocVQA models. These attacks are designed for two different adversarial
scenarios: a white-box setting, where the attacker has full access to the model
architecture and parameters, and a black-box setting, where only the model's
outputs are available. Notably, our attacks assume the adversary lacks access
to auxiliary datasets, which is more realistic in practice but also more
challenging. Our unsupervised methods outperform existing state-of-the-art
membership inference attacks across a variety of DocVQA models and datasets,
demonstrating their effectiveness and highlighting the privacy risks in this
domain.


http://arxiv.org/abs/2502.03052
Understanding and Enhancing the Transferability of Jailbreaking Attacks. (31%)
Runqi Lin; Bo Han; Fengwang Li; Tongling Liu
  Jailbreaking attacks can effectively manipulate open-source large language
models (LLMs) to produce harmful responses. However, these attacks exhibit
limited transferability, failing to disrupt proprietary LLMs consistently. To
reliably identify vulnerabilities in proprietary LLMs, this work investigates
the transferability of jailbreaking attacks by analysing their impact on the
model's intent perception. By incorporating adversarial sequences, these
attacks can redirect the source LLM's focus away from malicious-intent tokens
in the original input, thereby obstructing the model's intent recognition and
eliciting harmful responses. Nevertheless, these adversarial sequences fail to
mislead the target LLM's intent perception, allowing the target LLM to refocus
on malicious-intent tokens and abstain from responding. Our analysis further
reveals the inherent distributional dependency within the generated adversarial
sequences, whose effectiveness stems from overfitting the source LLM's
parameters, resulting in limited transferability to target LLMs. To this end,
we propose the Perceived-importance Flatten (PiF) method, which uniformly
disperses the model's focus across neutral-intent tokens in the original input,
thus obscuring malicious-intent tokens without relying on overfitted
adversarial sequences. Extensive experiments demonstrate that PiF provides an
effective and efficient red-teaming evaluation for proprietary LLMs.


http://arxiv.org/abs/2502.05214
CoRPA: Adversarial Image Generation for Chest X-rays Using Concept Vector Perturbations and Generative Models. (99%)
Amy Rafferty; Rishi Ramaesh; Ajitha Rajan
  Deep learning models for medical image classification tasks are becoming
widely implemented in AI-assisted diagnostic tools, aiming to enhance
diagnostic accuracy, reduce clinician workloads, and improve patient outcomes.
However, their vulnerability to adversarial attacks poses significant risks to
patient safety. Current attack methodologies use general techniques such as
model querying or pixel value perturbations to generate adversarial examples
designed to fool a model. These approaches may not adequately address the
unique characteristics of clinical errors stemming from missed or incorrectly
identified clinical features. We propose the Concept-based Report Perturbation
Attack (CoRPA), a clinically-focused black-box adversarial attack framework
tailored to the medical imaging domain. CoRPA leverages clinical concepts to
generate adversarial radiological reports and images that closely mirror
realistic clinical misdiagnosis scenarios. We demonstrate the utility of CoRPA
using the MIMIC-CXR-JPG dataset of chest X-rays and radiological reports. Our
evaluation reveals that deep learning models exhibiting strong resilience to
conventional adversarial attacks are significantly less robust when subjected
to CoRPA's clinically-focused perturbations. This underscores the importance of
addressing domain-specific vulnerabilities in medical AI systems. By
introducing a specialized adversarial attack framework, this study provides a
foundation for developing robust, real-world-ready AI models in healthcare,
ensuring their safe and reliable deployment in high-stakes clinical
environments.


http://arxiv.org/abs/2502.02096
Dual-Flow: Transferable Multi-Target, Instance-Agnostic Attacks via In-the-wild Cascading Flow Optimization. (97%)
Yixiao Chen; Shikun Sun; Jianshu Li; Ruoyu Li; Zhe Li; Junliang Xing
  Adversarial attacks are widely used to evaluate model robustness, and in
black-box scenarios, the transferability of these attacks becomes crucial.
Existing generator-based attacks have excellent generalization and
transferability due to their instance-agnostic nature. However, when training
generators for multi-target tasks, the success rate of transfer attacks is
relatively low due to the limitations of the model's capacity. To address these
challenges, we propose a novel Dual-Flow framework for multi-target
instance-agnostic adversarial attacks, utilizing Cascading Distribution Shift
Training to develop an adversarial velocity function. Extensive experiments
demonstrate that Dual-Flow significantly improves transferability over previous
multi-target generative attacks. For example, it increases the success rate
from Inception-v3 to ResNet-152 by 34.58%. Furthermore, our attack method shows
substantially stronger robustness against defense mechanisms, such as
adversarially trained models.


http://arxiv.org/abs/2502.04360
MARAGE: Transferable Multi-Model Adversarial Attack for Retrieval-Augmented Generation Data Extraction. (91%)
Xiao Hu; Eric Liu; Weizhou Wang; Xiangyu Guo; David Lie
  Retrieval-Augmented Generation (RAG) offers a solution to mitigate
hallucinations in Large Language Models (LLMs) by grounding their outputs to
knowledge retrieved from external sources. The use of private resources and
data in constructing these external data stores can expose them to risks of
extraction attacks, in which attackers attempt to steal data from these private
databases. Existing RAG extraction attacks often rely on manually crafted
prompts, which limit their effectiveness. In this paper, we introduce a
framework called MARAGE for optimizing an adversarial string that, when
appended to user queries submitted to a target RAG system, causes outputs
containing the retrieved RAG data verbatim. MARAGE leverages a continuous
optimization scheme that integrates gradients from multiple models with
different architectures simultaneously to enhance the transferability of the
optimized string to unseen models. Additionally, we propose a strategy that
emphasizes the initial tokens in the target RAG data, further improving the
attack's generalizability. Evaluations show that MARAGE consistently
outperforms both manual and optimization-based baselines across multiple LLMs
and RAG datasets, while maintaining robust transferability to previously unseen
models. Moreover, we conduct probing tasks to shed light on the reasons why
MARAGE is more effective compared to the baselines and to analyze the impact of
our approach on the model's internal state.


http://arxiv.org/abs/2502.02290
FRAUD-RLA: A new reinforcement learning adversarial attack against credit card fraud detection. (87%)
Daniele Lunghi; Yannick Molinghen; Alkis Simitsis; Tom Lenaerts; Gianluca Bontempi
  Adversarial attacks pose a significant threat to data-driven systems, and
researchers have spent considerable resources studying them. Despite its
economic relevance, this trend largely overlooked the issue of credit card
fraud detection. To address this gap, we propose a new threat model that
demonstrates the limitations of existing attacks and highlights the necessity
to investigate new approaches. We then design a new adversarial attack for
credit card fraud detection, employing reinforcement learning to bypass
classifiers. This attack, called FRAUD-RLA, is designed to maximize the
attacker's reward by optimizing the exploration-exploitation tradeoff and
working with significantly less required knowledge than competitors. Our
experiments, conducted on three different heterogeneous datasets and against
two fraud detection systems, indicate that FRAUD-RLA is effective, even
considering the severe limitations imposed by our threat model.


http://arxiv.org/abs/2502.02537
Uncertainty Quantification for Collaborative Object Detection Under Adversarial Attacks. (83%)
Huiqun Huang; Cong Chen; Jean-Philippe Monteuuis; Jonathan Petit; Fei Miao
  Collaborative Object Detection (COD) and collaborative perception can
integrate data or features from various entities, and improve object detection
accuracy compared with individual perception. However, adversarial attacks pose
a potential threat to the deep learning COD models, and introduce high output
uncertainty. With unknown attack models, it becomes even more challenging to
improve COD resiliency and quantify the output uncertainty for highly dynamic
perception scenes such as autonomous vehicles. In this study, we propose the
Trusted Uncertainty Quantification in Collaborative Perception framework
(TUQCP). TUQCP leverages both adversarial training and uncertainty
quantification techniques to enhance the adversarial robustness of existing COD
models. More specifically, TUQCP first adds perturbations to the shared
information of randomly selected agents during object detection collaboration
by adversarial training. TUQCP then alleviates the impacts of adversarial
attacks by providing output uncertainty estimation through learning-based
module and uncertainty calibration through conformal prediction. Our framework
works for early and intermediate collaboration COD models and single-agent
object detection models. We evaluate TUQCP on V2X-Sim, a comprehensive
collaborative perception dataset for autonomous driving, and demonstrate a
80.41% improvement in object detection accuracy compared to the baselines under
the same adversarial attacks. TUQCP demonstrates the importance of uncertainty
quantification to COD under adversarial attacks.


http://arxiv.org/abs/2502.02260
Adversarial ML Problems Are Getting Harder to Solve and to Evaluate. (81%)
Javier Rando; Jie Zhang; Nicholas Carlini; Florian Tramèr
  In the past decade, considerable research effort has been devoted to securing
machine learning (ML) models that operate in adversarial settings. Yet,
progress has been slow even for simple "toy" problems (e.g., robustness to
small adversarial perturbations) and is often hindered by non-rigorous
evaluations. Today, adversarial ML research has shifted towards studying
larger, general-purpose language models. In this position paper, we argue that
the situation is now even worse: in the era of LLMs, the field of adversarial
ML studies problems that are (1) less clearly defined, (2) harder to solve, and
(3) even more challenging to evaluate. As a result, we caution that yet another
decade of work on adversarial ML may fail to produce meaningful progress.


http://arxiv.org/abs/2502.02844
Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning. (75%)
Sunwoo Lee; Jaebak Hwang; Yonghyeon Jo; Seungyul Han
  Traditional robust methods in multi-agent reinforcement learning (MARL) often
struggle against coordinated adversarial attacks in cooperative scenarios. To
address this limitation, we propose the Wolfpack Adversarial Attack framework,
inspired by wolf hunting strategies, which targets an initial agent and its
assisting agents to disrupt cooperation. Additionally, we introduce the
Wolfpack-Adversarial Learning for MARL (WALL) framework, which trains robust
MARL policies to defend against the proposed Wolfpack attack by fostering
systemwide collaboration. Experimental results underscore the devastating
impact of the Wolfpack attack and the significant robustness improvements
achieved by WALL. Our code is available at
https://github.com/sunwoolee0504/WALL.


http://arxiv.org/abs/2502.02438
Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment. (68%)
Yaling Shen; Zhixiong Zhuang; Kun Yuan; Maria-Irina Nicolae; Nassir Navab; Nicolas Padoy; Mario Fritz
  Medical multimodal large language models (MLLMs) are becoming an instrumental
part of healthcare systems, assisting medical personnel with decision making
and results analysis. Models for radiology report generation are able to
interpret medical imagery, thus reducing the workload of radiologists. As
medical data is scarce and protected by privacy regulations, medical MLLMs
represent valuable intellectual property. However, these assets are potentially
vulnerable to model stealing, where attackers aim to replicate their
functionality via black-box access. So far, model stealing for the medical
domain has focused on classification; however, existing attacks are not
effective against MLLMs. In this paper, we introduce Adversarial Domain
Alignment (ADA-STEAL), the first stealing attack against medical MLLMs.
ADA-STEAL relies on natural images, which are public and widely available, as
opposed to their medical counterparts. We show that data augmentation with
adversarial noise is sufficient to overcome the data distribution gap between
natural images and the domain-specific distribution of the victim MLLM.
Experiments on the IU X-RAY and MIMIC-CXR radiology datasets demonstrate that
Adversarial Domain Alignment enables attackers to steal the medical MLLM
without any access to medical data.


http://arxiv.org/abs/2502.02230
An Attack-Driven Incident Response and Defense System (ADIRDS). (4%)
Anthony Cheuk Tung Lai; Siu Ming Yiu; Ping Fan Ke; Alan Ho
  One of the major goals of incident response is to help an organization or a
system owner to quickly identify and halt the attacks to minimize the damages
(and financial loss) to the system being attacked. Typical incident responses
rely very much on the log information captured by the system during the attacks
and if needed, may need to isolate the victim from the network to avoid further
destructive attacks. However, there are real cases that there are insufficient
log records/information for the incident response team to identify the attacks
and their origins while the attacked system cannot be stopped due to service
requirements (zero downtime online systems) such as online gaming sites.
Typical incident response procedures and industrial standards do not provide an
adequate solution to address this scenario. In this paper, being motivated by a
real case, we propose a solution, called "Attack-Driven Incident Response and
Defense System (ADIRDS)" to tackle this problem. ADIRDS is an online monitoring
system to run with the real system. By modeling the real system as a graph,
critical nodes/assets of the system are closely monitored. Instead of relying
on the original logging system, evidence will be collected from the attack
technique perspectives. To migrate the risks, realistic honeypots with very
similar business context as the real system are deployed to trap the attackers.
We successfully apply this system to a real case. Based on our experiments, we
verify that our new approach of designing the realistic honeypots is effective,
38 unique attacker's IP addresses were captured. We also compare the
performance of our realistic honey with both low and high interactive honeypots
proposed in the literature, the results found that our proposed honeypot can
successfully cheat the attackers to attack our honeypot, which verifies that
our honeypot is more effective.


http://arxiv.org/abs/2502.02730
Semantic Entanglement-Based Ransomware Detection via Probabilistic Latent Encryption Mapping. (4%)
Mohammad Eisa; Quentin Yardley; Rafael Witherspoon; Harriet Pendlebury; Clement Rutherford
  Encryption-based attacks have introduced significant challenges for detection
mechanisms that rely on predefined signatures, heuristic indicators, or static
rule-based classifications. Probabilistic Latent Encryption Mapping presents an
alternative detection framework that models ransomware-induced encryption
behaviors through statistical representations of entropy deviations and
probabilistic dependencies in execution traces. Unlike conventional approaches
that depend on explicit bytecode analysis or predefined cryptographic function
call monitoring, probabilistic inference techniques classify encryption
anomalies based on their underlying statistical characteristics, ensuring
greater adaptability to polymorphic attack strategies. Evaluations demonstrate
that entropy-driven classification reduces false positive rates while
maintaining high detection accuracy across diverse ransomware families and
encryption methodologies. Experimental results further highlight the
framework's ability to differentiate between benign encryption workflows and
adversarial cryptographic manipulations, ensuring that classification
performance remains effective across cloud-based and localized execution
environments. Benchmark comparisons illustrate that probabilistic modeling
exhibits advantages over heuristic and machine learning-based detection
approaches, particularly in handling previously unseen encryption techniques
and adversarial obfuscation strategies. Computational efficiency analysis
confirms that detection latency remains within operational feasibility
constraints, reinforcing the viability of probabilistic encryption
classification for real-time security infrastructures. The ability to
systematically infer encryption-induced deviations without requiring static
attack signatures strengthens detection robustness against adversarial evasion
techniques.


http://arxiv.org/abs/2502.02710
Achievable distributional robustness when the robust risk is only partially identified. (1%)
Julia Kostin; Nicola Gnecco; Fanny Yang
  In safety-critical applications, machine learning models should generalize
well under worst-case distribution shifts, that is, have a small robust risk.
Invariance-based algorithms can provably take advantage of structural
assumptions on the shifts when the training distributions are heterogeneous
enough to identify the robust risk. However, in practice, such identifiability
conditions are rarely satisfied -- a scenario so far underexplored in the
theoretical literature. In this paper, we aim to fill the gap and propose to
study the more general setting when the robust risk is only partially
identifiable. In particular, we introduce the worst-case robust risk as a new
measure of robustness that is always well-defined regardless of
identifiability. Its minimum corresponds to an algorithm-independent
(population) minimax quantity that measures the best achievable robustness
under partial identifiability. While these concepts can be defined more
broadly, in this paper we introduce and derive them explicitly for a linear
model for concreteness of the presentation. First, we show that existing
robustness methods are provably suboptimal in the partially identifiable case.
We then evaluate these methods and the minimizer of the (empirical) worst-case
robust risk on real-world gene expression data and find a similar trend: the
test error of existing robustness methods grows increasingly suboptimal as the
fraction of data from unseen environments increases, whereas accounting for
partial identifiability allows for better generalization.


http://arxiv.org/abs/2502.02017
Multi-Domain Graph Foundation Models: Robust Knowledge Transfer via Topology Alignment. (1%)
Shuo Wang; Bokui Wang; Zhixiang Shen; Boyan Deng; Zhao Kang
  Recent advances in CV and NLP have inspired researchers to develop
general-purpose graph foundation models through pre-training across diverse
domains. However, a fundamental challenge arises from the substantial
differences in graph topologies across domains. Additionally, real-world graphs
are often sparse and prone to noisy connections and adversarial attacks. To
address these issues, we propose the Multi-Domain Graph Foundation Model
(MDGFM), a unified framework that aligns and leverages cross-domain topological
information to facilitate robust knowledge transfer. MDGFM bridges different
domains by adaptively balancing features and topology while refining original
graphs to eliminate noise and align topological structures. To further enhance
knowledge transfer, we introduce an efficient prompt-tuning approach. By
aligning topologies, MDGFM not only improves multi-domain pre-training but also
enables robust knowledge transfer to unseen domains. Theoretical analyses
provide guarantees of MDGFM's effectiveness and domain generalization
capabilities. Extensive experiments on both homophilic and heterophilic graph
datasets validate the robustness and efficacy of our method.


http://arxiv.org/abs/2502.01262
FSPGD: Rethinking Black-box Attacks on Semantic Segmentation. (99%)
Eun-Sol Park; MiSo Park; Seung Park; Yong-Goo Shin
  Transferability, the ability of adversarial examples crafted for one model to
deceive other models, is crucial for black-box attacks. Despite advancements in
attack methods for semantic segmentation, transferability remains limited,
reducing their effectiveness in real-world applications. To address this, we
introduce the Feature Similarity Projected Gradient Descent (FSPGD) attack, a
novel black-box approach that enhances both attack performance and
transferability. Unlike conventional segmentation attacks that rely on output
predictions for gradient calculation, FSPGD computes gradients from
intermediate layer features. Specifically, our method introduces a loss
function that targets local information by comparing features between clean
images and adversarial examples, while also disrupting contextual information
by accounting for spatial relationships between objects. Experiments on Pascal
VOC 2012 and Cityscapes datasets demonstrate that FSPGD achieves superior
transferability and attack performance, establishing a new state-of-the-art
benchmark. Code is available at https://github.com/KU-AIVS/FSPGD.


http://arxiv.org/abs/2502.01576
Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models. (95%)
Hashmat Shadab Malik; Fahad Shamshad; Muzammal Naseer; Karthik Nandakumar; Fahad Khan; Salman Khan
  Multi-modal Large Language Models (MLLMs) excel in vision-language tasks but
remain vulnerable to visual adversarial perturbations that can induce
hallucinations, manipulate responses, or bypass safety mechanisms. Existing
methods seek to mitigate these risks by applying constrained adversarial
fine-tuning to CLIP vision encoders on ImageNet-scale data, ensuring their
generalization ability is preserved. However, this limited adversarial training
restricts robustness and broader generalization. In this work, we explore an
alternative approach of leveraging existing vision classification models that
have been adversarially pre-trained on large-scale data. Our analysis reveals
two principal contributions: (1) the extensive scale and diversity of
adversarial pre-training enables these models to demonstrate superior
robustness against diverse adversarial threats, ranging from imperceptible
perturbations to advanced jailbreaking attempts, without requiring additional
adversarial training, and (2) end-to-end MLLM integration with these robust
models facilitates enhanced adaptation of language components to robust visual
features, outperforming existing plug-and-play methodologies on complex
reasoning tasks. Through systematic evaluation across visual
question-answering, image captioning, and jail-break attacks, we demonstrate
that MLLMs trained with these robust models achieve superior adversarial
robustness while maintaining favorable clean performance. Our framework
achieves 2x and 1.5x average robustness gains in captioning and VQA tasks,
respectively, and delivers over 10% improvement against jailbreak attacks. Code
and pretrained models will be available at
https://github.com/HashmatShadab/Robust-LLaVA.


http://arxiv.org/abs/2502.05208
Mitigation of Camouflaged Adversarial Attacks in Autonomous Vehicles--A Case Study Using CARLA Simulator. (92%)
Yago Romano Martinez; Brady Carter; Abhijeet Solanki; Wesam Al Amiri; Syed Rafay Hasan; Terry N. Guo
Autonomous vehicles (AVs) rely heavily on cameras and artificial intelligence (AI) to make safe and accurate driving decisions. However, since AI is the core enabling technology, this raises serious cyber threats that hinder the large-scale adoption of AVs. Therefore, it becomes crucial to analyze the resilience of AV security systems against sophisticated attacks that manipulate camera inputs, deceiving AI models. In this paper, we develop camera-camouflaged adversarial attacks targeting traffic sign recognition (TSR) in AVs. Specifically, if the attack is initiated by modifying the texture of a stop sign to fool the AV's object detection system, thereby affecting the AV actuators. The attack's effectiveness is tested using the CARLA AV simulator and the results show that such an attack can delay the auto-braking response to the stop sign, resulting in potential safety issues. We conduct extensive experiments under various conditions, confirming that our new attack is effective and robust. Additionally, we address the attack by presenting mitigation strategies. The proposed attack and defense methods are applicable to other end-to-end trained autonomous cyber-physical systems.

http://arxiv.org/abs/2502.01272
Boosting Graph Robustness Against Backdoor Attacks: An Over-Similarity Perspective. (78%)
Chang Liu; Hai Huang; Yujie Xing; Xingquan Zuo
  Graph Neural Networks (GNNs) have achieved notable success in tasks such as
social and transportation networks. However, recent studies have highlighted
the vulnerability of GNNs to backdoor attacks, raising significant concerns
about their reliability in real-world applications. Despite initial efforts to
defend against specific graph backdoor attacks, existing defense methods face
two main challenges: either the inability to establish a clear distinction
between triggers and clean nodes, resulting in the removal of many clean nodes,
or the failure to eliminate the impact of triggers, making it challenging to
restore the target nodes to their pre-attack state. Through empirical analysis
of various existing graph backdoor attacks, we observe that the triggers
generated by these methods exhibit over-similarity in both features and
structure. Based on this observation, we propose a novel graph backdoor defense
method SimGuard. We first utilizes a similarity-based metric to detect triggers
and then employs contrastive learning to train a backdoor detector that
generates embeddings capable of separating triggers from clean nodes, thereby
improving detection efficiency. Extensive experiments conducted on real-world
datasets demonstrate that our proposed method effectively defends against
various graph backdoor attacks while preserving performance on clean nodes. The
code will be released upon acceptance.


http://arxiv.org/abs/2502.01386
Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Models. (67%)
Yuyang Gong; Zhuo Chen; Miaokun Chen; Fengchang Yu; Wei Lu; Xiaofeng Wang; Xiaozhong Liu; Jiawei Liu
  Retrieval-Augmented Generation (RAG) systems based on Large Language Models
(LLMs) have become essential for tasks such as question answering and content
generation. However, their increasing impact on public opinion and information
dissemination has made them a critical focus for security research due to
inherent vulnerabilities. Previous studies have predominantly addressed attacks
targeting factual or single-query manipulations. In this paper, we address a
more practical scenario: topic-oriented adversarial opinion manipulation
attacks on RAG models, where LLMs are required to reason and synthesize
multiple perspectives, rendering them particularly susceptible to systematic
knowledge poisoning. Specifically, we propose Topic-FlipRAG, a two-stage
manipulation attack pipeline that strategically crafts adversarial
perturbations to influence opinions across related queries. This approach
combines traditional adversarial ranking attack techniques and leverages the
extensive internal relevant knowledge and reasoning capabilities of LLMs to
execute semantic-level perturbations. Experiments show that the proposed
attacks effectively shift the opinion of the model's outputs on specific
topics, significantly impacting user information perception. Current mitigation
methods cannot effectively defend against such attacks, highlighting the
necessity for enhanced safeguards for RAG systems, and offering crucial
insights for LLM security research.


http://arxiv.org/abs/2502.01385
Detecting Backdoor Samples in Contrastive Language Image Pretraining. (61%)
Hanxun Huang; Sarah Erfani; Yige Li; Xingjun Ma; James Bailey
  Contrastive language-image pretraining (CLIP) has been found to be vulnerable
to poisoning backdoor attacks where the adversary can achieve an almost perfect
attack success rate on CLIP models by poisoning only 0.01\% of the training
dataset. This raises security concerns on the current practice of pretraining
large-scale models on unscrutinized web data using CLIP. In this work, we
analyze the representations of backdoor-poisoned samples learned by CLIP models
and find that they exhibit unique characteristics in their local subspace,
i.e., their local neighborhoods are far more sparse than that of clean samples.
Based on this finding, we conduct a systematic study on detecting CLIP backdoor
attacks and show that these attacks can be easily and efficiently detected by
traditional density ratio-based local outlier detectors, whereas existing
backdoor sample detection methods fail. Our experiments also reveal that an
unintentional backdoor already exists in the original CC3M dataset and has been
trained into a popular open-source model released by OpenCLIP. Based on our
detector, one can clean up a million-scale web dataset (e.g., CC3M) efficiently
within 15 minutes using 4 Nvidia A100 GPUs. The code is publicly available in
our \href{https://github.com/HanxunH/Detect-CLIP-Backdoor-Samples}{GitHub
repository}.


http://arxiv.org/abs/2502.01936
Query-Based and Unnoticeable Graph Injection Attack from Neighborhood Perspective. (61%)
Chang Liu; Hai Huang; Yujie Xing; Xingquan Zuo
  The robustness of Graph Neural Networks (GNNs) has become an increasingly
important topic due to their expanding range of applications. Various attack
methods have been proposed to explore the vulnerabilities of GNNs, ranging from
Graph Modification Attacks (GMA) to the more practical and flexible Graph
Injection Attacks (GIA). However, existing methods face two key challenges: (i)
their reliance on surrogate models, which often leads to reduced attack
effectiveness due to structural differences and prior biases, and (ii) existing
GIA methods often sacrifice attack success rates in undefended settings to
bypass certain defense models, thereby limiting their overall effectiveness. To
overcome these limitations, we propose QUGIA, a Query-based and Unnoticeable
Graph Injection Attack. QUGIA injects nodes by first selecting edges based on
victim node connections and then generating node features using a Bayesian
framework. This ensures that the injected nodes are similar to the original
graph nodes, implicitly preserving homophily and making the attack more
unnoticeable. Unlike previous methods, QUGIA does not rely on surrogate models,
thereby avoiding performance degradation and achieving better generalization.
Extensive experiments on six real-world datasets with diverse characteristics
demonstrate that QUGIA achieves unnoticeable attacks and outperforms
state-of-the-art attackers. The code will be released upon acceptance.


http://arxiv.org/abs/2502.05211
Decoding FL Defenses: Systemization, Pitfalls, and Remedies. (50%)
Momin Ahmad Khan; Virat Shejwalkar; Yasra Chandio; Amir Houmansadr; Fatima Muhammad Anwar
  While the community has designed various defenses to counter the threat of
poisoning attacks in Federated Learning (FL), there are no guidelines for
evaluating these defenses. These defenses are prone to subtle pitfalls in their
experimental setups that lead to a false sense of security, rendering them
unsuitable for practical deployment. In this paper, we systematically
understand, identify, and provide a better approach to address these
challenges. First, we design a comprehensive systemization of FL defenses along
three dimensions: i) how client updates are processed, ii) what the server
knows, and iii) at what stage the defense is applied. Next, we thoroughly
survey 50 top-tier defense papers and identify the commonly used components in
their evaluation setups. Based on this survey, we uncover six distinct pitfalls
and study their prevalence. For example, we discover that around 30% of these
works solely use the intrinsically robust MNIST dataset, and 40% employ
simplistic attacks, which may inadvertently portray their defense as robust.
Using three representative defenses as case studies, we perform a critical
reevaluation to study the impact of the identified pitfalls and show how they
lead to incorrect conclusions about robustness. We provide actionable
recommendations to help researchers overcome each pitfall.


http://arxiv.org/abs/2502.01896
INTACT: Inducing Noise Tolerance through Adversarial Curriculum Training for LiDAR-based Safety-Critical Perception and Autonomy. (50%)
Nastaran Darabi; Divake Kumar; Sina Tayebati; Amit Ranjan Trivedi
  In this work, we present INTACT, a novel two-phase framework designed to
enhance the robustness of deep neural networks (DNNs) against noisy LiDAR data
in safety-critical perception tasks. INTACT combines meta-learning with
adversarial curriculum training (ACT) to systematically address challenges
posed by data corruption and sparsity in 3D point clouds. The meta-learning
phase equips a teacher network with task-agnostic priors, enabling it to
generate robust saliency maps that identify critical data regions. The ACT
phase leverages these saliency maps to progressively expose a student network
to increasingly complex noise patterns, ensuring targeted perturbation and
improved noise resilience. INTACT's effectiveness is demonstrated through
comprehensive evaluations on object detection, tracking, and classification
benchmarks using diverse datasets, including KITTI, Argoverse, and ModelNet40.
Results indicate that INTACT improves model robustness by up to 20% across all
tasks, outperforming standard adversarial and curriculum training methods. This
framework not only addresses the limitations of conventional training
strategies but also offers a scalable and efficient solution for real-world
deployment in resource-constrained safety-critical systems. INTACT's principled
integration of meta-learning and adversarial training establishes a new
paradigm for noise-tolerant 3D perception in safety-critical applications.
INTACT improved KITTI Multiple Object Tracking Accuracy (MOTA) by 9.6% (64.1%
-> 75.1%) and by 12.4% under Gaussian noise (52.5% -> 73.7%). Similarly, KITTI
mean Average Precision (mAP) rose from 59.8% to 69.8% (50% point drop) and
49.3% to 70.9% (Gaussian noise), highlighting the framework's ability to
enhance deep learning model resilience in safety-critical object tracking
scenarios.


http://arxiv.org/abs/2502.01152
Gradient Norm-based Fine-Tuning for Backdoor Defense in Automatic Speech Recognition. (50%)
Nanjun Zhou; Weilin Lin; Li Liu
  Backdoor attacks have posed a significant threat to the security of deep
neural networks (DNNs). Despite considerable strides in developing defenses
against backdoor attacks in the visual domain, the specialized defenses for the
audio domain remain empty. Furthermore, the defenses adapted from the visual to
audio domain demonstrate limited effectiveness. To fill this gap, we propose
Gradient Norm-based FineTuning (GN-FT), a novel defense strategy against the
attacks in the audio domain, based on the observation from the corresponding
backdoored models. Specifically, we first empirically find that the backdoored
neurons exhibit greater gradient values compared to other neurons, while clean
neurons stay the lowest. On this basis, we fine-tune the backdoored model by
incorporating the gradient norm regularization, aiming to weaken and reduce the
backdoored neurons. We further approximate the loss computation for lower
implementation costs. Extensive experiments on two speech recognition datasets
across five models demonstrate the superior performance of our proposed method.
To the best of our knowledge, this work is the first specialized and effective
defense against backdoor attacks in the audio domain.


http://arxiv.org/abs/2502.01486
Quantum Quandaries: Unraveling Encoding Vulnerabilities in Quantum Neural Networks. (10%)
Suryansh Upadhyay; Swaroop Ghosh
  Quantum computing (QC) has the potential to revolutionize fields like machine
learning, security, and healthcare. Quantum machine learning (QML) has emerged
as a promising area, enhancing learning algorithms using quantum computers.
However, QML models are lucrative targets due to their high training costs and
extensive training times. The scarcity of quantum resources and long wait times
further exacerbate the challenge. Additionally, QML providers may rely on third
party quantum clouds for hosting models, exposing them and their training data
to potential threats. As QML as a Service (QMLaaS) becomes more prevalent,
reliance on third party quantum clouds poses a significant security risk. This
work demonstrates that adversaries in quantum cloud environments can exploit
white box access to QML models to infer the users encoding scheme by analyzing
circuit transpilation artifacts. The extracted data can be reused for training
clone models or sold for profit. We validate the proposed attack through
simulations, achieving high accuracy in distinguishing between encoding
schemes. We report that 95% of the time, the encoding can be predicted
correctly. To mitigate this threat, we propose a transient obfuscation layer
that masks encoding fingerprints using randomized rotations and entanglement,
reducing adversarial detection to near random chance 42% , with a depth
overhead of 8.5% for a 5 layer QNN design.


http://arxiv.org/abs/2502.01349
Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendations. (9%)
Giorgos Filandrianos; Angeliki Dimitriou; Maria Lymperaiou; Konstantinos Thomas; Giorgos Stamou
  The advent of Large Language Models (LLMs) has revolutionized product
recommendation systems, yet their susceptibility to adversarial manipulation
poses critical challenges, particularly in real-world commercial applications.
Our approach is the first one to tap into human psychological principles,
seamlessly modifying product descriptions, making these adversarial
manipulations hard to detect. In this work, we investigate cognitive biases as
black-box adversarial strategies, drawing parallels between their effects on
LLMs and human purchasing behavior. Through extensive experiments on LLMs of
varying scales, we reveal significant vulnerabilities in their use as
recommenders, providing critical insights into safeguarding these systems.


http://arxiv.org/abs/2502.05209
Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities. (9%)
Zora Che; Stephen Casper; Robert Kirk; Anirudh Satheesh; Stewart Slocum; Lev E McKinney; Rohit Gandikota; Aidan Ewart; Domenic Rosati; Zichu Wu; Zikui Cai; Bilal Chughtai; Yarin Gal; Furong Huang; Dylan Hadfield-Menell
  Evaluations of large language model (LLM) risks and capabilities are
increasingly being incorporated into AI risk management and governance
frameworks. Currently, most risk evaluations are conducted by designing inputs
that elicit harmful behaviors from the system. However, this approach suffers
from two limitations. First, input-output evaluations cannot evaluate realistic
risks from open-weight models. Second, the behaviors identified during any
particular input-output evaluation can only lower-bound the model's
worst-possible-case input-output behavior. As a complementary method for
eliciting harmful behaviors, we propose evaluating LLMs with model tampering
attacks which allow for modifications to latent activations or weights. We pit
state-of-the-art techniques for removing harmful LLM capabilities against a
suite of 5 input-space and 6 model tampering attacks. In addition to
benchmarking these methods against each other, we show that (1) model
resilience to capability elicitation attacks lies on a low-dimensional
robustness subspace; (2) the attack success rate of model tampering attacks can
empirically predict and offer conservative estimates for the success of
held-out input-space attacks; and (3) state-of-the-art unlearning methods can
easily be undone within 16 steps of fine-tuning. Together these results
highlight the difficulty of suppressing harmful LLM capabilities and show that
model tampering attacks enable substantially more rigorous evaluations than
input-space attacks alone.


http://arxiv.org/abs/2502.01609
Breaking Focus: Contextual Distraction Curse in Large Language Models. (2%)
Yue Huang; Yanbo Wang; Zixiang Xu; Chujie Gao; Siyuan Wu; Jiayi Ye; Xiuying Chen; Pin-Yu Chen; Xiangliang Zhang
  Recent advances in Large Language Models (LLMs) have revolutionized
generative systems, achieving excellent performance across diverse domains.
Although these models perform well in controlled environments, their real-world
applications frequently encounter inputs containing both essential and
irrelevant details. Our investigation has revealed a critical vulnerability in
LLMs, which we term Contextual Distraction Vulnerability (CDV). This phenomenon
arises when models fail to maintain consistent performance on questions
modified with semantically coherent but irrelevant context. To systematically
investigate this vulnerability, we propose an efficient tree-based search
methodology to automatically generate CDV examples. Our approach successfully
generates CDV examples across four datasets, causing an average performance
degradation of approximately 45% in state-of-the-art LLMs. To address this
critical issue, we explore various mitigation strategies and find that
post-targeted training approaches can effectively enhance model robustness
against contextual distractions. Our findings highlight the fundamental nature
of CDV as an ability-level challenge rather than a knowledge-level issue since
models demonstrate the necessary knowledge by answering correctly in the
absence of distractions. This calls the community's attention to address CDV
during model development to ensure reliability. The code is available at
https://github.com/wyf23187/LLM_CDV.


http://arxiv.org/abs/2502.01154
Jailbreaking with Universal Multi-Prompts. (1%)
Yu-Ling Hsu; Hsuan Su; Shang-Tse Chen
  Large language models (LLMs) have seen rapid development in recent years,
revolutionizing various applications and significantly enhancing convenience
and productivity. However, alongside their impressive capabilities, ethical
concerns and new types of attacks, such as jailbreaking, have emerged. While
most prompting techniques focus on optimizing adversarial inputs for individual
cases, resulting in higher computational costs when dealing with large
datasets. Less research has addressed the more general setting of training a
universal attacker that can transfer to unseen tasks. In this paper, we
introduce JUMP, a prompt-based method designed to jailbreak LLMs using
universal multi-prompts. We also adapt our approach for defense, which we term
DUMP. Experimental results demonstrate that our method for optimizing universal
multi-prompts outperforms existing techniques.


http://arxiv.org/abs/2502.00765
AGNNCert: Defending Graph Neural Networks against Arbitrary Perturbations with Deterministic Certification. (98%)
Jiate Li; Binghui Wang
  Graph neural networks (GNNs) achieve the state-of-the-art on graph-relevant
tasks such as node and graph classification. However, recent works show GNNs
are vulnerable to adversarial perturbations include the perturbation on edges,
nodes, and node features, the three components forming a graph. Empirical
defenses against such attacks are soon broken by adaptive ones. While certified
defenses offer robustness guarantees, they face several limitations: 1) almost
all restrict the adversary's capability to only one type of perturbation, which
is impractical; 2) all are designed for a particular GNN task, which limits
their applicability; and 3) the robustness guarantees of all methods except one
are not 100% accurate.
  We address all these limitations by developing AGNNCert, the first certified
defense for GNNs against arbitrary (edge, node, and node feature) perturbations
with deterministic robustness guarantees, and applicable to the two most common
node and graph classification tasks. AGNNCert also encompass existing certified
defenses as special cases. Extensive evaluations on multiple benchmark
node/graph classification datasets and two real-world graph datasets, and
multiple GNNs validate the effectiveness of AGNNCert to provably defend against
arbitrary perturbations. AGNNCert also shows its superiority over the
state-of-the-art certified defenses against the individual edge perturbation
and node perturbation.


http://arxiv.org/abs/2502.05206
Safety at Scale: A Comprehensive Survey of Large Model Safety. (98%)
Xingjun Ma; Yifeng Gao; Yixu Wang; Ruofan Wang; Xin Wang; Ye Sun; Yifan Ding; Hengyuan Xu; Yunhao Chen; Yunhan Zhao; Hanxun Huang; Yige Li; Jiaming Zhang; Xiang Zheng; Yang Bai; Zuxuan Wu; Xipeng Qiu; Jingfeng Zhang; Yiming Li; Jun Sun; Cong Wang; Jindong Gu; Baoyuan Wu; Siheng Chen; Tianwei Zhang; Yang Liu; Mingming Gong; Tongliang Liu; Shirui Pan; Cihang Xie; Tianyu Pang; Yinpeng Dong; Ruoxi Jia; Yang Zhang; Shiqing Ma; Xiangyu Zhang; Neil Gong; Chaowei Xiao; Sarah Erfani; Bo Li; Masashi Sugiyama; Dacheng Tao; James Bailey; Yu-Gang Jiang
  The rapid advancement of large models, driven by their exceptional abilities
in learning and generalization through large-scale pre-training, has reshaped
the landscape of Artificial Intelligence (AI). These models are now
foundational to a wide range of applications, including conversational AI,
recommendation systems, autonomous driving, content generation, medical
diagnostics, and scientific discovery. However, their widespread deployment
also exposes them to significant safety risks, raising concerns about
robustness, reliability, and ethical implications. This survey provides a
systematic review of current safety research on large models, covering Vision
Foundation Models (VFMs), Large Language Models (LLMs), Vision-Language
Pre-training (VLP) models, Vision-Language Models (VLMs), Diffusion Models
(DMs), and large-model-based Agents. Our contributions are summarized as
follows: (1) We present a comprehensive taxonomy of safety threats to these
models, including adversarial attacks, data poisoning, backdoor attacks,
jailbreak and prompt injection attacks, energy-latency attacks, data and model
extraction attacks, and emerging agent-specific threats. (2) We review defense
strategies proposed for each type of attacks if available and summarize the
commonly used datasets and benchmarks for safety research. (3) Building on
this, we identify and discuss the open challenges in large model safety,
emphasizing the need for comprehensive safety evaluations, scalable and
effective defense mechanisms, and sustainable data practices. More importantly,
we highlight the necessity of collective efforts from the research community
and international collaboration. Our work can serve as a useful reference for
researchers and practitioners, fostering the ongoing development of
comprehensive defense systems and platforms to safeguard AI models.


http://arxiv.org/abs/2502.01027
Adversarial Robustness in Two-Stage Learning-to-Defer: Algorithms and Guarantees. (93%)
Yannis Montreuil; Axel Carlier; Lai Xing Ng; Wei Tsang Ooi
  Two-stage Learning-to-Defer (L2D) enables optimal task delegation by
assigning each input to either a fixed main model or one of several offline
experts, supporting reliable decision-making in complex, multi-agent
environments. However, existing L2D frameworks assume clean inputs and are
vulnerable to adversarial perturbations that can manipulate query
allocation--causing costly misrouting or expert overload. We present the first
comprehensive study of adversarial robustness in two-stage L2D systems. We
introduce two novel attack strategie--untargeted and targeted--which
respectively disrupt optimal allocations or force queries to specific agents.
To defend against such threats, we propose SARD, a convex learning algorithm
built on a family of surrogate losses that are provably Bayes-consistent and
$(\mathcal{R}, \mathcal{G})$-consistent. These guarantees hold across
classification, regression, and multi-task settings. Empirical results
demonstrate that SARD significantly improves robustness under adversarial
attacks while maintaining strong clean performance, marking a critical step
toward secure and trustworthy L2D deployment.


http://arxiv.org/abs/2502.00735
From Compliance to Exploitation: Jailbreak Prompt Attacks on Multimodal LLMs. (86%)
Chun Wai Chiu; Linghan Huang; Bo Li; Huaming Chen
  Large Language Models (LLMs) have seen widespread applications across various
domains due to their growing ability to process diverse types of input data,
including text, audio, image and video. While LLMs have demonstrated
outstanding performance in understanding and generating contexts for different
scenarios, they are vulnerable to prompt-based attacks, which are mostly via
text input. In this paper, we introduce the first voice-based jailbreak attack
against multimodal LLMs, termed as Flanking Attack, which can process different
types of input simultaneously towards the multimodal LLMs. Our work is
motivated by recent advancements in monolingual voice-driven large language
models, which have introduced new attack surfaces beyond traditional text-based
vulnerabilities for LLMs. To investigate these risks, we examine the frontier
multimodal LLMs, which can be accessed via different types of inputs such as
audio input, focusing on how adversarial prompts can bypass its defense
mechanisms. We propose a novel strategy, in which the disallowed prompt is
flanked by benign, narrative-driven prompts. It is integrated in the Flanking
Attack which attempts to humanizes the interaction context and execute the
attack through a fictional setting. To better evaluate the attack performance,
we present a semi-automated self-assessment framework for policy violation
detection. We demonstrate that Flank Attack is capable of manipulating
state-of-the-art LLMs into generating misaligned and forbidden outputs, which
achieves an average attack success rate ranging from 0.67 to 0.93 across seven
forbidden scenarios. These findings highlight both the potency of prompt-based
obfuscation in voice-enabled contexts and the limitations of current LLMs'
moderation safeguards and the urgent need for advanced defense strategies to
address the challenges posed by evolving, context-rich attacks.


http://arxiv.org/abs/2502.01014
Refining Adaptive Zeroth-Order Optimization at Ease. (73%)
Yao Shu; Qixin Zhang; Kun He; Zhongxiang Dai
  Recently, zeroth-order (ZO) optimization plays an essential role in scenarios
where gradient information is inaccessible or unaffordable, such as black-box
systems and resource-constrained environments. While existing adaptive methods
such as ZO-AdaMM have shown promise, they are fundamentally limited by their
underutilization of moment information during optimization, usually resulting
in underperforming convergence. To overcome these limitations, this paper
introduces Refined Adaptive Zeroth-Order Optimization (R-AdaZO). Specifically,
we first show the untapped variance reduction effect of first moment estimate
on ZO gradient estimation, which improves the accuracy and stability of ZO
updates. We then refine the second moment estimate based on these
variance-reduced gradient estimates to better capture the geometry of the
optimization landscape, enabling a more effective scaling of ZO updates. We
present rigorous theoretical analysis to show (a) the first analysis to the
variance reduction of first moment estimate in ZO optimization, (b) the
improved second moment estimates with a more accurate approximation of its
variance-free ideal, (c) the first variance-aware convergence framework for
adaptive ZO methods, which may be of independent interest, and (d) the faster
convergence of R-AdaZO than existing baselines like ZO-AdaMM. Our extensive
experiments, including synthetic problems, black-box adversarial attack, and
memory-efficient fine-tuning of large language models (LLMs), further verify
the superior convergence of R-AdaZO, indicating that R-AdaZO offers an improved
solution for real-world ZO optimization challenges.


http://arxiv.org/abs/2502.01032
Converting MLPs into Polynomials in Closed Form. (45%)
Nora Belrose; Alice Rigg
  Recent work has shown that purely quadratic functions can replace MLPs in
transformers with no significant loss in performance, while enabling new
methods of interpretability based on linear algebra. In this work, we
theoretically derive closed-form least-squares optimal approximations of
feedforward networks (multilayer perceptrons and gated linear units) using
polynomial functions of arbitrary degree. When the $R^2$ is high, this allows
us to interpret MLPs and GLUs by visualizing the eigendecomposition of the
coefficients of their linear and quadratic approximants. We also show that
these approximants can be used to create SVD-based adversarial examples. By
tracing the $R^2$ of linear and quadratic approximants across training time, we
find new evidence that networks start out simple, and get progressively more
complex. Even at the end of training, however, our quadratic approximants
explain over 95% of the variance in network outputs.


http://arxiv.org/abs/2502.00834
Boosting Adversarial Robustness and Generalization with Structural Prior. (31%)
Zhichao Hou; Weizhi Gao; Hamid Krim; Xiaorui Liu
  This work investigates a novel approach to boost adversarial robustness and
generalization by incorporating structural prior into the design of deep
learning models. Specifically, our study surprisingly reveals that existing
dictionary learning-inspired convolutional neural networks (CNNs) provide a
false sense of security against adversarial attacks. To address this, we
propose Elastic Dictionary Learning Networks (EDLNets), a novel ResNet
architecture that significantly enhances adversarial robustness and
generalization. This novel and effective approach is supported by a theoretical
robustness analysis using influence functions. Moreover, extensive and reliable
experiments demonstrate consistent and significant performance improvement on
open robustness leaderboards such as RobustBench, surpassing state-of-the-art
baselines. To the best of our knowledge, this is the first work to discover and
validate that structural prior can reliably enhance deep learning robustness
under strong adaptive attacks, unveiling a promising direction for future
research.


http://arxiv.org/abs/2502.00760
Privacy Preserving Properties of Vision Classifiers. (1%)
Pirzada Suhail; Amit Sethi
  Vision classifiers are often trained on proprietary datasets containing
sensitive information, yet the models themselves are frequently shared openly
under the privacy-preserving assumption. Although these models are assumed to
protect sensitive information in their training data, the extent to which this
assumption holds for different architectures remains unexplored. This
assumption is challenged by inversion attacks which attempt to reconstruct
training data from model weights, exposing significant privacy vulnerabilities.
In this study, we systematically evaluate the privacy-preserving properties of
vision classifiers across diverse architectures, including Multi-Layer
Perceptrons (MLPs), Convolutional Neural Networks (CNNs), and Vision
Transformers (ViTs). Using network inversion-based reconstruction techniques,
we assess the extent to which these architectures memorize and reveal training
data, quantifying the relative ease of reconstruction across models. Our
analysis highlights how architectural differences, such as input
representation, feature extraction mechanisms, and weight structures, influence
privacy risks. By comparing these architectures, we identify which are more
resilient to inversion attacks and examine the trade-offs between model
performance and privacy preservation, contributing to the development of secure
and privacy-respecting machine learning models for sensitive applications. Our
findings provide actionable insights into the design of secure and
privacy-aware machine learning systems, emphasizing the importance of
evaluating architectural decisions in sensitive applications involving
proprietary or personal data.


http://arxiv.org/abs/2502.00653
Towards Robust Multimodal Large Language Models Against Jailbreak Attacks. (98%)
Ziyi Yin; Yuanpu Cao; Han Liu; Ting Wang; Jinghui Chen; Fenhlong Ma
  While multimodal large language models (MLLMs) have achieved remarkable
success in recent advancements, their susceptibility to jailbreak attacks has
come to light. In such attacks, adversaries exploit carefully crafted prompts
to coerce models into generating harmful or undesirable content. Existing
defense mechanisms often rely on external inference steps or safety alignment
training, both of which are less effective and impractical when facing
sophisticated adversarial perturbations in white-box scenarios. To address
these challenges and bolster MLLM robustness, we introduce SafeMLLM by adopting
an adversarial training framework that alternates between an attack step for
generating adversarial noise and a model updating step. At the attack step,
SafeMLLM generates adversarial perturbations through a newly proposed
contrastive embedding attack (CoE-Attack), which optimizes token embeddings
under a contrastive objective. SafeMLLM then updates model parameters to
neutralize the perturbation effects while preserving model utility on benign
inputs. We evaluate SafeMLLM across six MLLMs and six jailbreak methods
spanning multiple modalities. Experimental results show that SafeMLLM
effectively defends against diverse attacks, maintaining robust performance and
utilities.


http://arxiv.org/abs/2502.00652
Reformulation is All You Need: Addressing Malicious Text Features in DNNs. (70%)
Yi Jiang; Oubo Ma; Yong Yang; Tong Zhang; Shouling Ji
  Human language encompasses a wide range of intricate and diverse implicit
features, which attackers can exploit to launch adversarial or backdoor
attacks, compromising DNN models for NLP tasks. Existing model-oriented
defenses often require substantial computational resources as model size
increases, whereas sample-oriented defenses typically focus on specific attack
vectors or schemes, rendering them vulnerable to adaptive attacks. We observe
that the root cause of both adversarial and backdoor attacks lies in the
encoding process of DNN models, where subtle textual features, negligible for
human comprehension, are erroneously assigned significant weight by less robust
or trojaned models. Based on it we propose a unified and adaptive defense
framework that is effective against both adversarial and backdoor attacks. Our
approach leverages reformulation modules to address potential malicious
features in textual inputs while preserving the original semantic integrity.
Extensive experiments demonstrate that our framework outperforms existing
sample-oriented defense baselines across a diverse range of malicious textual
features.


http://arxiv.org/abs/2502.00346
Actor Critic with Experience Replay-based automatic treatment planning for prostate cancer intensity modulated radiotherapy. (69%)
Md Mainul Abrar; Parvat Sapkota; Damon Sprouts; Xun Jia; Yujie Chi
  Background: Real-time treatment planning in IMRT is challenging due to
complex beam interactions. AI has improved automation, but existing models
require large, high-quality datasets and lack universal applicability. Deep
reinforcement learning (DRL) offers a promising alternative by mimicking human
trial-and-error planning.
  Purpose: Develop a stochastic policy-based DRL agent for automatic treatment
planning with efficient training, broad applicability, and robustness against
adversarial attacks using Fast Gradient Sign Method (FGSM).
  Methods: Using the Actor-Critic with Experience Replay (ACER) architecture,
the agent tunes treatment planning parameters (TPPs) in inverse planning.
Training is based on prostate cancer IMRT cases, using dose-volume histograms
(DVHs) as input. The model is trained on a single patient case, validated on
two independent cases, and tested on 300+ plans across three datasets. Plan
quality is assessed using ProKnow scores, and robustness is tested against
adversarial attacks.
  Results: Despite training on a single case, the model generalizes well.
Before ACER-based planning, the mean plan score was 6.20$\pm$1.84; after,
93.09% of cases achieved a perfect score of 9, with a mean of 8.93$\pm$0.27.
The agent effectively prioritizes optimal TPP tuning and remains robust against
adversarial attacks.
  Conclusions: The ACER-based DRL agent enables efficient, high-quality
treatment planning in prostate cancer IMRT, demonstrating strong
generalizability and robustness.


http://arxiv.org/abs/2502.00646
TrojanTime: Backdoor Attacks on Time Series Classification. (69%)
Chang Dong; Zechao Sun; Guangdong Bai; Shuying Piao; Weitong Chen; Wei Emma Zhang
  Time Series Classification (TSC) is highly vulnerable to backdoor attacks,
posing significant security threats. Existing methods primarily focus on data
poisoning during the training phase, designing sophisticated triggers to
improve stealthiness and attack success rate (ASR). However, in practical
scenarios, attackers often face restrictions in accessing training data.
Moreover, it is a challenge for the model to maintain generalization ability on
clean test data while remaining vulnerable to poisoned inputs when data is
inaccessible. To address these challenges, we propose TrojanTime, a novel
two-step training algorithm. In the first stage, we generate a pseudo-dataset
using an external arbitrary dataset through target adversarial attacks. The
clean model is then continually trained on this pseudo-dataset and its poisoned
version. To ensure generalization ability, the second stage employs a carefully
designed training strategy, combining logits alignment and batch norm freezing.
We evaluate TrojanTime using five types of triggers across four TSC
architectures in UCR benchmark datasets from diverse domains. The results
demonstrate the effectiveness of TrojanTime in executing backdoor attacks while
maintaining clean accuracy. Finally, to mitigate this threat, we propose a
defensive unlearning strategy that effectively reduces the ASR while preserving
clean accuracy.


http://arxiv.org/abs/2502.00456
Explorations of the Softmax Space: Knowing When the Neural Network Doesn't Know. (1%)
Daniel Sikar; Artur d'Avila Garcez; Tillman Weyde
  Ensuring the reliability of automated decision-making based on neural
networks will be crucial as Artificial Intelligence systems are deployed more
widely in critical situations. This paper proposes a new approach for measuring
confidence in the predictions of any neural network that relies on the
predictions of a softmax layer. We identify that a high-accuracy trained
network may have certain outputs for which there should be low confidence. In
such cases, decisions should be deferred and it is more appropriate for the
network to provide a \textit{not known} answer to a corresponding
classification task. Our approach clusters the vectors in the softmax layer to
measure distances between cluster centroids and network outputs. We show that a
cluster with centroid calculated simply as the mean softmax output for all
correct predictions can serve as a suitable proxy in the evaluation of
confidence. Defining a distance threshold for a class as the smallest distance
from an incorrect prediction to the given class centroid offers a simple
approach to adding \textit{not known} answers to any network classification
falling outside of the threshold. We evaluate the approach on the MNIST and
CIFAR-10 datasets using a Convolutional Neural Network and a Vision
Transformer, respectively. The results show that our approach is consistent
across datasets and network models, and indicate that the proposed distance
metric can offer an efficient way of determining when automated predictions are
acceptable and when they should be deferred to human operators.


http://arxiv.org/abs/2502.00384
It's Not Just a Phase: On Investigating Phase Transitions in Deep Learning-based Side-channel Analysis. (1%)
Sengim Karayalçin; Marina Krček; Stjepan Picek
  Side-channel analysis (SCA) represents a realistic threat where the attacker
can observe unintentional information to obtain secret data. Evaluation labs
also use the same SCA techniques in the security certification process. The
results in the last decade have shown that machine learning, especially deep
learning, is an extremely powerful SCA approach, allowing the breaking of
protected devices while achieving optimal attack performance. Unfortunately,
deep learning operates as a black-box, making it less useful for security
evaluators who must understand how attacks work to prevent them in the future.
This work demonstrates that mechanistic interpretability can effectively scale
to realistic scenarios where relevant information is sparse and well-defined
interchange interventions to the input are impossible due to side-channel
protections. Concretely, we reverse engineer the features the network learns
during phase transitions, eventually retrieving secret masks, allowing us to
move from black-box to white-box evaluation.


http://arxiv.org/abs/2502.05203
Adversarial Machine Learning: Attacking and Safeguarding Image Datasets. (99%)
Koushik Chowdhury
  This paper examines the vulnerabilities of convolutional neural networks
(CNNs) to adversarial attacks and explores a method for their safeguarding. In
this study, CNNs were implemented on four of the most common image datasets,
namely CIFAR-10, ImageNet, MNIST, and Fashion-MNIST, and achieved high baseline
accuracy. To assess the strength of these models, the Fast Gradient Sign Method
was used, which is a type of exploit on the model that is used to bring down
the models accuracies by adding a very minimal perturbation to the input image.
To counter the FGSM attack, a safeguarding approach went through, which
includes retraining the models on clear and pollutant or adversarial images to
increase their resistance ability. The next step involves applying FGSM again,
but this time to the adversarially trained models, to see how much the accuracy
of the models has gone down and evaluate the effectiveness of the defense. It
appears that while most level of robustness is achieved against the models
after adversarial training, there are still a few losses in the performance of
these models against adversarial perturbations. This work emphasizes the need
to create better defenses for models deployed in real-world scenarios against
adversaries.


http://arxiv.org/abs/2501.19040
Towards the Worst-case Robustness of Large Language Models. (99%)
Huanran Chen; Yinpeng Dong; Zeming Wei; Hang Su; Jun Zhu
  Recent studies have revealed the vulnerability of Large Language Models
(LLMs) to adversarial attacks, where the adversary crafts specific input
sequences to induce harmful, violent, private, or incorrect outputs. Although
various defenses have been proposed, they have not been evaluated by strong
adaptive attacks, leaving the worst-case robustness of LLMs still intractable.
By developing a stronger white-box attack, our evaluation results indicate that
most typical defenses achieve nearly 0\% robustness.To solve this, we propose
\textit{DiffTextPure}, a general defense that diffuses the (adversarial) input
prompt using any pre-defined smoothing distribution, and purifies the diffused
input using a pre-trained language model. Theoretically, we derive tight
robustness lower bounds for all smoothing distributions using Fractal Knapsack
or 0-1 Knapsack solvers. Under this framework, we certify the robustness of a
specific case -- smoothing LLMs using a uniform kernel -- against \textit{any
possible attack} with an average $\ell_0$ perturbation of 2.02 or an average
suffix length of 6.41.


http://arxiv.org/abs/2501.19202
Improving LLM Unlearning Robustness via Random Perturbations. (83%)
Dang Huu-Tien; Hoang Thanh-Tung; Anh Bui; Le-Minh Nguyen; Naoya Inoue
  In this paper, we show that current state-of-the-art LLM unlearning methods
inherently reduce models' robustness, causing them to misbehave even when a
single non-adversarial forget-token is in the retain-query. Toward
understanding underlying causes, we reframe the unlearning process as backdoor
attacks and defenses: forget-tokens act as backdoor triggers that, when
activated in retain-queries, cause disruptions in unlearned models' behaviors,
similar to successful backdoor attacks. To mitigate this vulnerability, we
propose Random Noise Augmentation (RNA) -- a plug-and-play, model and method
agnostic approach with theoretical guarantees for improving the robustness of
unlearned models. Extensive experiments demonstrate that RNA significantly
improves the robustness of unlearned models, maintains unlearning performances
while introducing no additional computational overhead.


http://arxiv.org/abs/2501.19403
Redefining Machine Unlearning: A Conformal Prediction-Motivated Approach. (83%)
Yingdan Shi; Ren Wang
  Machine unlearning seeks to systematically remove specified data from a
trained model, effectively achieving a state as though the data had never been
encountered during training. While metrics such as Unlearning Accuracy (UA) and
Membership Inference Attack (MIA) provide a baseline for assessing unlearning
performance, they fall short of evaluating the completeness and reliability of
forgetting. This is because the ground truth labels remain potential candidates
within the scope of uncertainty quantification, leaving gaps in the evaluation
of true forgetting. In this paper, we identify critical limitations in existing
unlearning metrics and propose enhanced evaluation metrics inspired by
conformal prediction. Our metrics can effectively capture the extent to which
ground truth labels are excluded from the prediction set. Furthermore, we
observe that many existing machine unlearning methods do not achieve
satisfactory forgetting performance when evaluated with our new metrics. To
address this, we propose an unlearning framework that integrates conformal
prediction insights into Carlini & Wagner adversarial attack loss. Extensive
experiments on the image classification task demonstrate that our enhanced
metrics offer deeper insights into unlearning effectiveness, and that our
unlearning framework significantly improves the forgetting quality of
unlearning methods.


http://arxiv.org/abs/2501.18998
Adversarial Attacks on AI-Generated Text Detection Models: A Token Probability-Based Approach Using Embeddings. (81%)
Ahmed K. Kadhim; Lei Jiao; Rishad Shafik; Ole-Christoffer Granmo
  In recent years, text generation tools utilizing Artificial Intelligence (AI)
have occasionally been misused across various domains, such as generating
student reports or creative writings. This issue prompts plagiarism detection
services to enhance their capabilities in identifying AI-generated content.
Adversarial attacks are often used to test the robustness of AI-text generated
detectors. This work proposes a novel textual adversarial attack on the
detection models such as Fast-DetectGPT. The method employs embedding models
for data perturbation, aiming at reconstructing the AI generated texts to
reduce the likelihood of detection of the true origin of the texts.
Specifically, we employ different embedding techniques, including the Tsetlin
Machine (TM), an interpretable approach in machine learning for this purpose.
By combining synonyms and embedding similarity vectors, we demonstrates the
state-of-the-art reduction in detection scores against Fast-DetectGPT.
Particularly, in the XSum dataset, the detection score decreased from 0.4431 to
0.2744 AUROC, and in the SQuAD dataset, it dropped from 0.5068 to 0.3532 AUROC.


http://arxiv.org/abs/2501.19180
Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning. (62%)
Xianglin Yang; Gelei Deng; Jieming Shi; Tianwei Zhang; Jin Song Dong
  Large language models (LLMs) are vital for a wide range of applications yet
remain susceptible to jailbreak threats, which could lead to the generation of
inappropriate responses. Conventional defenses, such as refusal and adversarial
training, often fail to cover corner cases or rare domains, leaving LLMs still
vulnerable to more sophisticated attacks. We propose a novel defense strategy,
Safety Chain-of-Thought (SCoT), which harnesses the enhanced \textit{reasoning
capabilities} of LLMs for proactive assessment of harmful inputs, rather than
simply blocking them. SCoT augments any refusal training datasets to critically
analyze the intent behind each request before generating answers. By employing
proactive reasoning, SCoT enhances the generalization of LLMs across varied
harmful queries and scenarios not covered in the safety alignment corpus.
Additionally, it generates detailed refusals specifying the rules violated.
Comparative evaluations show that SCoT significantly surpasses existing
defenses, reducing vulnerability to out-of-distribution issues and adversarial
manipulations while maintaining strong general capabilities.


http://arxiv.org/abs/2501.18934
Deep Learning Model Inversion Attacks and Defenses: A Comprehensive Survey. (54%)
Wencheng Yang; Song Wang; Di Wu; Taotao Cai; Yanming Zhu; Shicheng Wei; Yiying Zhang; Xu Yang; Yan Li
  The rapid adoption of deep learning in sensitive domains has brought
tremendous benefits. However, this widespread adoption has also given rise to
serious vulnerabilities, particularly model inversion (MI) attacks, posing a
significant threat to the privacy and integrity of personal data. The
increasing prevalence of these attacks in applications such as biometrics,
healthcare, and finance has created an urgent need to understand their
mechanisms, impacts, and defense methods. This survey aims to fill the gap in
the literature by providing a structured and in-depth review of MI attacks and
defense strategies. Our contributions include a systematic taxonomy of MI
attacks, extensive research on attack techniques and defense mechanisms, and a
discussion about the challenges and future research directions in this evolving
field. By exploring the technical and ethical implications of MI attacks, this
survey aims to offer insights into the impact of AI-powered systems on privacy,
security, and trust. In conjunction with this survey, we have developed a
comprehensive repository to support research on MI attacks and defenses. The
repository includes state-of-the-art research papers, datasets, evaluation
metrics, and other resources to meet the needs of both novice and experienced
researchers interested in MI attacks and defenses, as well as the broader field
of AI security and privacy. The repository will be continuously maintained to
ensure its relevance and utility. It is accessible at
https://github.com/overgter/Deep-Learning-Model-Inversion-Attacks-and-Defenses.


http://arxiv.org/abs/2501.19143
Imitation Game for Adversarial Disillusion with Multimodal Generative Chain-of-Thought Role-Play. (45%)
Ching-Chun Chang; Fan-Yun Chen; Shih-Hong Gu; Kai Gao; Hanrui Wang; Isao Echizen
  As the cornerstone of artificial intelligence, machine perception confronts a
fundamental threat posed by adversarial illusions. These adversarial attacks
manifest in two primary forms: deductive illusion, where specific stimuli are
crafted based on the victim model's general decision logic, and inductive
illusion, where the victim model's general decision logic is shaped by specific
stimuli. The former exploits the model's decision boundaries to create a
stimulus that, when applied, interferes with its decision-making process. The
latter reinforces a conditioned reflex in the model, embedding a backdoor
during its learning phase that, when triggered by a stimulus, causes aberrant
behaviours. The multifaceted nature of adversarial illusions calls for a
unified defence framework, addressing vulnerabilities across various forms of
attack. In this study, we propose a disillusion paradigm based on the concept
of an imitation game. At the heart of the imitation game lies a multimodal
generative agent, steered by chain-of-thought reasoning, which observes,
internalises and reconstructs the semantic essence of a sample, liberated from
the classic pursuit of reversing the sample to its original state. As a proof
of concept, we conduct experimental simulations using a multimodal generative
dialogue agent and evaluates the methodology under a variety of attack
scenarios.


http://arxiv.org/abs/2501.19089
Understanding Oversmoothing in GNNs as Consensus in Opinion Dynamics. (41%)
Keqin Wang; Yulong Yang; Ishan Saha; Christine Allen-Blanchette
  In contrast to classes of neural networks where the learned representations
become increasingly expressive with network depth, the learned representations
in graph neural networks (GNNs), tend to become increasingly similar. This
phenomena, known as oversmoothing, is characterized by learned representations
that cannot be reliably differentiated leading to reduced predictive
performance. In this paper, we propose an analogy between oversmoothing in GNNs
and consensus or agreement in opinion dynamics. Through this analogy, we show
that the message passing structure of recent continuous-depth GNNs is
equivalent to a special case of opinion dynamics (i.e., linear consensus
models) which has been theoretically proven to converge to consensus (i.e.,
oversmoothing) for all inputs. Using the understanding developed through this
analogy, we design a new continuous-depth GNN model based on nonlinear opinion
dynamics and prove that our model, which we call behavior-inspired message
passing neural network (BIMP) circumvents oversmoothing for general inputs.
Through extensive experiments, we show that BIMP is robust to oversmoothing and
adversarial attack, and consistently outperforms competitive baselines on
numerous benchmarks.


http://arxiv.org/abs/2502.00156
ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition. (1%)
Joseph Fioresi; Ishan Rajendrakumar Dave; Mubarak Shah
  Bias in machine learning models can lead to unfair decision making, and while
it has been well-studied in the image and text domains, it remains
underexplored in action recognition. Action recognition models often suffer
from background bias (i.e., inferring actions based on background cues) and
foreground bias (i.e., relying on subject appearance), which can be detrimental
to real-life applications such as autonomous vehicles or assisted living
monitoring. While prior approaches have mainly focused on mitigating background
bias using specialized augmentations, we thoroughly study both biases. We
propose ALBAR, a novel adversarial training method that mitigates foreground
and background biases without requiring specialized knowledge of the bias
attributes. Our framework applies an adversarial cross-entropy loss to the
sampled static clip (where all the frames are the same) and aims to make its
class probabilities uniform using a proposed entropy maximization loss.
Additionally, we introduce a gradient penalty loss for regularization against
the debiasing process. We evaluate our method on established background and
foreground bias protocols, setting a new state-of-the-art and strongly
improving combined debiasing performance by over 12% on HMDB51. Furthermore, we
identify an issue of background leakage in the existing UCF101 protocol for
bias evaluation which provides a shortcut to predict actions and does not
provide an accurate measure of the debiasing capability of a model. We address
this issue by proposing more fine-grained segmentation boundaries for the
actor, where our method also outperforms existing approaches. Project Page:
https://joefioresi718.github.io/ALBAR_webpage/


http://arxiv.org/abs/2501.18877
Distorting Embedding Space for Safety: A Defense Mechanism for Adversarially Robust Diffusion Models. (92%)
Jaesin Ahn; Heechul Jung
  Text-to-image diffusion models show remarkable generation performance
following text prompts, but risk generating Not Safe For Work (NSFW) contents
from unsafe prompts. Existing approaches, such as prompt filtering or concept
unlearning, fail to defend against adversarial attacks while maintaining benign
image quality. In this paper, we propose a novel approach called Distorting
Embedding Space (DES), a text encoder-based defense mechanism that effectively
tackles these issues through innovative embedding space control. DES transforms
unsafe embeddings, extracted from a text encoder using unsafe prompts, toward
carefully calculated safe embedding regions to prevent unsafe contents
generation, while reproducing the original safe embeddings. DES also
neutralizes the nudity embedding, extracted using prompt ``nudity", by aligning
it with neutral embedding to enhance robustness against adversarial attacks.
These methods ensure both robust defense and high-quality image generation.
Additionally, DES can be adopted in a plug-and-play manner and requires zero
inference overhead, facilitating its deployment. Extensive experiments on
diverse attack types, including black-box and white-box scenarios, demonstrate
DES's state-of-the-art performance in both defense capability and benign image
generation quality. Our model is available at https://github.com/aei13/DES.


http://arxiv.org/abs/2501.18536
Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges. (83%)
Manveer Singh Tamber; Jimmy Lin
  Consider a scenario in which a user searches for information, only to
encounter texts flooded with misleading or non-relevant content. This scenario
exemplifies a simple yet potent vulnerability in neural Information Retrieval
(IR) pipelines: content injection attacks. We find that embedding models for
retrieval, rerankers, and large language model (LLM) relevance judges are
vulnerable to these attacks, in which adversaries insert misleading text into
passages to manipulate model judgements. We identify two primary threats: (1)
inserting unrelated or harmful content within passages that still appear
deceptively "relevant", and (2) inserting entire queries or key query terms
into passages to boost their perceived relevance. While the second tactic has
been explored in prior research, we present, to our knowledge, the first
empirical analysis of the first threat, demonstrating how state-of-the-art
models can be easily misled. Our study systematically examines the factors that
influence an attack's success, such as the placement of injected content and
the balance between relevant and non-relevant material. Additionally, we
explore various defense strategies, including adversarial passage classifiers,
retriever fine-tuning to discount manipulated content, and prompting LLM judges
to adopt a more cautious approach. However, we find that these countermeasures
often involve trade-offs, sacrificing effectiveness for attack robustness and
sometimes penalizing legitimate documents in the process. Our findings
highlight the need for stronger defenses against these evolving adversarial
strategies to maintain the trustworthiness of IR systems. We release our code
and scripts to facilitate further research.


http://arxiv.org/abs/2501.18841
Trading Inference-Time Compute for Adversarial Robustness. (68%)
Wojciech Zaremba; Evgenia Nitishinskaya; Boaz Barak; Stephanie Lin; Sam Toyer; Yaodong Yu; Rachel Dias; Eric Wallace; Kai Xiao; Johannes Heidecke; Amelia Glaese
  We conduct experiments on the impact of increasing inference-time compute in
reasoning models (specifically OpenAI o1-preview and o1-mini) on their
robustness to adversarial attacks. We find that across a variety of attacks,
increased inference-time compute leads to improved robustness. In many cases
(with important exceptions), the fraction of model samples where the attack
succeeds tends to zero as the amount of test-time compute grows. We perform no
adversarial training for the tasks we study, and we increase inference-time
compute by simply allowing the models to spend more compute on reasoning,
independently of the form of attack. Our results suggest that inference-time
compute has the potential to improve adversarial robustness for Large Language
Models. We also explore new attacks directed at reasoning models, as well as
settings where inference-time compute does not improve reliability, and
speculate on the reasons for these as well as ways to address them.


http://arxiv.org/abs/2501.18280
Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models. (2%)
Haoyu Liang; Youran Sun; Yunfeng Cai; Jun Zhu; Bo Zhang
  The security issue of large language models (LLMs) has gained wide attention
recently, with various defense mechanisms developed to prevent harmful output,
among which safeguards based on text embedding models serve as a fundamental
defense. Through testing, we discover that the output distribution of text
embedding models is severely biased with a large mean. Inspired by this
observation, we propose novel, efficient methods to search for **universal
magic words** that attack text embedding models. Universal magic words as
suffixes can shift the embedding of any text towards the bias direction, thus
manipulating the similarity of any text pair and misleading safeguards.
Attackers can jailbreak the safeguards by appending magic words to user prompts
and requiring LLMs to end answers with magic words. Experiments show that magic
word attacks significantly degrade safeguard performance on JailbreakBench,
cause real-world chatbots to produce harmful outputs in full-pipeline attacks,
and generalize across input/output texts, models, and languages. To eradicate
this security risk, we also propose defense methods against such attacks, which
can correct the bias of text embeddings and improve downstream performance in a
train-free manner.


http://arxiv.org/abs/2501.18837
Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming. (2%)
Mrinank Sharma; Meg Tong; Jesse Mu; Jerry Wei; Jorrit Kruthoff; Scott Goodfriend; Euan Ong; Alwin Peng; Raj Agarwal; Cem Anil; Amanda Askell; Nathan Bailey; Joe Benton; Emma Bluemke; Samuel R. Bowman; Eric Christiansen; Hoagy Cunningham; Andy Dau; Anjali Gopal; Rob Gilson; Logan Graham; Logan Howard; Nimit Kalra; Taesung Lee; Kevin Lin; Peter Lofgren; Francesco Mosconi; Clare O'Hara; Catherine Olsson; Linda Petrini; Samir Rajani; Nikhil Saxena; Alex Silverstein; Tanya Singh; Theodore Sumers; Leonard Tang; Kevin K. Troy; Constantin Weisser; Ruiqi Zhong; Giulio Zhou; Jan Leike; Jared Kaplan; Ethan Perez
  Large language models (LLMs) are vulnerable to universal jailbreaks-prompting
strategies that systematically bypass model safeguards and enable users to
carry out harmful processes that require many model interactions, like
manufacturing illegal substances at scale. To defend against these attacks, we
introduce Constitutional Classifiers: safeguards trained on synthetic data,
generated by prompting LLMs with natural language rules (i.e., a constitution)
specifying permitted and restricted content. In over 3,000 estimated hours of
red teaming, no red teamer found a universal jailbreak that could extract
information from an early classifier-guarded LLM at a similar level of detail
to an unguarded model across most target queries. On automated evaluations,
enhanced classifiers demonstrated robust defense against held-out
domain-specific jailbreaks. These classifiers also maintain deployment
viability, with an absolute 0.38% increase in production-traffic refusals and a
23.7% inference overhead. Our work demonstrates that defending against
universal jailbreaks while maintaining practical deployment viability is
tractable.


http://arxiv.org/abs/2501.18006
Topological Signatures of Adversaries in Multimodal Alignments. (93%)
Minh Vu; Geigh Zollicoffer; Huy Mai; Ben Nebgen; Boian Alexandrov; Manish Bhattarai
  Multimodal Machine Learning systems, particularly those aligning text and
image data like CLIP/BLIP models, have become increasingly prevalent, yet
remain susceptible to adversarial attacks. While substantial research has
addressed adversarial robustness in unimodal contexts, defense strategies for
multimodal systems are underexplored. This work investigates the topological
signatures that arise between image and text embeddings and shows how
adversarial attacks disrupt their alignment, introducing distinctive
signatures. We specifically leverage persistent homology and introduce two
novel Topological-Contrastive losses based on Total Persistence and Multi-scale
kernel methods to analyze the topological signatures introduced by adversarial
perturbations. We observe a pattern of monotonic changes in the proposed
topological losses emerging in a wide range of attacks on image-text
alignments, as more adversarial samples are introduced in the data. By
designing an algorithm to back-propagate these signatures to input samples, we
are able to integrate these signatures into Maximum Mean Discrepancy tests,
creating a novel class of tests that leverage topological signatures for better
adversarial detection.


http://arxiv.org/abs/2501.17667
CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximization. (92%)
Derui Wang; Kristen Moore; Diksha Goel; Minjune Kim; Gang Li; Yang Li; Robin Doss; Minhui Xue; Bo Li; Seyit Camtepe; Liming Zhu
  Deep reinforcement learning (DRL) has gained widespread adoption in control
and decision-making tasks due to its strong performance in dynamic
environments. However, DRL agents are vulnerable to noisy observations and
adversarial attacks, and concerns about the adversarial robustness of DRL
systems have emerged. Recent efforts have focused on addressing these
robustness issues by establishing rigorous theoretical guarantees for the
returns achieved by DRL agents in adversarial settings. Among these approaches,
policy smoothing has proven to be an effective and scalable method for
certifying the robustness of DRL agents. Nevertheless, existing certifiably
robust DRL relies on policies trained with simple Gaussian augmentations,
resulting in a suboptimal trade-off between certified robustness and certified
return. To address this issue, we introduce a novel paradigm dubbed
\texttt{C}ertified-r\texttt{A}dius-\texttt{M}aximizing \texttt{P}olicy
(\texttt{CAMP}) training. \texttt{CAMP} is designed to enhance DRL policies,
achieving better utility without compromising provable robustness. By
leveraging the insight that the global certified radius can be derived from
local certified radii based on training-time statistics, \texttt{CAMP}
formulates a surrogate loss related to the local certified radius and optimizes
the policy guided by this surrogate loss. We also introduce \textit{policy
imitation} as a novel technique to stabilize \texttt{CAMP} training.
Experimental results demonstrate that \texttt{CAMP} significantly improves the
robustness-return trade-off across various tasks. Based on the results,
\texttt{CAMP} can achieve up to twice the certified expected return compared to
that of baselines. Our code is available at
https://github.com/NeuralSec/camp-robust-rl.


http://arxiv.org/abs/2501.18098
Disentangling Safe and Unsafe Corruptions via Anisotropy and Locality. (87%)
Ramchandran Muthukumar; Ambar Pal; Jeremias Sulam; Rene Vidal
  State-of-the-art machine learning systems are vulnerable to small
perturbations to their input, where ``small'' is defined according to a threat
model that assigns a positive threat to each perturbation. Most prior works
define a task-agnostic, isotropic, and global threat, like the $\ell_p$ norm,
where the magnitude of the perturbation fully determines the degree of the
threat and neither the direction of the attack nor its position in space
matter. However, common corruptions in computer vision, such as blur,
compression, or occlusions, are not well captured by such threat models. This
paper proposes a novel threat model called \texttt{Projected Displacement} (PD)
to study robustness beyond existing isotropic and global threat models. The
proposed threat model measures the threat of a perturbation via its alignment
with \textit{unsafe directions}, defined as directions in the input space along
which a perturbation of sufficient magnitude changes the ground truth class
label. Unsafe directions are identified locally for each input based on
observed training data. In this way, the PD threat model exhibits anisotropy
and locality. Experiments on Imagenet-1k data indicate that, for any input, the
set of perturbations with small PD threat includes \textit{safe} perturbations
of large $\ell_p$ norm that preserve the true label, such as noise, blur and
compression, while simultaneously excluding \textit{unsafe} perturbations that
alter the true label. Unlike perceptual threat models based on embeddings of
large-vision models, the PD threat model can be readily computed for arbitrary
classification tasks without pre-training or finetuning. Further additional
task annotation such as sensitivity to image regions or concept hierarchies can
be easily integrated into the assessment of threat and thus the PD threat model
presents practitioners with a flexible, task-driven threat specification.


http://arxiv.org/abs/2501.18052
SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders. (47%)
Bartosz Cywiński; Kamil Deja
  Recent machine unlearning approaches offer promising solution for removing
unwanted concepts from diffusion models. However, traditional methods, which
largely rely on fine-tuning, provide little insight into the changes they
introduce to the base model, making it unclear whether concepts are truly
removed or only masked. In this work, we introduce SAeUron, a novel method
leveraging features learned by sparse autoencoders (SAEs) to unlearn unwanted
concepts in text-to-image diffusion models. First, we demonstrate that SAEs,
trained in an unsupervised manner on activations from multiple denoising
timesteps of the diffusion model, capture sparse and interpretable features
corresponding to specific concepts. Building on this, we propose a method of
selecting concept-specific features. This enables precise interventions on the
model's activations to block targeted content while preserving the model's
overall performance. Evaluation on the competitive UnlearnCanvas benchmark on
object and style unlearning highlights SAeUron's state-of-the-art performance.
Moreover, we show that with a single SAE, we can remove multiple concepts
simultaneously and that in contrast to other methods, SAeUron dismisses the
possibility of generating unwanted content, even under adversarial attack.


http://arxiv.org/abs/2501.17813
P-TAME: Explain Any Image Classifier with Trained Perturbations. (3%)
Mariano V. Ntrougkas; Vasileios Mezaris; Ioannis Patras
  The adoption of Deep Neural Networks (DNNs) in critical fields where
predictions need to be accompanied by justifications is hindered by their
inherent black-box nature. In this paper, we introduce P-TAME
(Perturbation-based Trainable Attention Mechanism for Explanations), a
model-agnostic method for explaining DNN-based image classifiers. P-TAME
employs an auxiliary image classifier to extract features from the input image,
bypassing the need to tailor the explanation method to the internal
architecture of the backbone classifier being explained. Unlike traditional
perturbation-based methods, which have high computational requirements, P-TAME
offers an efficient alternative by generating high-resolution explanations in a
single forward pass during inference. We apply P-TAME to explain the decisions
of VGG-16, ResNet-50, and ViT-B-16, three distinct and widely used image
classifiers. Quantitative and qualitative results show that our method matches
or outperforms previous explainability methods, including model-specific
approaches. Code and trained models will be released upon acceptance.


http://arxiv.org/abs/2501.17501
How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning. (1%)
Fabio Salerno; Ali Al-Kaswan; Maliheh Izadi
  Code language models, while widely popular, are often trained on unsanitized
source code gathered from across the Internet. Previous work revealed that
pre-trained models can remember the content of their training data and
regurgitate them through data extraction attacks. Due to the large size of
current models, only a few entities have the resources for pre-training such
models. However, fine-tuning requires fewer resources and is increasingly used
by both small and large entities for its effectiveness on specialized data.
Such small curated data for fine-tuning might contain sensitive information or
proprietary assets. In this study, we attack both pre-trained and fine-tuned
code language models to investigate the extent of data extractability. We first
develop a custom benchmark to assess the vulnerability of both pre-training and
fine-tuning samples to extraction attacks. Our findings reveal that 54.9% of
extractable pre-training data could be retrieved from StarCoder2-15B, whereas
this number decreased to 23.5% after fine-tuning. This indicates that
fine-tuning reduces the extractability of pre-training data. However, compared
to larger models, fine-tuning smaller models increases their vulnerability to
data extraction attacks on fine-tuning data. Given the potential sensitivity of
fine-tuning data, this can lead to more severe consequences. Lastly, we also
manually analyzed 2000 extractable samples before and after fine-tuning. We
also found that data carriers and licensing information are the most likely
data categories to be memorized from pre-trained and fine-tuned models, while
the latter is the most likely to be forgotten after fine-tuning.


http://arxiv.org/abs/2501.16843
Bones of Contention: Exploring Query-Efficient Attacks Against Skeleton Recognition Systems. (99%)
Yuxin Cao; Kai Ye; Derui Wang; Minhui Xue; Hao Ge; Chenxiong Qian; Jin Song Dong
  Skeleton action recognition models have secured more attention than
video-based ones in various applications due to privacy preservation and lower
storage requirements. Skeleton data are typically transmitted to cloud servers
for action recognition, with results returned to clients via Apps/APIs.
However, the vulnerability of skeletal models against adversarial perturbations
gradually reveals the unreliability of these systems. Existing black-box
attacks all operate in a decision-based manner, resulting in numerous queries
that hinder efficiency and feasibility in real-world applications. Moreover,
all attacks off the shelf focus on only restricted perturbations, while
ignoring model weaknesses when encountered with non-semantic perturbations. In
this paper, we propose two query-effIcient Skeletal Adversarial AttaCks,
ISAAC-K and ISAAC-N. As a black-box attack, ISAAC-K utilizes Grad-CAM in a
surrogate model to extract key joints where minor sparse perturbations are then
added to fool the classifier. To guarantee natural adversarial motions, we
introduce constraints of both bone length and temporal consistency. ISAAC-K
finds stronger adversarial examples on $\ell_\infty$ norm, which can encompass
those on other norms. Exhaustive experiments substantiate that ISAAC-K can
uplift the attack efficiency of the perturbations under 10 skeletal models.
Additionally, as a byproduct, ISAAC-N fools the classifier by replacing
skeletons unrelated to the action. We surprisingly find that skeletal models
are vulnerable to large perturbations where the part-wise non-semantic joints
are just replaced, leading to a query-free no-box attack without any prior
knowledge. Based on that, four adaptive defenses are eventually proposed to
improve the robustness of skeleton recognition models.


http://arxiv.org/abs/2501.16750
HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaigns. (97%)
Xinyue Shen; Yixin Wu; Yiting Qu; Michael Backes; Savvas Zannettou; Yang Zhang
  Large Language Models (LLMs) have raised increasing concerns about their
misuse in generating hate speech. Among all the efforts to address this issue,
hate speech detectors play a crucial role. However, the effectiveness of
different detectors against LLM-generated hate speech remains largely unknown.
In this paper, we propose HateBench, a framework for benchmarking hate speech
detectors on LLM-generated hate speech. We first construct a hate speech
dataset of 7,838 samples generated by six widely-used LLMs covering 34 identity
groups, with meticulous annotations by three labelers. We then assess the
effectiveness of eight representative hate speech detectors on the
LLM-generated dataset. Our results show that while detectors are generally
effective in identifying LLM-generated hate speech, their performance degrades
with newer versions of LLMs. We also reveal the potential of LLM-driven hate
campaigns, a new threat that LLMs bring to the field of hate speech detection.
By leveraging advanced techniques like adversarial attacks and model stealing
attacks, the adversary can intentionally evade the detector and automate hate
campaigns online. The most potent adversarial attack achieves an attack success
rate of 0.966, and its attack efficiency can be further improved by
$13-21\times$ through model stealing attacks with acceptable attack
performance. We hope our study can serve as a call to action for the research
community and platform moderators to fortify defenses against these emerging
threats.


http://arxiv.org/abs/2501.16904
Adversarial Masked Autoencoder Purifier with Defense Transferability. (75%)
Yuan-Chih Chen; Chun-Shien Lu
  The study of adversarial defense still struggles to combat with advanced
adversarial attacks. In contrast to most prior studies that rely on the
diffusion model for test-time defense to remarkably increase the inference
time, we propose Masked AutoEncoder Purifier (MAEP), which integrates Masked
AutoEncoder (MAE) into an adversarial purifier framework for test-time
purification. While MAEP achieves promising adversarial robustness, it
particularly features model defense transferability and attack generalization
without relying on using additional data that is different from the training
dataset. To our knowledge, MAEP is the first study of adversarial purifier
based on MAE. Extensive experimental results demonstrate that our method can
not only maintain clear accuracy with only a slight drop but also exhibit a
close gap between the clean and robust accuracy. Notably, MAEP trained on
CIFAR10 achieves state-of-the-art performance even when tested directly on
ImageNet, outperforming existing diffusion-based models trained specifically on
ImageNet.


http://arxiv.org/abs/2501.17151
Scanning Trojaned Models Using Out-of-Distribution Samples. (33%)
Hossein Mirzaei; Ali Ansari; Bahar Dibaei Nia; Mojtaba Nafez; Moein Madadi; Sepehr Rezaee; Zeinab Sadat Taghavi; Arad Maleki; Kian Shamsaie; Mahdi Hajialilue; Jafar Habibi; Mohammad Sabokrou; Mohammad Hossein Rohban
  Scanning for trojan (backdoor) in deep neural networks is crucial due to
their significant real-world applications. There has been an increasing focus
on developing effective general trojan scanning methods across various trojan
attacks. Despite advancements, there remains a shortage of methods that perform
effectively without preconceived assumptions about the backdoor attack method.
Additionally, we have observed that current methods struggle to identify
classifiers trojaned using adversarial training. Motivated by these challenges,
our study introduces a novel scanning method named TRODO (TROjan scanning by
Detection of adversarial shifts in Out-of-distribution samples). TRODO
leverages the concept of "blind spots"--regions where trojaned classifiers
erroneously identify out-of-distribution (OOD) samples as in-distribution (ID).
We scan for these blind spots by adversarially shifting OOD samples towards
in-distribution. The increased likelihood of perturbed OOD samples being
classified as ID serves as a signature for trojan detection. TRODO is both
trojan and label mapping agnostic, effective even against adversarially trained
trojaned classifiers. It is applicable even in scenarios where training data is
absent, demonstrating high accuracy and adaptability across various scenarios
and datasets, highlighting its potential as a robust trojan scanning strategy.


http://arxiv.org/abs/2501.16902
Document Screenshot Retrievers are Vulnerable to Pixel Poisoning Attacks. (22%)
Shengyao Zhuang; Ekaterina Khramtsova; Xueguang Ma; Bevan Koopman; Jimmy Lin; Guido Zuccon
  Recent advancements in dense retrieval have introduced vision-language model
(VLM)-based retrievers, such as DSE and ColPali, which leverage document
screenshots embedded as vectors to enable effective search and offer a
simplified pipeline over traditional text-only methods. In this study, we
propose three pixel poisoning attack methods designed to compromise VLM-based
retrievers and evaluate their effectiveness under various attack settings and
parameter configurations. Our empirical results demonstrate that injecting even
a single adversarial screenshot into the retrieval corpus can significantly
disrupt search results, poisoning the top-10 retrieved documents for 41.9% of
queries in the case of DSE and 26.4% for ColPali. These vulnerability rates
notably exceed those observed with equivalent attacks on text-only retrievers.
Moreover, when targeting a small set of known queries, the attack success rate
raises, achieving complete success in certain cases. By exposing the
vulnerabilities inherent in vision-language models, this work highlights the
potential risks associated with their deployment.


http://arxiv.org/abs/2501.17381
Do We Really Need to Design New Byzantine-robust Aggregation Rules? (13%)
Minghong Fang; Seyedsina Nabavirazavi; Zhuqing Liu; Wei Sun; Sundararaja Sitharama Iyengar; Haibo Yang
  Federated learning (FL) allows multiple clients to collaboratively train a
global machine learning model through a server, without exchanging their
private training data. However, the decentralized aspect of FL makes it
susceptible to poisoning attacks, where malicious clients can manipulate the
global model by sending altered local model updates. To counter these attacks,
a variety of aggregation rules designed to be resilient to Byzantine failures
have been introduced. Nonetheless, these methods can still be vulnerable to
sophisticated attacks or depend on unrealistic assumptions about the server. In
this paper, we demonstrate that there is no need to design new Byzantine-robust
aggregation rules; instead, FL can be secured by enhancing the robustness of
well-established aggregation rules. To this end, we present FoundationFL, a
novel defense mechanism against poisoning attacks. FoundationFL involves the
server generating synthetic updates after receiving local model updates from
clients. It then applies existing Byzantine-robust foundational aggregation
rules, such as Trimmed-mean or Median, to combine clients' model updates with
the synthetic ones. We theoretically establish the convergence performance of
FoundationFL under Byzantine settings. Comprehensive experiments across several
real-world datasets validate the efficiency of our FoundationFL method.


http://arxiv.org/abs/2501.18638
Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation. (12%)
Daniel Schwartz; Dmitriy Bespalov; Zhe Wang; Ninad Kulkarni; Yanjun Qi
  As large language models (LLMs) become increasingly prevalent, ensuring their
robustness against adversarial misuse is crucial. This paper introduces the GAP
(Graph of Attacks with Pruning) framework, an advanced approach for generating
stealthy jailbreak prompts to evaluate and enhance LLM safeguards. GAP
addresses limitations in existing tree-based LLM jailbreak methods by
implementing an interconnected graph structure that enables knowledge sharing
across attack paths. Our experimental evaluation demonstrates GAP's superiority
over existing techniques, achieving a 20.8% increase in attack success rates
while reducing query costs by 62.7%. GAP consistently outperforms
state-of-the-art methods for attacking both open and closed LLMs, with attack
success rates of >96%. Additionally, we present specialized variants like
GAP-Auto for automated seed generation and GAP-VLM for multimodal attacks.
GAP-generated prompts prove highly effective in improving content moderation
systems, increasing true positive detection rates by 108.5% and accuracy by
183.6% when used for fine-tuning. Our implementation is available at
https://github.com/dsbuddy/GAP-LLM-Safety.


http://arxiv.org/abs/2501.16727
xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM Jailbreaking. (11%)
Sunbowen Lee; Shiwen Ni; Chi Wei; Shuaimin Li; Liyang Fan; Ahmadreza Argha; Hamid Alinejad-Rokny; Ruifeng Xu; Yicheng Gong; Min Yang
  Safety alignment mechanism are essential for preventing large language models
(LLMs) from generating harmful information or unethical content. However,
cleverly crafted prompts can bypass these safety measures without accessing the
model's internal parameters, a phenomenon known as black-box jailbreak.
Existing heuristic black-box attack methods, such as genetic algorithms, suffer
from limited effectiveness due to their inherent randomness, while recent
reinforcement learning (RL) based methods often lack robust and informative
reward signals. To address these challenges, we propose a novel black-box
jailbreak method leveraging RL, which optimizes prompt generation by analyzing
the embedding proximity between benign and malicious prompts. This approach
ensures that the rewritten prompts closely align with the intent of the
original prompts while enhancing the attack's effectiveness. Furthermore, we
introduce a comprehensive jailbreak evaluation framework incorporating
keywords, intent matching, and answer validation to provide a more rigorous and
holistic assessment of jailbreak success. Experimental results show the
superiority of our approach, achieving state-of-the-art (SOTA) performance on
several prominent open and closed-source LLMs, including Qwen2.5-7B-Instruct,
Llama3.1-8B-Instruct, and GPT-4o-0806. Our method sets a new benchmark in
jailbreak attack effectiveness, highlighting potential vulnerabilities in LLMs.
The codebase for this work is available at
https://github.com/Aegis1863/xJailbreak.


http://arxiv.org/abs/2501.16971
RODEO: Robust Outlier Detection via Exposing Adaptive Out-of-Distribution Samples. (9%)
Hossein Mirzaei; Mohammad Jafari; Hamid Reza Dehbashi; Ali Ansari; Sepehr Ghobadi; Masoud Hadi; Arshia Soltani Moakhar; Mohammad Azizmalayeri; Mahdieh Soleymani Baghshah; Mohammad Hossein Rohban
  In recent years, there have been significant improvements in various forms of
image outlier detection. However, outlier detection performance under
adversarial settings lags far behind that in standard settings. This is due to
the lack of effective exposure to adversarial scenarios during training,
especially on unseen outliers, leading to detection models failing to learn
robust features. To bridge this gap, we introduce RODEO, a data-centric
approach that generates effective outliers for robust outlier detection. More
specifically, we show that incorporating outlier exposure (OE) and adversarial
training can be an effective strategy for this purpose, as long as the exposed
training outliers meet certain characteristics, including diversity, and both
conceptual differentiability and analogy to the inlier samples. We leverage a
text-to-image model to achieve this goal. We demonstrate both quantitatively
and qualitatively that our adaptive OE method effectively generates ``diverse''
and ``near-distribution'' outliers, leveraging information from both text and
image domains. Moreover, our experimental results show that utilizing our
synthesized outliers significantly enhances the performance of the outlier
detector, particularly in adversarial settings.


http://arxiv.org/abs/2501.17384
A Dual-Agent Adversarial Framework for Robust Generalization in Deep Reinforcement Learning. (9%)
Zhengpeng Xie; Jiahang Cao; Yulong Zhang; Qiang Zhang; Renjing Xu
  Recently, empowered with the powerful capabilities of neural networks,
reinforcement learning (RL) has successfully tackled numerous challenging
tasks. However, while these models demonstrate enhanced decision-making
abilities, they are increasingly prone to overfitting. For instance, a trained
RL model often fails to generalize to even minor variations of the same task,
such as a change in background color or other minor semantic differences. To
address this issue, we propose a dual-agent adversarial policy learning
framework, which allows agents to spontaneously learn the underlying semantics
without introducing any human prior knowledge. Specifically, our framework
involves a game process between two agents: each agent seeks to maximize the
impact of perturbing on the opponent's policy by producing representation
differences for the same state, while maintaining its own stability against
such perturbations. This interaction encourages agents to learn generalizable
policies, capable of handling irrelevant features from the high-dimensional
observations. Extensive experimental results on the Procgen benchmark
demonstrate that the adversarial process significantly improves the
generalization performance of both agents, while also being applied to various
RL algorithms, e.g., Proximal Policy Optimization (PPO). With the adversarial
framework, the RL agent outperforms the baseline methods by a significant
margin, especially in hard-level tasks, marking a significant step forward in
the generalization capabilities of deep reinforcement learning.


http://arxiv.org/abs/2501.17328
WASUP: Interpretable Classification with Weight-Input Alignment and Class-Discriminative SUPports Vectors. (1%)
Tom Nuno Wolf; Christian Wachinger
  The deployment of deep learning models in critical domains necessitates a
balance between high accuracy and interpretability. We introduce WASUP, an
inherently interpretable neural network that provides local and global
explanations of its decision-making process. We prove that these explanations
are faithful by fulfilling established axioms for explanations. Leveraging the
concept of case-based reasoning, WASUP extracts class-representative support
vectors from training images, ensuring they capture relevant features while
suppressing irrelevant ones. Classification decisions are made by calculating
and aggregating similarity scores between these support vectors and the input's
latent feature vector. We employ B-Cos transformations, which align model
weights with inputs to enable faithful mappings of latent features back to the
input space, facilitating local explanations in addition to global explanations
of case-based reasoning. We evaluate WASUP on three tasks: fine-grained
classification on Stanford Dogs, multi-label classification on Pascal VOC, and
pathology detection on the RSNA dataset. Results indicate that WASUP not only
achieves competitive accuracy compared to state-of-the-art black-box models but
also offers insightful explanations verified through theoretical analysis. Our
findings underscore WASUP's potential for applications where understanding
model decisions is as critical as the decisions themselves.


http://arxiv.org/abs/2501.18629
The Relationship Between Network Similarity and Transferability of Adversarial Attacks. (99%)
Gerrit Klause; Niklas Bunzel
  Neural networks are vulnerable to adversarial attacks, and several defenses
have been proposed. Designing a robust network is a challenging task given the
wide range of attacks that have been developed. Therefore, we aim to provide
insight into the influence of network similarity on the success rate of
transferred adversarial attacks. Network designers can then compare their new
network with existing ones to estimate its vulnerability. To achieve this, we
investigate the complex relationship between network similarity and the success
rate of transferred adversarial attacks. We applied the Centered Kernel
Alignment (CKA) network similarity score and used various methods to find a
correlation between a large number of Convolutional Neural Networks (CNNs) and
adversarial attacks. Network similarity was found to be moderate across
different CNN architectures, with more complex models such as DenseNet showing
lower similarity scores due to their architectural complexity. Layer similarity
was highest for consistent, basic layers such as DataParallel, Dropout and
Conv2d, while specialized layers showed greater variability. Adversarial attack
success rates were generally consistent for non-transferred attacks, but varied
significantly for some transferred attacks, with complex networks being more
vulnerable. We found that a DecisionTreeRegressor can predict the success rate
of transferred attacks for all black-box and Carlini & Wagner attacks with an
accuracy of over 90%, suggesting that predictive models may be viable under
certain conditions. However, the variability of results across different data
subsets underscores the complexity of these relationships and suggests that
further research is needed to generalize these findings across different attack
scenarios and network architectures.


http://arxiv.org/abs/2501.16671
Data-Free Model-Related Attacks: Unleashing the Potential of Generative AI. (31%)
Dayong Ye; Tianqing Zhu; Shang Wang; Bo Liu; Leo Yu Zhang; Wanlei Zhou; Yang Zhang
  Generative AI technology has become increasingly integrated into our daily
lives, offering powerful capabilities to enhance productivity. However, these
same capabilities can be exploited by adversaries for malicious purposes. While
existing research on adversarial applications of generative AI predominantly
focuses on cyberattacks, less attention has been given to attacks targeting
deep learning models. In this paper, we introduce the use of generative AI for
facilitating model-related attacks, including model extraction, membership
inference, and model inversion. Our study reveals that adversaries can launch a
variety of model-related attacks against both image and text models in a
data-free and black-box manner, achieving comparable performance to baseline
methods that have access to the target models' training data and parameters in
a white-box manner. This research serves as an important early warning to the
community about the potential risks associated with generative AI-powered
attacks on deep learning models.


http://arxiv.org/abs/2501.18632
Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare. (12%)
Hang Zhang; Qian Lou; Yanshan Wang
  Large language models (LLMs) are increasingly utilized in healthcare
applications. However, their deployment in clinical practice raises significant
safety concerns, including the potential spread of harmful information. This
study systematically assesses the vulnerabilities of six LLMs to three advanced
black-box jailbreaking techniques within medical contexts. To quantify the
effectiveness of these techniques, we propose an automated and domain-adapted
agentic evaluation pipeline. Experiment results indicate that leading
commercial and open-source LLMs are highly vulnerable to medical jailbreaking
attacks. To bolster model safety and reliability, we further investigate the
effectiveness of Continual Fine-Tuning (CFT) in defending against medical
adversarial attacks. Our findings underscore the necessity for evolving attack
methods evaluation, domain-specific safety alignment, and LLM safety-utility
balancing. This research offers actionable insights for advancing the safety
and reliability of AI clinicians, contributing to ethical and effective AI
deployment in healthcare.


http://arxiv.org/abs/2501.18628
Indiana Jones: There Are Always Some Useful Ancient Relics. (3%)
Junchen Ding; Jiahao Zhang; Yi Liu; Ziqi Ding; Gelei Deng; Yuekang Li
  This paper introduces Indiana Jones, an innovative approach to jailbreaking
Large Language Models (LLMs) by leveraging inter-model dialogues and
keyword-driven prompts. Through orchestrating interactions among three
specialised LLMs, the method achieves near-perfect success rates in bypassing
content safeguards in both white-box and black-box LLMs. The research exposes
systemic vulnerabilities within contemporary models, particularly their
susceptibility to producing harmful or unethical outputs when guided by
ostensibly innocuous prompts framed in historical or contextual contexts.
Experimental evaluations highlight the efficacy and adaptability of Indiana
Jones, demonstrating its superiority over existing jailbreak methods. These
findings emphasise the urgent need for enhanced ethical safeguards and robust
security measures in the development of LLMs. Moreover, this work provides a
critical foundation for future studies aimed at fortifying LLMs against
adversarial exploitation while preserving their utility and flexibility.


http://arxiv.org/abs/2501.15850
LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Models. (2%)
Yuewen Mei; Tong Nie; Jian Sun; Ye Tian
  Ensuring and improving the safety of autonomous driving systems (ADS) is
crucial for the deployment of highly automated vehicles, especially in
safety-critical events. To address the rarity issue, adversarial scenario
generation methods are developed, in which behaviors of traffic participants
are manipulated to induce safety-critical events. However, existing methods
still face two limitations. First, identification of the adversarial
participant directly impacts the effectiveness of the generation. However, the
complexity of real-world scenarios, with numerous participants and diverse
behaviors, makes identification challenging. Second, the potential of generated
safety-critical scenarios to continuously improve ADS performance remains
underexplored. To address these issues, we propose LLM-attacker: a closed-loop
adversarial scenario generation framework leveraging large language models
(LLMs). Specifically, multiple LLM agents are designed and coordinated to
identify optimal attackers. Then, the trajectories of the attackers are
optimized to generate adversarial scenarios. These scenarios are iteratively
refined based on the performance of ADS, forming a feedback loop to improve
ADS. Experimental results show that LLM-attacker can create more dangerous
scenarios than other methods, and the ADS trained with it achieves a collision
rate half that of training with normal scenarios. This indicates the ability of
LLM-attacker to test and enhance the safety and robustness of ADS. Video
demonstrations are provided at:
https://drive.google.com/file/d/1Zv4V3iG7825oyiKbUwS2Y-rR0DQIE1ZA/view.


http://arxiv.org/abs/2501.16663
Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearning. (1%)
Dayong Ye; Tianqing Zhu; Jiayang Li; Kun Gao; Bo Liu; Leo Yu Zhang; Wanlei Zhou; Yang Zhang
  Duplication is a prevalent issue within datasets. Existing research has
demonstrated that the presence of duplicated data in training datasets can
significantly influence both model performance and data privacy. However, the
impact of data duplication on the unlearning process remains largely
unexplored. This paper addresses this gap by pioneering a comprehensive
investigation into the role of data duplication, not only in standard machine
unlearning but also in federated and reinforcement unlearning paradigms.
Specifically, we propose an adversary who duplicates a subset of the target
model's training set and incorporates it into the training set. After training,
the adversary requests the model owner to unlearn this duplicated subset, and
analyzes the impact on the unlearned model. For example, the adversary can
challenge the model owner by revealing that, despite efforts to unlearn it, the
influence of the duplicated subset remains in the model. Moreover, to
circumvent detection by de-duplication techniques, we propose three novel
near-duplication methods for the adversary, each tailored to a specific
unlearning paradigm. We then examine their impacts on the unlearning process
when de-duplication techniques are applied. Our findings reveal several crucial
insights: 1) the gold standard unlearning method, retraining from scratch,
fails to effectively conduct unlearning under certain conditions; 2) unlearning
duplicated data can lead to significant model degradation in specific
scenarios; and 3) meticulously crafted duplicates can evade detection by
de-duplication methods.


http://arxiv.org/abs/2501.15434
Mitigating Spurious Negative Pairs for Robust Industrial Anomaly Detection. (98%)
Hossein Mirzaei; Mojtaba Nafez; Jafar Habibi; Mohammad Sabokrou; Mohammad Hossein Rohban
  Despite significant progress in Anomaly Detection (AD), the robustness of
existing detection methods against adversarial attacks remains a challenge,
compromising their reliability in critical real-world applications such as
autonomous driving. This issue primarily arises from the AD setup, which
assumes that training data is limited to a group of unlabeled normal samples,
making the detectors vulnerable to adversarial anomaly samples during testing.
Additionally, implementing adversarial training as a safeguard encounters
difficulties, such as formulating an effective objective function without
access to labels. An ideal objective function for adversarial training in AD
should promote strong perturbations both within and between the normal and
anomaly groups to maximize margin between normal and anomaly distribution. To
address these issues, we first propose crafting a pseudo-anomaly group derived
from normal group samples. Then, we demonstrate that adversarial training with
contrastive loss could serve as an ideal objective function, as it creates both
inter- and intra-group perturbations. However, we notice that spurious negative
pairs compromise the conventional contrastive loss to achieve robust AD.
Spurious negative pairs are those that should be closely mapped but are
erroneously separated. These pairs introduce noise and misguide the direction
of inter-group adversarial perturbations. To overcome the effect of spurious
negative pairs, we define opposite pairs and adversarially pull them apart to
strengthen inter-group perturbations. Experimental results demonstrate our
superior performance in both clean and adversarial scenarios, with a 26.1%
improvement in robust detection across various challenging benchmark datasets.
The implementation of our work is available at:
https://github.com/rohban-lab/COBRA.


http://arxiv.org/abs/2501.15563
PCAP-Backdoor: Backdoor Poisoning Generator for Network Traffic in CPS/IoT Environments. (76%)
Ajesh Koyatan Chathoth; Stephen Lee
  The rapid expansion of connected devices has made them prime targets for
cyberattacks. To address these threats, deep learning-based, data-driven
intrusion detection systems (IDS) have emerged as powerful tools for detecting
and mitigating such attacks. These IDSs analyze network traffic to identify
unusual patterns and anomalies that may indicate potential security breaches.
However, prior research has shown that deep learning models are vulnerable to
backdoor attacks, where attackers inject triggers into the model to manipulate
its behavior and cause misclassifications of network traffic. In this paper, we
explore the susceptibility of deep learning-based IDS systems to backdoor
attacks in the context of network traffic analysis. We introduce
\texttt{PCAP-Backdoor}, a novel technique that facilitates backdoor poisoning
attacks on PCAP datasets. Our experiments on real-world Cyber-Physical Systems
(CPS) and Internet of Things (IoT) network traffic datasets demonstrate that
attackers can effectively backdoor a model by poisoning as little as 1\% or
less of the entire training dataset. Moreover, we show that an attacker can
introduce a trigger into benign traffic during model training yet cause the
backdoored model to misclassify malicious traffic when the trigger is present.
Finally, we highlight the difficulty of detecting this trigger-based backdoor,
even when using existing backdoor defense techniques.


http://arxiv.org/abs/2501.15718
CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling. (75%)
Kaiyuan Zhang; Siyuan Cheng; Guangyu Shen; Bruno Ribeiro; Shengwei An; Pin-Yu Chen; Xiangyu Zhang; Ninghui Li
  Federated learning collaboratively trains a neural network on a global
server, where each local client receives the current global model weights and
sends back parameter updates (gradients) based on its local private data. The
process of sending these model updates may leak client's private data
information. Existing gradient inversion attacks can exploit this vulnerability
to recover private training instances from a client's gradient vectors.
Recently, researchers have proposed advanced gradient inversion techniques that
existing defenses struggle to handle effectively. In this work, we present a
novel defense tailored for large neural network models. Our defense capitalizes
on the high dimensionality of the model parameters to perturb gradients within
a subspace orthogonal to the original gradient. By leveraging cold posteriors
over orthogonal subspaces, our defense implements a refined gradient update
mechanism. This enables the selection of an optimal gradient that not only
safeguards against gradient inversion attacks but also maintains model utility.
We conduct comprehensive experiments across three different datasets and
evaluate our defense against various state-of-the-art attacks and defenses.
Code is available at https://censor-gradient.github.io.


http://arxiv.org/abs/2501.15653
A Privacy Enhancing Technique to Evade Detection by Street Video Cameras Without Using Adversarial Accessories. (11%)
Jacob Shams; Ben Nassi; Satoru Koda; Asaf Shabtai; Yuval Elovici
  In this paper, we propose a privacy-enhancing technique leveraging an
inherent property of automatic pedestrian detection algorithms, namely, that
the training of deep neural network (DNN) based methods is generally performed
using curated datasets and laboratory settings, while the operational areas of
these methods are dynamic real-world environments. In particular, we leverage a
novel side effect of this gap between the laboratory and the real world:
location-based weakness in pedestrian detection. We demonstrate that the
position (distance, angle, height) of a person, and ambient light level,
directly impact the confidence of a pedestrian detector when detecting the
person. We then demonstrate that this phenomenon is present in pedestrian
detectors observing a stationary scene of pedestrian traffic, with blind spot
areas of weak detection of pedestrians with low confidence. We show how
privacy-concerned pedestrians can leverage these blind spots to evade detection
by constructing a minimum confidence path between two points in a scene,
reducing the maximum confidence and average confidence of the path by up to
0.09 and 0.13, respectively, over direct and random paths through the scene. To
counter this phenomenon, and force the use of more costly and sophisticated
methods to leverage this vulnerability, we propose a novel countermeasure to
improve the confidence of pedestrian detectors in blind spots, raising the
max/average confidence of paths generated by our technique by 0.09 and 0.05,
respectively. In addition, we demonstrate that our countermeasure improves a
Faster R-CNN-based pedestrian detector's TPR and average true positive
confidence by 0.03 and 0.15, respectively.


http://arxiv.org/abs/2501.15257
Towards Communication-Efficient Adversarial Federated Learning for Robust Edge Intelligence. (98%)
Yu Qiao; Apurba Adhikary; Huy Q. Le; Eui-Nam Huh; Zhu Han; Choong Seon Hong
  Federated learning (FL) has gained significant attention for enabling
decentralized training on edge networks without exposing raw data. However, FL
models remain susceptible to adversarial attacks and performance degradation in
non-IID data settings, thus posing challenges to both robustness and accuracy.
This paper aims to achieve communication-efficient adversarial federated
learning (AFL) by leveraging a pre-trained model to enhance both robustness and
accuracy under adversarial attacks and non-IID challenges in AFL. By leveraging
the knowledge from a pre-trained model for both clean and adversarial images,
we propose a pre-trained model-guided adversarial federated learning (PM-AFL)
framework. This framework integrates vanilla and adversarial mixture knowledge
distillation to effectively balance accuracy and robustness while promoting
local models to learn from diverse data. Specifically, for clean accuracy, we
adopt a dual distillation strategy where the class probabilities of randomly
paired images, and their blended versions are aligned between the teacher model
and the local models. For adversarial robustness, we employ a similar
distillation approach but replace clean samples on the local side with
adversarial examples. Moreover, by considering the bias between local and
global models, we also incorporate a consistency regularization term to ensure
that local adversarial predictions stay aligned with their corresponding global
clean ones. These strategies collectively enable local models to absorb diverse
knowledge from the teacher model while maintaining close alignment with the
global model, thereby mitigating overfitting to local optima and enhancing the
generalization of the global model. Experiments demonstrate that the
PM-AFL-based framework not only significantly outperforms other methods but
also maintains communication efficiency.


http://arxiv.org/abs/2501.15271
Killing it with Zero-Shot: Adversarially Robust Novelty Detection. (50%)
Hossein Mirzaei; Mohammad Jafari; Hamid Reza Dehbashi; Zeinab Sadat Taghavi; Mohammad Sabokrou; Mohammad Hossein Rohban
  Novelty Detection (ND) plays a crucial role in machine learning by
identifying new or unseen data during model inference. This capability is
especially important for the safe and reliable operation of automated systems.
Despite advances in this field, existing techniques often fail to maintain
their performance when subject to adversarial attacks. Our research addresses
this gap by marrying the merits of nearest-neighbor algorithms with robust
features obtained from models pretrained on ImageNet. We focus on enhancing the
robustness and performance of ND algorithms. Experimental results demonstrate
that our approach significantly outperforms current state-of-the-art methods
across various benchmarks, particularly under adversarial conditions. By
incorporating robust pretrained features into the k-NN algorithm, we establish
a new standard for performance and robustness in the field of robust ND. This
work opens up new avenues for research aimed at fortifying machine learning
systems against adversarial vulnerabilities. Our implementation is publicly
available at https://github.com/rohban-lab/ZARND.


http://arxiv.org/abs/2501.15395
Hiding in Plain Sight: An IoT Traffic Camouflage Framework for Enhanced Privacy. (12%)
Daniel Adu Worae; Spyridon Mastorakis
  The rapid growth of Internet of Things (IoT) devices has introduced
significant challenges to privacy, particularly as network traffic analysis
techniques evolve. While encryption protects data content, traffic attributes
such as packet size and timing can reveal sensitive information about users and
devices. Existing single-technique obfuscation methods, such as packet padding,
often fall short in dynamic environments like smart homes due to their
predictability, making them vulnerable to machine learning-based attacks. This
paper introduces a multi-technique obfuscation framework designed to enhance
privacy by disrupting traffic analysis. The framework leverages six
techniques-Padding, Padding with XORing, Padding with Shifting, Constant Size
Padding, Fragmentation, and Delay Randomization-to obscure traffic patterns
effectively. Evaluations on three public datasets demonstrate significant
reductions in classifier performance metrics, including accuracy, precision,
recall, and F1 score. We assess the framework's robustness against adversarial
tactics by retraining and fine-tuning neural network classifiers on obfuscated
traffic. The results reveal a notable degradation in classifier performance,
underscoring the framework's resilience against adaptive attacks. Furthermore,
we evaluate communication and system performance, showing that higher
obfuscation levels enhance privacy but may increase latency and communication
overhead.


http://arxiv.org/abs/2501.15101
Comprehensive Evaluation of Cloaking Backdoor Attacks on Object Detector in Real-World. (11%)
Hua Ma; Alsharif Abuadbba; Yansong Gao; Hyoungshick Kim; Surya Nepal
  The exploration of backdoor vulnerabilities in object detectors, particularly
in real-world scenarios, remains limited. A significant challenge lies in the
absence of a natural physical backdoor dataset, and constructing such a dataset
is both time- and labor-intensive. In this work, we address this gap by
creating a large-scale dataset comprising approximately 11,800 images/frames
with annotations featuring natural objects (e.g., T-shirts and hats) as
triggers to incur cloaking adversarial effects in diverse real-world scenarios.
This dataset is tailored for the study of physical backdoors in object
detectors. Leveraging this dataset, we conduct a comprehensive evaluation of an
insidious cloaking backdoor effect against object detectors, wherein the
bounding box around a person vanishes when the individual is near a natural
object (e.g., a commonly available T-shirt) in front of the detector. Our
evaluations encompass three prevalent attack surfaces: data outsourcing, model
outsourcing, and the use of pretrained models. The cloaking effect is
successfully implanted in object detectors across all three attack surfaces. We
extensively evaluate four popular object detection algorithms (anchor-based
Yolo-V3, Yolo-V4, Faster R-CNN, and anchor-free CenterNet) using 19 videos
(totaling approximately 11,800 frames) in real-world scenarios. Our results
demonstrate that the backdoor attack exhibits remarkable robustness against
various factors, including movement, distance, angle, non-rigid deformation,
and lighting. In data and model outsourcing scenarios, the attack success rate
(ASR) in most videos reaches 100% or near it, while the clean data accuracy of
the backdoored model remains indistinguishable from that of the clean model,
making it impossible to detect backdoor behavior through a validation set.


http://arxiv.org/abs/2501.15269
Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink. (10%)
Yining Wang; Mi Zhang; Junjie Sun; Chenyue Wang; Min Yang; Hui Xue; Jialing Tao; Ranjie Duan; Jiexi Liu
  Fusing visual understanding into language generation, Multi-modal Large
Language Models (MLLMs) are revolutionizing visual-language applications. Yet,
these models are often plagued by the hallucination problem, which involves
generating inaccurate objects, attributes, and relationships that do not match
the visual content. In this work, we delve into the internal attention
mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing
the inherent vulnerabilities in the instruction-tuning process.
  We propose a novel hallucination attack against MLLMs that exploits attention
sink behaviors to trigger hallucinated content with minimal image-text
relevance, posing a significant threat to critical downstream applications.
Distinguished from previous adversarial methods that rely on fixed patterns,
our approach generates dynamic, effective, and highly transferable visual
adversarial inputs, without sacrificing the quality of model responses.
Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our
attack in compromising black-box MLLMs even with extensive mitigating
mechanisms, as well as the promising results against cutting-edge commercial
APIs, such as GPT-4o and Gemini 1.5. Our code is available at
https://huggingface.co/RachelHGF/Mirage-in-the-Eyes.


http://arxiv.org/abs/2501.15363
AI-Driven Secure Data Sharing: A Trustworthy and Privacy-Preserving Approach. (4%)
Al Amin; Kamrul Hasan; Sharif Ullah; Liang Hong
  In the era of data-driven decision-making, ensuring the privacy and security
of shared data is paramount across various domains. Applying existing deep
neural networks (DNNs) to encrypted data is critical and often compromises
performance, security, and computational overhead. To address these
limitations, this research introduces a secure framework consisting of a
learnable encryption method based on the block-pixel operation to encrypt the
data and subsequently integrate it with the Vision Transformer (ViT). The
proposed framework ensures data privacy and security by creating unique
scrambling patterns per key, providing robust performance against adversarial
attacks without compromising computational efficiency and data integrity. The
framework was tested on sensitive medical datasets to validate its efficacy,
proving its ability to handle highly confidential information securely. The
suggested framework was validated with a 94\% success rate after extensive
testing on real-world datasets, such as MRI brain tumors and histological scans
of lung and colon cancers. Additionally, the framework was tested under diverse
adversarial attempts against secure data sharing with optimum performance and
demonstrated its effectiveness in various threat scenarios. These comprehensive
analyses underscore its robustness, making it a trustworthy solution for secure
data sharing in critical applications.


http://arxiv.org/abs/2501.14999
VideoPure: Diffusion-based Adversarial Purification for Video Recognition. (99%)
Kaixun Jiang; Zhaoyu Chen; Jiyuan Fu; Lingyi Hong; Jinglun Li; Wenqiang Zhang
  Recent work indicates that video recognition models are vulnerable to
adversarial examples, posing a serious security risk to downstream
applications. However, current research has primarily focused on adversarial
attacks, with limited work exploring defense mechanisms. Furthermore, due to
the spatial-temporal complexity of videos, existing video defense methods face
issues of high cost, overfitting, and limited defense performance. Recently,
diffusion-based adversarial purification methods have achieved robust defense
performance in the image domain. However, due to the additional temporal
dimension in videos, directly applying these diffusion-based adversarial
purification methods to the video domain suffers performance and efficiency
degradation. To achieve an efficient and effective video adversarial defense
method, we propose the first diffusion-based video purification framework to
improve video recognition models' adversarial robustness: VideoPure. Given an
adversarial example, we first employ temporal DDIM inversion to transform the
input distribution into a temporally consistent and trajectory-defined
distribution, covering adversarial noise while preserving more video structure.
Then, during DDIM denoising, we leverage intermediate results at each denoising
step and conduct guided spatial-temporal optimization, removing adversarial
noise while maintaining temporal consistency. Finally, we input the list of
optimized intermediate results into the video recognition model for multi-step
voting to obtain the predicted class. We investigate the defense performance of
our method against black-box, gray-box, and adaptive attacks on benchmark
datasets and models. Compared with other adversarial purification methods, our
method overall demonstrates better defense performance against different
attacks. Our code is available at https://github.com/deep-kaixun/VideoPure.


http://arxiv.org/abs/2501.14496
A Note on Implementation Errors in Recent Adaptive Attacks Against Multi-Resolution Self-Ensembles. (62%)
Stanislav Fort
  This note documents an implementation issue in recent adaptive attacks (Zhang
et al. [2024]) against the multi-resolution self-ensemble defense (Fort and
Lakshminarayanan [2024]). The implementation allowed adversarial perturbations
to exceed the standard $L_\infty = 8/255$ bound by up to a factor of
20$\times$, reaching magnitudes of up to $L_\infty = 160/255$. When attacks are
properly constrained within the intended bounds, the defense maintains
non-trivial robustness. Beyond highlighting the importance of careful
validation in adversarial machine learning research, our analysis reveals an
intriguing finding: properly bounded adaptive attacks against strong
multi-resolution self-ensembles often align with human perception, suggesting
the need to reconsider how we measure adversarial robustness.


http://arxiv.org/abs/2501.14250
Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors. (16%)
Yi Zhao; Youzhi Zhang
  Large language models (LLMs) are widely used in real-world applications,
raising concerns about their safety and trustworthiness. While red-teaming with
jailbreak prompts exposes the vulnerabilities of LLMs, current efforts focus
primarily on single-turn attacks, overlooking the multi-turn strategies used by
real-world adversaries. Existing multi-turn methods rely on static patterns or
predefined logical chains, failing to account for the dynamic strategies during
attacks. We propose Siren, a learning-based multi-turn attack framework
designed to simulate real-world human jailbreak behaviors. Siren consists of
three stages: (1) training set construction utilizing Turn-Level LLM feedback
(Turn-MF), (2) post-training attackers with supervised fine-tuning (SFT) and
direct preference optimization (DPO), and (3) interactions between the
attacking and target LLMs. Experiments demonstrate that Siren achieves an
attack success rate (ASR) of 90% with LLaMA-3-8B as the attacker against
Gemini-1.5-Pro as the target model, and 70% with Mistral-7B against GPT-4o,
significantly outperforming single-turn baselines. Moreover, Siren with a
7B-scale model achieves performance comparable to a multi-turn baseline that
leverages GPT-4o as the attacker, while requiring fewer turns and employing
decomposition strategies that are better semantically aligned with attack
goals. We hope Siren inspires the development of stronger defenses against
advanced multi-turn jailbreak attacks under realistic scenarios. Code is
available at https://github.com/YiyiyiZhao/siren. Warning: This paper contains
potentially harmful text.


http://arxiv.org/abs/2501.14005
Device-aware Optical Adversarial Attack for a Portable Projector-camera System. (99%)
Ning School of Software & Microelectronics, Peking University, Beijing, China Mashang Consumer Finance Co., Ltd., Chongqing, China Jiang; Yanhong Mashang Consumer Finance Co., Ltd., Chongqing, China Liu; Dingheng Mashang Consumer Finance Co., Ltd., Chongqing, China Zeng; Yue Mashang Consumer Finance Co., Ltd., Chongqing, China Feng; Weihong Mashang Consumer Finance Co., Ltd., Chongqing, China Deng; Ying School of Software & Microelectronics, Peking University, Beijing, China Li
  Deep-learning-based face recognition (FR) systems are susceptible to
adversarial examples in both digital and physical domains. Physical attacks
present a greater threat to deployed systems as adversaries can easily access
the input channel, allowing them to provide malicious inputs to impersonate a
victim. This paper addresses the limitations of existing projector-camera-based
adversarial light attacks in practical FR setups. By incorporating device-aware
adaptations into the digital attack algorithm, such as resolution-aware and
color-aware adjustments, we mitigate the degradation from digital to physical
domains. Experimental validation showcases the efficacy of our proposed
algorithm against real and spoof adversaries, achieving high physical
similarity scores in FR models and state-of-the-art commercial systems. On
average, there is only a 14% reduction in scores from digital to physical
attacks, with high attack success rate in both white- and black-box scenarios.


http://arxiv.org/abs/2501.14230
GreedyPixel: Fine-Grained Black-Box Adversarial Attack Via Greedy Algorithm. (99%)
Hanrui Wang; Ching-Chun Chang; Chun-Shien Lu; Christopher Leckie; Isao Echizen
  A critical requirement for deep learning models is ensuring their robustness
against adversarial attacks. These attacks commonly introduce noticeable
perturbations, compromising the visual fidelity of adversarial examples.
Another key challenge is that while white-box algorithms can generate effective
adversarial perturbations, they require access to the model gradients, limiting
their practicality in many real-world scenarios. Existing attack mechanisms
struggle to achieve similar efficacy without access to these gradients. In this
paper, we introduce GreedyPixel, a novel pixel-wise greedy algorithm designed
to generate high-quality adversarial examples using only query-based feedback
from the target model. GreedyPixel improves computational efficiency in what is
typically a brute-force process by perturbing individual pixels in sequence,
guided by a pixel-wise priority map. This priority map is constructed by
ranking gradients obtained from a surrogate model, providing a structured path
for perturbation. Our results demonstrate that GreedyPixel achieves attack
success rates comparable to white-box methods without the need for gradient
information, and surpasses existing algorithms in black-box settings, offering
higher success rates, reduced computational time, and imperceptible
perturbations. These findings underscore the advantages of GreedyPixel in terms
of attack efficacy, time efficiency, and visual quality.


http://arxiv.org/abs/2501.14122
Reinforcement Learning Platform for Adversarial Black-box Attacks with Custom Distortion Filters. (99%)
Soumyendu Sarkar; Ashwin Ramesh Babu; Sajad Mousavi; Vineet Gundecha; Sahand Ghorbanpour; Avisek Naug; Ricardo Luna Gutierrez; Antonio Guillen
  We present a Reinforcement Learning Platform for Adversarial Black-box
untargeted and targeted attacks, RLAB, that allows users to select from various
distortion filters to create adversarial examples. The platform uses a
Reinforcement Learning agent to add minimum distortion to input images while
still causing misclassification by the target model. The agent uses a novel
dual-action method to explore the input image at each step to identify
sensitive regions for adding distortions while removing noises that have less
impact on the target model. This dual action leads to faster and more efficient
convergence of the attack. The platform can also be used to measure the
robustness of image classification models against specific distortion types.
Also, retraining the model with adversarial samples significantly improved
robustness when evaluated on benchmark datasets. The proposed platform
outperforms state-of-the-art methods in terms of the average number of queries
required to cause misclassification. This advances trustworthiness with a
positive social impact.


http://arxiv.org/abs/2501.13563
Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving. (99%)
Lu Wang; Tianyuan Zhang; Yang Qu; Siyuan Liang; Yuwei Chen; Aishan Liu; Xianglong Liu; Dacheng Tao
  Vision-language models (VLMs) have significantly advanced autonomous driving
(AD) by enhancing reasoning capabilities; however, these models remain highly
susceptible to adversarial attacks. While existing research has explored
white-box attacks to some extent, the more practical and challenging black-box
scenarios remain largely underexplored due to their inherent difficulty. In
this paper, we take the first step toward designing black-box adversarial
attacks specifically targeting VLMs in AD. We identify two key challenges for
achieving effective black-box attacks in this context: the effectiveness across
driving reasoning chains in AD systems and the dynamic nature of driving
scenarios. To address this, we propose Cascading Adversarial Disruption (CAD).
It first introduces Decision Chain Disruption, which targets low-level
reasoning breakdown by generating and injecting deceptive semantics, ensuring
the perturbations remain effective across the entire decision-making chain.
Building on this, we present Risky Scene Induction, which addresses dynamic
adaptation by leveraging a surrogate VLM to understand and construct high-level
risky scenarios that are likely to result in critical errors in the current
driving contexts. Extensive experiments conducted on multiple AD VLMs and
benchmarks demonstrate that CAD achieves state-of-the-art attack effectiveness,
significantly outperforming existing methods (+13.43% on average). Moreover, we
validate its practical applicability through real-world attacks on AD vehicles
powered by VLMs, where the route completion rate drops by 61.11% and the
vehicle crashes directly into the obstacle vehicle with adversarial patches.
Finally, we release CADA dataset, comprising 18,808 adversarial
visual-question-answer pairs, to facilitate further evaluation and research in
this critical domain. Our codes and dataset will be available after paper's
acceptance.


http://arxiv.org/abs/2501.13782
Defending against Adversarial Malware Attacks on ML-based Android Malware Detection Systems. (98%)
Ping He; Lorenzo Cavallaro; Shouling Ji
  Android malware presents a persistent threat to users' privacy and data
integrity. To combat this, researchers have proposed machine learning-based
(ML-based) Android malware detection (AMD) systems. However, adversarial
Android malware attacks compromise the detection integrity of the ML-based AMD
systems, raising significant concerns. Existing defenses against adversarial
Android malware provide protections against feature space attacks which
generate adversarial feature vectors only, leaving protection against realistic
threats from problem space attacks which generate real adversarial malware an
open problem. In this paper, we address this gap by proposing ADD, a practical
adversarial Android malware defense framework designed as a plug-in to enhance
the adversarial robustness of the ML-based AMD systems against problem space
attacks. Our extensive evaluation across various ML-based AMD systems
demonstrates that ADD is effective against state-of-the-art problem space
adversarial Android malware attacks. Additionally, ADD shows the defense
effectiveness in enhancing the adversarial robustness of real-world antivirus
solutions.


http://arxiv.org/abs/2501.13776
Crossfire: An Elastic Defense Framework for Graph Neural Networks Under Bit Flip Attacks. (83%)
Lorenz Kummer; Samir Moustafa; Wilfried Gansterer; Nils Kriege
  Bit Flip Attacks (BFAs) are a well-established class of adversarial attacks,
originally developed for Convolutional Neural Networks within the computer
vision domain. Most recently, these attacks have been extended to target Graph
Neural Networks (GNNs), revealing significant vulnerabilities. This new
development naturally raises questions about the best strategies to defend GNNs
against BFAs, a challenge for which no solutions currently exist. Given the
applications of GNNs in critical fields, any defense mechanism must not only
maintain network performance, but also verifiably restore the network to its
pre-attack state. Verifiably restoring the network to its pre-attack state also
eliminates the need for costly evaluations on test data to ensure network
quality. We offer first insights into the effectiveness of existing honeypot-
and hashing-based defenses against BFAs adapted from the computer vision domain
to GNNs, and characterize the shortcomings of these approaches. To overcome
their limitations, we propose Crossfire, a hybrid approach that exploits weight
sparsity and combines hashing and honeypots with bit-level correction of
out-of-distribution weight elements to restore network integrity. Crossfire is
retraining-free and does not require labeled data. Averaged over 2,160
experiments on six benchmark datasets, Crossfire offers a 21.8% higher
probability than its competitors of reconstructing a GNN attacked by a BFA to
its pre-attack state. These experiments cover up to 55 bit flips from various
attacks. Moreover, it improves post-repair prediction quality by 10.85%.
Computational and storage overheads are negligible compared to the inherent
complexity of even the simplest GNNs.


http://arxiv.org/abs/2501.13676
Certified Robustness Under Bounded Levenshtein Distance. (67%)
Elias Abad Rocamora; Grigorios G. Chrysos; Volkan Cevher
  Text classifiers suffer from small perturbations, that if chosen
adversarially, can dramatically change the output of the model. Verification
methods can provide robustness certificates against such adversarial
perturbations, by computing a sound lower bound on the robust accuracy.
Nevertheless, existing verification methods incur in prohibitive costs and
cannot practically handle Levenshtein distance constraints. We propose the
first method for computing the Lipschitz constant of convolutional classifiers
with respect to the Levenshtein distance. We use these Lipschitz constant
estimates for training 1-Lipschitz classifiers. This enables computing the
certified radius of a classifier in a single forward pass. Our method, LipsLev,
is able to obtain $38.80$% and $13.93$% verified accuracy at distance $1$ and
$2$ respectively in the AG-News dataset, while being $4$ orders of magnitude
faster than existing approaches. We believe our work can open the door to more
efficient verification in the text domain.


http://arxiv.org/abs/2501.14050
GraphRAG under Fire. (13%)
Jiacheng Liang; Yuhui Wang; Changjiang Li; Rongyi Zhu; Tanqiu Jiang; Neil Gong; Ting Wang
  GraphRAG advances retrieval-augmented generation (RAG) by structuring
external knowledge as multi-scale knowledge graphs, enabling language models to
integrate both broad context and granular details in their generation. While
GraphRAG has demonstrated success across domains, its security implications
remain largely unexplored. To bridge this gap, this work examines GraphRAG's
vulnerability to poisoning attacks, uncovering an intriguing security paradox:
existing RAG poisoning attacks are less effective under GraphRAG than
conventional RAG, due to GraphRAG's graph-based indexing and retrieval; yet,
the same features also create new attack surfaces. We present GragPoison, a
novel attack that exploits shared relations in the underlying knowledge graph
to craft poisoning text capable of compromising multiple queries
simultaneously. GragPoison employs three key strategies: (i) relation injection
to introduce false knowledge, (ii) relation enhancement to amplify poisoning
influence, and (iii) narrative generation to embed malicious content within
coherent text. Empirical evaluation across diverse datasets and models shows
that GragPoison substantially outperforms existing attacks in terms of
effectiveness (up to 98% success rate) and scalability (using less than 68%
poisoning text) on multiple variations of GraphRAG. We also explore potential
defensive measures and their limitations, identifying promising directions for
future research.


http://arxiv.org/abs/2501.13894
Logical Maneuvers: Detecting and Mitigating Adversarial Hardware Faults in Space. (1%)
Fatemeh Khojasteh Dana; Saleh Khalaj Monfared; Shahin Tajik
  Satellites are highly vulnerable to adversarial glitches or high-energy
radiation in space, which could cause faults on the onboard computer. Various
radiation- and fault-tolerant methods, such as error correction codes (ECC) and
redundancy-based approaches, have been explored over the last decades to
mitigate temporary soft errors on software and hardware. However, conventional
ECC methods fail to deal with hard errors or permanent faults in the hardware
components. This work introduces a detection- and response-based countermeasure
to deal with partially damaged processor chips. It recovers the processor chip
from permanent faults and enables continuous operation with available undamaged
resources on the chip. We incorporate digitally-compatible delay-based sensors
on the target processor's chip to reliably detect the incoming radiation or
glitching attempts on the physical fabric of the chip, even before a fault
occurs. Upon detecting a fault in one or more components of the processor's
arithmetic logic unit (ALU), our countermeasure employs adaptive software
recompilations to resynthesize and substitute the affected instructions with
instructions of still functioning components to accomplish the task.
Furthermore, if the fault is more widespread and prevents the correct operation
of the entire processor, our approach deploys adaptive hardware partial
reconfigurations to replace and reroute the failed components to undamaged
locations of the chip. To validate our claims, we deploy a high-energy
near-infrared (NIR) laser beam on a RISC-V processor implemented on a 28~nm
FPGA to emulate radiation and even hard errors by partially damaging the FPGA
fabric. We demonstrate that our sensor can confidently detect the radiation and
trigger the processor testing and fault recovery mechanisms. Finally, we
discuss the overhead imposed by our countermeasure.


http://arxiv.org/abs/2501.12761
Modality Unified Attack for Omni-Modality Person Re-Identification. (98%)
Yuan Bian; Min Liu; Yunqi Yi; Xueping Wang; Yunfeng Ma; Yaonan Wang
  Deep learning based person re-identification (re-id) models have been widely
employed in surveillance systems. Recent studies have demonstrated that
black-box single-modality and cross-modality re-id models are vulnerable to
adversarial examples (AEs), leaving the robustness of multi-modality re-id
models unexplored. Due to the lack of knowledge about the specific type of
model deployed in the target black-box surveillance system, we aim to generate
modality unified AEs for omni-modality (single-, cross- and multi-modality)
re-id models. Specifically, we propose a novel Modality Unified Attack method
to train modality-specific adversarial generators to generate AEs that
effectively attack different omni-modality models. A multi-modality model is
adopted as the surrogate model, wherein the features of each modality are
perturbed by metric disruption loss before fusion. To collapse the common
features of omni-modality models, Cross Modality Simulated Disruption approach
is introduced to mimic the cross-modality feature embeddings by intentionally
feeding images to non-corresponding modality-specific subnetworks of the
surrogate model. Moreover, Multi Modality Collaborative Disruption strategy is
devised to facilitate the attacker to comprehensively corrupt the informative
content of person images by leveraging a multi modality feature collaborative
metric disruption loss. Extensive experiments show that our MUA method can
effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%,
49.0% and 62.7% mean mAP Drop Rate, respectively.


http://arxiv.org/abs/2501.13094
Robust Representation Consistency Model via Contrastive Denoising. (84%)
Jiachen Lei; Julius Berner; Jiongxiao Wang; Zhongzhu Chen; Zhongjia Ba; Kui Ren; Jun Zhu; Anima Anandkumar
  Robustness is essential for deep neural networks, especially in
security-sensitive applications. To this end, randomized smoothing provides
theoretical guarantees for certifying robustness against adversarial
perturbations. Recently, diffusion models have been successfully employed for
randomized smoothing to purify noise-perturbed samples before making
predictions with a standard classifier. While these methods excel at small
perturbation radii, they struggle with larger perturbations and incur a
significant computational overhead during inference compared to classical
methods. To address this, we reformulate the generative modeling task along the
diffusion trajectories in pixel space as a discriminative task in the latent
space. Specifically, we use instance discrimination to achieve consistent
representations along the trajectories by aligning temporally adjacent points.
After fine-tuning based on the learned representations, our model enables
implicit denoising-then-classification via a single prediction, substantially
reducing inference costs. We conduct extensive experiments on various datasets
and achieve state-of-the-art performance with minimal computation budget during
inference. For example, our method outperforms the certified accuracy of
diffusion-based methods on ImageNet across all perturbation radii by 5.3% on
average, with up to 11.6% at larger radii, while reducing inference costs by
85$\times$ on average. Codes are available at:
https://github.com/jiachenlei/rRCM.


http://arxiv.org/abs/2501.13302
Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers. (11%)
Akshit Achara; Anshuman Chhabra
  AI Safety Moderation (ASM) classifiers are designed to moderate content on
social media platforms and to serve as guardrails that prevent Large Language
Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential
for disparate impact, it is crucial to ensure that these classifiers: (1) do
not unfairly classify content belonging to users from minority groups as unsafe
compared to those from majority groups and (2) that their behavior remains
robust and consistent across similar inputs. In this work, we thus examine the
fairness and robustness of four widely-used, closed-source ASM classifiers:
OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL)
API, and Clarifai API. We assess fairness using metrics such as demographic
parity and conditional statistical parity, comparing their performance against
ASM models and a fair-only baseline. Additionally, we analyze robustness by
testing the classifiers' sensitivity to small and natural input perturbations.
Our findings reveal potential fairness and robustness gaps, highlighting the
need to mitigate these issues in future versions of these models.


http://arxiv.org/abs/2501.12736
Bad-PFL: Exploring Backdoor Attacks against Personalized Federated Learning. (9%)
Mingyuan Fan; Zhanyi Hu; Fuyi Wang; Cen Chen
  Data heterogeneity and backdoor attacks rank among the most significant
challenges facing federated learning (FL). For data heterogeneity, personalized
federated learning (PFL) enables each client to maintain a private personalized
model to cater to client-specific knowledge. Meanwhile, vanilla FL has proven
vulnerable to backdoor attacks. However, recent advancements in PFL community
have demonstrated a potential immunity against such attacks. This paper
explores this intersection further, revealing that existing federated backdoor
attacks fail in PFL because backdoors about manually designed triggers struggle
to survive in personalized models. To tackle this, we design Bad-PFL, which
employs features from natural data as our trigger. As long as the model is
trained on natural data, it inevitably embeds the backdoor associated with our
trigger, ensuring its longevity in personalized models. Moreover, our trigger
undergoes mutual reinforcement training with the model, further solidifying the
backdoor's durability and enhancing attack effectiveness. The large-scale
experiments across three benchmark datasets demonstrate the superior
performance of our attack against various PFL methods, even when equipped with
state-of-the-art defense mechanisms.


http://arxiv.org/abs/2501.13291
Are We Learning the Right Features? A Framework for Evaluating DL-Based Software Vulnerability Detection Solutions. (2%)
Satyaki Das; Syeda Tasnim Fabiha; Saad Shafiq; Nenad Medvidovic
  Recent research has revealed that the reported results of an emerging body of
DL-based techniques for detecting software vulnerabilities are not
reproducible, either across different datasets or on unseen samples. This paper
aims to provide the foundation for properly evaluating the research in this
domain. We do so by analyzing prior work and existing vulnerability datasets
for the syntactic and semantic features of code that contribute to
vulnerability, as well as features that falsely correlate with vulnerability.
We provide a novel, uniform representation to capture both sets of features,
and use this representation to detect the presence of both vulnerability and
spurious features in code. To this end, we design two types of code
perturbations: feature preserving perturbations (FPP) ensure that the
vulnerability feature remains in a given code sample, while feature eliminating
perturbations (FEP) eliminate the feature from the code sample. These
perturbations aim to measure the influence of spurious and vulnerability
features on the predictions of a given vulnerability detection solution. To
evaluate how the two classes of perturbations influence predictions, we
conducted a large-scale empirical study on five state-of-the-art DL-based
vulnerability detectors. Our study shows that, for vulnerability features, only
~2% of FPPs yield the undesirable effect of a prediction changing among the
five detectors on average. However, on average, ~84% of FEPs yield the
undesirable effect of retaining the vulnerability predictions. For spurious
features, we observed that FPPs yielded a drop in recall up to 29% for
graph-based detectors. We present the reasons underlying these results and
suggest strategies for improving DNN-based vulnerability detectors. We provide
our perturbation-based evaluation framework as a public resource to enable
independent future evaluation of vulnerability detectors.


http://arxiv.org/abs/2501.11901
Enhancing Adversarial Transferability via Component-Wise Transformation. (99%)
Hangyu Liu; Bo Peng; Can Cui; Pengxiang Ding; Donglin Wang
  Deep Neural Networks (DNNs) are highly vulnerable to adversarial examples,
which pose significant challenges in security-sensitive applications. Among
various adversarial attack strategies, input transformation-based attacks have
demonstrated remarkable effectiveness in enhancing adversarial transferability.
However, existing methods still perform poorly across different architectures,
even though they have achieved promising results within the same architecture.
This limitation arises because, while models of the same architecture may focus
on different regions of the object, the variation is even more pronounced
across different architectures. Unfortunately, current approaches fail to
effectively guide models to attend to these diverse regions. To address this
issue, this paper proposes a novel input transformation-based attack method,
termed Component-Wise Transformation (CWT). CWT applies interpolation and
selective rotation to individual image blocks, ensuring that each transformed
image highlights different target regions, thereby improving the
transferability of adversarial examples. Extensive experiments on the standard
ImageNet dataset show that CWT consistently outperforms state-of-the-art
methods in both attack success rates and stability across CNN- and
Transformer-based models.


http://arxiv.org/abs/2501.12275
With Great Backbones Comes Great Adversarial Transferability. (98%)
Erik Arakelyan; Karen Hambardzumyan; Davit Papikyan; Pasquale Minervini; Albert Gordo; Isabelle Augenstein; Aram H. Markosyan
  Advances in self-supervised learning (SSL) for machine vision have improved
representation robustness and model performance, giving rise to pre-trained
backbones like \emph{ResNet} and \emph{ViT} models tuned with SSL methods such
as \emph{SimCLR}. Due to the computational and data demands of pre-training,
the utilization of such backbones becomes a strenuous necessity. However,
employing these backbones may inherit vulnerabilities to adversarial attacks.
While adversarial robustness has been studied under \emph{white-box} and
\emph{black-box} settings, the robustness of models tuned on pre-trained
backbones remains largely unexplored. Additionally, the role of tuning
meta-information in mitigating exploitation risks is unclear. This work
systematically evaluates the adversarial robustness of such models across
$20,000$ combinations of tuning meta-information, including fine-tuning
techniques, backbone families, datasets, and attack types. We propose using
proxy models to transfer attacks, simulating varying levels of target knowledge
by fine-tuning these proxies with diverse configurations. Our findings reveal
that proxy-based attacks approach the effectiveness of \emph{white-box}
methods, even with minimal tuning knowledge. We also introduce a naive
"backbone attack," leveraging only the backbone to generate adversarial
samples, which outperforms \emph{black-box} attacks and rivals \emph{white-box}
methods, highlighting critical risks in model-sharing practices. Finally, our
ablations reveal how increasing tuning meta-information impacts attack
transferability, measuring each meta-information combination.


http://arxiv.org/abs/2501.11902
Transferable Adversarial Attacks on Audio Deepfake Detection. (98%)
Muhammad Umar Farooq; Awais Khan; Kutub Uddin; Khalid Mahmood Malik
  Audio deepfakes pose significant threats, including impersonation, fraud, and
reputation damage. To address these risks, audio deepfake detection (ADD)
techniques have been developed, demonstrating success on benchmarks like
ASVspoof2019. However, their resilience against transferable adversarial
attacks remains largely unexplored. In this paper, we introduce a transferable
GAN-based adversarial attack framework to evaluate the effectiveness of
state-of-the-art (SOTA) ADD systems. By leveraging an ensemble of surrogate ADD
models and a discriminator, the proposed approach generates transferable
adversarial attacks that better reflect real-world scenarios. Unlike previous
methods, the proposed framework incorporates a self-supervised audio model to
ensure transcription and perceptual integrity, resulting in high-quality
adversarial attacks. Experimental results on benchmark dataset reveal that SOTA
ADD systems exhibit significant vulnerabilities, with accuracies dropping from
98% to 26%, 92% to 54%, and 94% to 84% in white-box, gray-box, and black-box
scenarios, respectively. When tested in other data sets, performance drops of
91% to 46%, and 94% to 67% were observed against the In-the-Wild and WaveFake
data sets, respectively. These results highlight the significant
vulnerabilities of existing ADD systems and emphasize the need to enhance their
robustness against advanced adversarial threats to ensure security and
reliability.


http://arxiv.org/abs/2501.12516
Robustness of Selected Learning Models under Label-Flipping Attack. (82%)
Sarvagya Bhargava; Mark Stamp
  In this paper we compare traditional machine learning and deep learning
models trained on a malware dataset when subjected to adversarial attack based
on label-flipping. Specifically, we investigate the robustness of Support
Vector Machines (SVM), Random Forest, Gaussian Naive Bayes (GNB), Gradient
Boosting Machine (GBM), LightGBM, XGBoost, Multilayer Perceptron (MLP),
Convolutional Neural Network (CNN), MobileNet, and DenseNet models when facing
varying percentages of misleading labels. We empirically assess the the
accuracy of each of these models under such an adversarial attack on the
training data. This research aims to provide insights into which models are
inherently more robust, in the sense of being better able to resist intentional
disruptions to the training data. We find wide variation in the robustness of
the models tested to adversarial attack, with our MLP model achieving the best
combination of initial accuracy and robustness.


http://arxiv.org/abs/2501.17882
Heterogeneous Multi-Player Multi-Armed Bandits Robust To Adversarial Attacks. (82%)
Akshayaa Magesh; Venugopal V. Veeravalli
  We consider a multi-player multi-armed bandit setting in the presence of
adversaries that attempt to negatively affect the rewards received by the
players in the system. The reward distributions for any given arm are
heterogeneous across the players. In the event of a collision (more than one
player choosing the same arm), all the colliding users receive zero rewards.
The adversaries use collisions to affect the rewards received by the players,
i.e., if an adversary attacks an arm, any player choosing that arm will receive
zero reward. At any time step, the adversaries may attack more than one arm. It
is assumed that the players in the system do not deviate from a pre-determined
policy used by all the players, and that the probability that none of the arms
face adversarial attacks is strictly positive at every time step. In order to
combat the adversarial attacks, the players are allowed to communicate using a
single bit for $O(\log T)$ time units, where $T$ is the time horizon, and each
player can only observe their own actions and rewards at all time steps. We
propose a {policy that is used by all the players, which} achieves near order
optimal regret of order $O(\log^{1+\delta}T + W)$, where $W$ is total number of
time units for which there was an adversarial attack on at least one arm.


http://arxiv.org/abs/2501.12183
Extend Adversarial Policy Against Neural Machine Translation via Unknown Token. (80%)
Wei Zou; Shujian Huang; Jiajun Chen
  Generating adversarial examples contributes to mainstream neural machine
translation~(NMT) robustness. However, popular adversarial policies are apt for
fixed tokenization, hindering its efficacy for common character perturbations
involving versatile tokenization. Based on existing adversarial generation via
reinforcement learning~(RL), we propose the `DexChar policy' that introduces
character perturbations for the existing mainstream adversarial policy based on
token substitution. Furthermore, we improve the self-supervised matching that
provides feedback in RL to cater to the semantic constraints required during
training adversaries. Experiments show that our method is compatible with the
scenario where baseline adversaries fail, and can generate high-efficiency
adversarial examples for analysis and optimization of the system.


http://arxiv.org/abs/2501.12522
Topology of Out-of-Distribution Examples in Deep Neural Networks. (13%)
Esha Datta; Johanna Hennig; Eva Domschot; Connor Mattes; Michael R. Smith
  As deep neural networks (DNNs) become increasingly common, concerns about
their robustness do as well. A longstanding problem for deployed DNNs is their
behavior in the face of unfamiliar inputs; specifically, these models tend to
be overconfident and incorrect when encountering out-of-distribution (OOD)
examples. In this work, we present a topological approach to characterizing OOD
examples using latent layer embeddings from DNNs. Our goal is to identify
topological features, referred to as landmarks, that indicate OOD examples. We
conduct extensive experiments on benchmark datasets and a realistic DNN model,
revealing a key insight for OOD detection. Well-trained DNNs have been shown to
induce a topological simplification on training data for simple models and
datasets; we show that this property holds for realistic, large-scale test and
training data, but does not hold for OOD examples. More specifically, we find
that the average lifetime (or persistence) of OOD examples is statistically
longer than that of training or test examples. This indicates that DNNs
struggle to induce topological simplification on unfamiliar inputs. Our
empirical results provide novel evidence of topological simplification in
realistic DNNs and lay the groundwork for topologically-informed OOD detection
strategies.


http://arxiv.org/abs/2501.12269
Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems. (5%)
Stefano Carlo Lambertenghi; Hannes Leonhard; Andrea Stocco
  Advanced Driver Assistance Systems (ADAS) based on deep neural networks
(DNNs) are widely used in autonomous vehicles for critical perception tasks
such as object detection, semantic segmentation, and lane recognition. However,
these systems are highly sensitive to input variations, such as noise and
changes in lighting, which can compromise their effectiveness and potentially
lead to safety-critical failures.
  This study offers a comprehensive empirical evaluation of image
perturbations, techniques commonly used to assess the robustness of DNNs, to
validate and improve the robustness and generalization of ADAS perception
systems. We first conducted a systematic review of the literature, identifying
38 categories of perturbations. Next, we evaluated their effectiveness in
revealing failures in two different ADAS, both at the component and at the
system level. Finally, we explored the use of perturbation-based data
augmentation and continuous learning strategies to improve ADAS adaptation to
new operational design domains. Our results demonstrate that all categories of
image perturbations successfully expose robustness issues in ADAS and that the
use of dataset augmentation and continuous learning significantly improves ADAS
performance in novel, unseen environments.


http://arxiv.org/abs/2501.12191
A margin-based replacement for cross-entropy loss. (1%)
Michael W. Spratling; Heiko H. Schütt
  Cross-entropy (CE) loss is the de-facto standard for training deep neural
networks to perform classification. However, CE-trained deep neural networks
struggle with robustness and generalisation issues. To alleviate these issues,
we propose high error margin (HEM) loss, a variant of multi-class margin loss
that overcomes the training issues of other margin-based losses. We evaluate
HEM extensively on a range of architectures and datasets. We find that HEM loss
is more effective than cross-entropy loss across a wide range of tasks: unknown
class rejection, adversarial robustness, learning with imbalanced data,
continual learning, and semantic segmentation (a pixel-level classification
task). Despite all training hyper-parameters being chosen for CE loss, HEM is
inferior to CE only in terms of clean accuracy and this difference is
insignificant. We also compare HEM to specialised losses that have previously
been proposed to improve performance on specific tasks. LogitNorm, a loss
achieving state-of-the-art performance on unknown class rejection, produces
similar performance to HEM for this task, but is much poorer for continual
learning and semantic segmentation. Logit-adjusted loss, designed for
imbalanced data, has superior results to HEM for that task, but performs more
poorly on unknown class rejection and semantic segmentation. DICE, a popular
loss for semantic segmentation, is inferior to HEM loss on all tasks, including
semantic segmentation. Thus, HEM often out-performs specialised losses, and in
contrast to them, is a general-purpose replacement for CE loss.


http://arxiv.org/abs/2501.12210
You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defense. (1%)
Wuyuao Mai; Geng Hong; Pei Chen; Xudong Pan; Baojun Liu; Yuan Zhang; Haixin Duan; Min Yang
  With the rise of generative large language models (LLMs) like LLaMA and
ChatGPT, these models have significantly transformed daily life and work by
providing advanced insights. However, as jailbreak attacks continue to
circumvent built-in safety mechanisms, exploiting carefully crafted scenarios
or tokens, the safety risks of LLMs have come into focus. While numerous
defense strategies--such as prompt detection, modification, and model
fine-tuning--have been proposed to counter these attacks, a critical question
arises: do these defenses compromise the utility and usability of LLMs for
legitimate users? Existing research predominantly focuses on the effectiveness
of defense strategies without thoroughly examining their impact on performance,
leaving a gap in understanding the trade-offs between LLM safety and
performance. Our research addresses this gap by conducting a comprehensive
study on the utility degradation, safety elevation, and exaggerated-safety
escalation of LLMs with jailbreak defense strategies. We propose USEBench, a
novel benchmark designed to evaluate these aspects, along with USEIndex, a
comprehensive metric for assessing overall model performance. Through
experiments on seven state-of-the-art LLMs, we found that mainstream jailbreak
defenses fail to ensure both safety and performance simultaneously. Although
model-finetuning performs the best overall, their effectiveness varies across
LLMs. Furthermore, vertical comparisons reveal that developers commonly
prioritize performance over safety when iterating or fine-tuning their LLMs.


http://arxiv.org/abs/2501.11568
Graph Defense Diffusion Model. (97%)
Xin He; Wenqi Fan; Yili Wang; Chengyi Liu; Rui Miao; Xin Juan; Xin Wang
  Graph Neural Networks (GNNs) demonstrate significant potential in various
applications but remain highly vulnerable to adversarial attacks, which can
greatly degrade their performance. Existing graph purification methods attempt
to address this issue by filtering attacked graphs; however, they struggle to
effectively defend against multiple types of adversarial attacks simultaneously
due to their limited flexibility, and they lack comprehensive modeling of graph
data due to their heavy reliance on heuristic prior knowledge. To overcome
these challenges, we propose a more versatile approach for defending against
adversarial attacks on graphs. In this work, we introduce the Graph Defense
Diffusion Model (GDDM), a flexible purification method that leverages the
denoising and modeling capabilities of diffusion models. The iterative nature
of diffusion models aligns well with the stepwise process of adversarial
attacks, making them particularly suitable for defense. By iteratively adding
and removing noise, GDDM effectively purifies attacked graphs, restoring their
original structure and features. Our GDDM consists of two key components: (1)
Graph Structure-Driven Refiner, which preserves the basic fidelity of the graph
during the denoising process, and ensures that the generated graph remains
consistent with the original scope; and (2) Node Feature-Constrained
Regularizer, which removes residual impurities from the denoised graph, further
enhances the purification effect. Additionally, we design tailored denoising
strategies to handle different types of adversarial attacks, improving the
model's adaptability to various attack scenarios. Extensive experiments
conducted on three real-world datasets demonstrate that GDDM outperforms
state-of-the-art methods in defending against a wide range of adversarial
attacks, showcasing its robustness and effectiveness.


http://arxiv.org/abs/2501.11848
FedMUA: Exploring the Vulnerabilities of Federated Learning to Malicious Unlearning Attacks. (67%)
Jian Chen; Zehui Lin; Wanyu Lin; Wenlong Shi; Xiaoyan Yin; Di Wang
  Recently, the practical needs of ``the right to be forgotten'' in federated
learning gave birth to a paradigm known as federated unlearning, which enables
the server to forget personal data upon the client's removal request. Existing
studies on federated unlearning have primarily focused on efficiently
eliminating the influence of requested data from the client's model without
retraining from scratch, however, they have rarely doubted the reliability of
the global model posed by the discrepancy between its prediction performance
before and after unlearning. To bridge this gap, we take the first step by
introducing a novel malicious unlearning attack dubbed FedMUA, aiming to unveil
potential vulnerabilities emerging from federated learning during the
unlearning process. The crux of FedMUA is to mislead the global model into
unlearning more information associated with the influential samples for the
target sample than anticipated, thus inducing adverse effects on target samples
from other clients. To achieve this, we design a novel two-step method, known
as Influential Sample Identification and Malicious Unlearning Generation, to
identify and subsequently generate malicious feature unlearning requests within
the influential samples. By doing so, we can significantly alter the
predictions pertaining to the target sample by initiating the malicious feature
unlearning requests, leading to the deliberate manipulation for the user
adversely. Additionally, we design a new defense mechanism that is highly
resilient against malicious unlearning attacks. Extensive experiments on three
realistic datasets reveal that FedMUA effectively induces misclassification on
target samples and can achieve an 80% attack success rate by triggering only
0.3% malicious unlearning requests.


http://arxiv.org/abs/2501.11759
Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systems. (50%)
Fatemeh Nazary; Yashar Deldjoo; Noia Tommaso di
  This study presents Poison-RAG, a framework for adversarial data poisoning
attacks targeting retrieval-augmented generation (RAG)-based recommender
systems. Poison-RAG manipulates item metadata, such as tags and descriptions,
to influence recommendation outcomes. Using item metadata generated through a
large language model (LLM) and embeddings derived via the OpenAI API, we
explore the impact of adversarial poisoning attacks on provider-side, where
attacks are designed to promote long-tail items and demote popular ones. Two
attack strategies are proposed: local modifications, which personalize tags for
each item using BERT embeddings, and global modifications, applying uniform
tags across the dataset. Experiments conducted on the MovieLens dataset in a
black-box setting reveal that local strategies improve manipulation
effectiveness by up to 50\%, while global strategies risk boosting already
popular items. Results indicate that popular items are more susceptible to
attacks, whereas long-tail items are harder to manipulate. Approximately 70\%
of items lack tags, presenting a cold-start challenge; data augmentation and
synthesis are proposed as potential defense mechanisms to enhance RAG-based
systems' resilience. The findings emphasize the need for robust metadata
management to safeguard recommendation frameworks. Code and data are available
at https://github.com/atenanaz/Poison-RAG.


http://arxiv.org/abs/2501.11462
On the Adversarial Vulnerabilities of Transfer Learning in Remote Sensing. (41%)
Tao Bai; Xingjian Tian; Yonghao Xu; Bihan Wen
  The use of pretrained models from general computer vision tasks is widespread
in remote sensing, significantly reducing training costs and improving
performance. However, this practice also introduces vulnerabilities to
downstream tasks, where publicly available pretrained models can be used as a
proxy to compromise downstream models. This paper presents a novel Adversarial
Neuron Manipulation method, which generates transferable perturbations by
selectively manipulating single or multiple neurons in pretrained models.
Unlike existing attacks, this method eliminates the need for domain-specific
information, making it more broadly applicable and efficient. By targeting
multiple fragile neurons, the perturbations achieve superior attack
performance, revealing critical vulnerabilities in deep learning models.
Experiments on diverse models and remote sensing datasets validate the
effectiveness of the proposed method. This low-access adversarial neuron
manipulation technique highlights a significant security risk in transfer
learning models, emphasizing the urgent need for more robust defenses in their
design when addressing the safety-critical remote sensing tasks.


http://arxiv.org/abs/2501.11852
Cross-Entropy Attacks to Language Models via Rare Event Simulation. (12%)
Mingze Ni; Yongshun Gong; Wei Liu
  Black-box textual adversarial attacks are challenging due to the lack of
model information and the discrete, non-differentiable nature of text. Existing
methods often lack versatility for attacking different models, suffer from
limited attacking performance due to the inefficient optimization with word
saliency ranking, and frequently sacrifice semantic integrity to achieve better
attack outcomes. This paper introduces a novel approach to textual adversarial
attacks, which we call Cross-Entropy Attacks (CEA), that uses Cross-Entropy
optimization to address the above issues. Our CEA approach defines adversarial
objectives for both soft-label and hard-label settings and employs CE
optimization to identify optimal replacements. Through extensive experiments on
document classification and language translation problems, we demonstrate that
our attack method excels in terms of attacking performance, imperceptibility,
and sentence quality.


http://arxiv.org/abs/2501.11577
Rethinking Membership Inference Attacks Against Transfer Learning. (9%)
Cong Wu; Jing Chen; Qianru Fang; Kun He; Ziming Zhao; Hao Ren; Guowen Xu; Yang Liu; Yang Xiang
  Transfer learning, successful in knowledge translation across related tasks,
faces a substantial privacy threat from membership inference attacks (MIAs).
These attacks, despite posing significant risk to ML model's training data,
remain limited-explored in transfer learning. The interaction between teacher
and student models in transfer learning has not been thoroughly explored in
MIAs, potentially resulting in an under-examined aspect of privacy
vulnerabilities within transfer learning. In this paper, we propose a new MIA
vector against transfer learning, to determine whether a specific data point
was used to train the teacher model while only accessing the student model in a
white-box setting. Our method delves into the intricate relationship between
teacher and student models, analyzing the discrepancies in hidden layer
representations between the student model and its shadow counterpart. These
identified differences are then adeptly utilized to refine the shadow model's
training process and to inform membership inference decisions effectively. Our
method, evaluated across four datasets in diverse transfer learning tasks,
reveals that even when an attacker only has access to the student model, the
teacher model's training data remains susceptible to MIAs. We believe our work
unveils the unexplored risk of membership inference in transfer learning.


http://arxiv.org/abs/2501.11815
CogMorph: Cognitive Morphing Attacks for Text-to-Image Models. (1%)
Zonglei Jing; Zonghao Ying; Le Wang; Siyuan Liang; Aishan Liu; Xianglong Liu; Dacheng Tao
  The development of text-to-image (T2I) generative models, that enable the
creation of high-quality synthetic images from textual prompts, has opened new
frontiers in creative design and content generation. However, this paper
reveals a significant and previously unrecognized ethical risk inherent in this
technology and introduces a novel method, termed the Cognitive Morphing Attack
(CogMorph), which manipulates T2I models to generate images that retain the
original core subjects but embeds toxic or harmful contextual elements. This
nuanced manipulation exploits the cognitive principle that human perception of
concepts is shaped by the entire visual scene and its context, producing images
that amplify emotional harm far beyond attacks that merely preserve the
original semantics. To address this, we first construct an imagery toxicity
taxonomy spanning 10 major and 48 sub-categories, aligned with human
cognitive-perceptual dimensions, and further build a toxicity risk matrix
resulting in 1,176 high-quality T2I toxic prompts. Based on this, our CogMorph
first introduces Cognitive Toxicity Augmentation, which develops a cognitive
toxicity knowledge base with rich external toxic representations for humans
(e.g., fine-grained visual features) that can be utilized to further guide the
optimization of adversarial prompts. In addition, we present Contextual
Hierarchical Morphing, which hierarchically extracts critical parts of the
original prompt (e.g., scenes, subjects, and body parts), and then iteratively
retrieves and fuses toxic features to inject harmful contexts. Extensive
experiments on multiple open-sourced T2I models and black-box commercial APIs
(e.g., DALLE-3) demonstrate the efficacy of CogMorph which significantly
outperforms other baselines by large margins (+20.62% on average).


http://arxiv.org/abs/2501.10996
Effectiveness of Adversarial Benign and Malware Examples in Evasion and Poisoning Attacks. (99%)
Matouš Kozák; Martin Jureček
  Adversarial attacks present significant challenges for malware detection
systems. This research investigates the effectiveness of benign and malicious
adversarial examples (AEs) in evasion and poisoning attacks on the Portable
Executable file domain. A novel focus of this study is on benign AEs, which,
although not directly harmful, can increase false positives and undermine trust
in antivirus solutions. We propose modifying existing adversarial malware
generators to produce benign AEs and show they are as successful as malware AEs
in evasion attacks. Furthermore, our data show that benign AEs have a more
decisive influence in poisoning attacks than standard malware AEs,
demonstrating their superior ability to decrease the model's performance. Our
findings introduce new opportunities for adversaries and further increase the
attack surface that needs to be protected by security researchers.


http://arxiv.org/abs/2501.10985
GRID: Protecting Training Graph from Link Stealing Attacks on GNN Models. (12%)
Jiadong Lou; Xu Yuan; Rui Zhang; Xingliang Yuan; Neil Gong; Nian-Feng Tzeng
  Graph neural networks (GNNs) have exhibited superior performance in various
classification tasks on graph-structured data. However, they encounter the
potential vulnerability from the link stealing attacks, which can infer the
presence of a link between two nodes via measuring the similarity of its
incident nodes' prediction vectors produced by a GNN model. Such attacks pose
severe security and privacy threats to the training graph used in GNN models.
In this work, we propose a novel solution, called Graph Link Disguise (GRID),
to defend against link stealing attacks with the formal guarantee of GNN model
utility for retaining prediction accuracy. The key idea of GRID is to add
carefully crafted noises to the nodes' prediction vectors for disguising
adjacent nodes as n-hop indirect neighboring nodes. We take into account the
graph topology and select only a subset of nodes (called core nodes) covering
all links for adding noises, which can avert the noises offset and have the
further advantages of reducing both the distortion loss and the computation
cost. Our crafted noises can ensure 1) the noisy prediction vectors of any two
adjacent nodes have their similarity level like that of two non-adjacent nodes
and 2) the model prediction is unchanged to ensure zero utility loss. Extensive
experiments on five datasets are conducted to show the effectiveness of our
proposed GRID solution against different representative link-stealing attacks
under transductive settings and inductive settings respectively, as well as two
influence-based attacks. Meanwhile, it achieves a much better privacy-utility
trade-off than existing methods when extended to GNNs.


http://arxiv.org/abs/2501.11183
Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecurity. (10%)
David Williams-King; Linh Le; Adam Oberman; Yoshua Bengio
  As LLMs develop increasingly advanced capabilities, there is an increased
need to minimize the harm that could be caused to society by certain model
outputs; hence, most LLMs have safety guardrails added, for example via
fine-tuning. In this paper, we argue the position that current safety
fine-tuning is very similar to a traditional cat-and-mouse game (or arms race)
between attackers and defenders in cybersecurity. Model jailbreaks and attacks
are patched with bandaids to target the specific attack mechanism, but many
similar attack vectors might remain. When defenders are not proactively coming
up with principled mechanisms, it becomes very easy for attackers to sidestep
any new defenses. We show how current defenses are insufficient to prevent new
adversarial jailbreak attacks, reward hacking, and loss of control problems. In
order to learn from past mistakes in cybersecurity, we draw analogies with
historical examples and develop lessons learned that can be applied to LLM
safety. These arguments support the need for new and more principled approaches
to designing safe models, which are architected for security from the
beginning. We describe several such approaches from the AI literature.


http://arxiv.org/abs/2501.13115
Dagger Behind Smile: Fool LLMs with a Happy Ending Story. (5%)
Xurui Song; Zhixin Xie; Shuo Huai; Jiayi Kong; Jun Luo
  The wide adoption of Large Language Models (LLMs) has attracted significant
attention from \textit{jailbreak} attacks, where adversarial prompts crafted
through optimization or manual design exploit LLMs to generate malicious
content. However, optimization-based attacks have limited efficiency and
transferability, while manual designs are either easily detectable or demand
intricate interactions with LLMs. In this paper, we first point out a novel
perspective for jailbreak attacks: LLMs are more responsive to
\textit{positive} prompts. Based on this, we deploy Happy Ending Attack (HEA)
to wrap up a malicious request in a scenario template involving a positive
prompt formed mainly via a \textit{happy ending}, it thus fools LLMs into
jailbreaking either immediately or at a follow-up malicious request. This has
made HEA both efficient and effective, as it requires only up to two steps to
fully jailbreak LLMs. Extensive experiments show that our HEA can successfully
jailbreak on state-of-the-art LLMs, including GPT-4o, Llama3-70b, Gemini-pro,
and achieves 88.79\% Attack Success Rate on average. We also provide potential
quantitative explanations for the success of HEA.


http://arxiv.org/abs/2501.11054
Temporal Analysis of Adversarial Attacks in Federated Learning. (2%)
Rohit Mapakshi; Sayma Akther; Mark Stamp
  In this paper, we experimentally analyze the robustness of selected Federated
Learning (FL) systems in the presence of adversarial clients. We find that
temporal attacks significantly affect model performance in the FL models
tested, especially when the adversaries are active throughout or during the
later rounds. We consider a variety of classic learning models, including
Multinominal Logistic Regression (MLR), Random Forest, XGBoost, Support Vector
Classifier (SVC), as well as various Neural Network models including Multilayer
Perceptron (MLP), Convolution Neural Network (CNN), Recurrent Neural Network
(RNN), and Long Short-Term Memory (LSTM). Our results highlight the
effectiveness of temporal attacks and the need to develop strategies to make
the FL process more robust against such attacks. We also briefly consider the
effectiveness of defense mechanisms, including outlier detection in the
aggregation algorithm.


http://arxiv.org/abs/2501.11171
Counteracting temporal attacks in Video Copy Detection. (1%)
Katarzyna Fojcik; Piotr Syga
  Video Copy Detection (VCD) plays a crucial role in copyright protection and
content verification by identifying duplicates and near-duplicates in
large-scale video databases. The META AI Challenge on video copy detection
provided a benchmark for evaluating state-of-the-art methods, with the
Dual-level detection approach emerging as a winning solution. This method
integrates Video Editing Detection and Frame Scene Detection to handle
adversarial transformations and large datasets efficiently. However, our
analysis reveals significant limitations in the VED component, particularly in
its ability to handle exact copies. Moreover, Dual-level detection shows
vulnerability to temporal attacks. To address it, we propose an improved frame
selection strategy based on local maxima of interframe differences, which
enhances robustness against adversarial temporal modifications while
significantly reducing computational overhead. Our method achieves an increase
of 1.4 to 5.8 times in efficiency over the standard 1 FPS approach. Compared to
Dual-level detection method, our approach maintains comparable micro-average
precision ($\mu$AP) while also demonstrating improved robustness against
temporal attacks. Given 56\% reduced representation size and the inference time
of more than 2 times faster, our approach is more suitable to real-world
resource restriction.


http://arxiv.org/abs/2501.10906
Explainable Adversarial Attacks on Coarse-to-Fine Classifiers. (86%)
Akram Heidarizadeh; Connor Hatfield; Lorenzo Lazzarotto; HanQin Cai; George Atia
  Traditional adversarial attacks typically aim to alter the predicted labels
of input images by generating perturbations that are imperceptible to the human
eye. However, these approaches often lack explainability. Moreover, most
existing work on adversarial attacks focuses on single-stage classifiers, but
multi-stage classifiers are largely unexplored. In this paper, we introduce
instance-based adversarial attacks for multi-stage classifiers, leveraging
Layer-wise Relevance Propagation (LRP), which assigns relevance scores to
pixels based on their influence on classification outcomes. Our approach
generates explainable adversarial perturbations by utilizing LRP to identify
and target key features critical for both coarse and fine-grained
classifications. Unlike conventional attacks, our method not only induces
misclassification but also enhances the interpretability of the model's
behavior across classification stages, as demonstrated by experimental results.


http://arxiv.org/abs/2501.10817
A comprehensive survey on RPL routing-based attacks, defences and future directions in Internet of Things. (12%)
Anil K Prajapati; Emmanuel S Pilli; Ramesh B Battula; Vijay Varadharajan; Abhishek Verma; R C Joshi
  The Internet of Things (IoT) is a network of digital devices like sensors,
processors, embedded and communication devices that can connect to and exchange
data with other devices and systems over the internet. IoT devices have
limitations on power, memory, and computational resources. Researchers have
developed the IPv6 Over Low-power Wireless Personal Area Network (6LoWPAN)
protocols to provide wireless connectivity among these devices while overcoming
the constraints on resources. 6LoWPAN has been approved subsequently by the
Internet Engineering Task Force (IETF). The IETF Routing Over Low-power and
Lossy Networks (ROLL) standardized the Routing Protocol for LLNs known as RPL
(IETF RFC 6550), which is part of the 6LoWPAN stack. However, IoT devices are
vulnerable to various attacks on RPL-based routing. This survey provides an in
depth study of existing RPL-based attacks and defense published from year 2011
to 2024 from highly reputed journals and conferences. By thematic analysis of
existing routing attacks on RPL, we developed a novel attack taxonomy which
focuses on the nature of routing attacks and classifies them into 12 major
categories. Subsequently, the impact of each attack on the network is analyzed
and discussed real life scenarios of these attacks. Another contribution of
this survey proposed a novel taxonomy for classification of defense mechanisms
into 8 major categories against routing attacks based on type of defense
strategy. The detailed analysis of each defense mechanism with real life
applicability is explained. Furthermore, evaluation tools such as testbeds and
simulators for RPL-based attack and defense are discussed and critically
analyzed in terms of real world applicability. Finally, open research
challenges are presented on the basis of research gaps of existing literature
along with research directions for practitioners and researchers.


http://arxiv.org/abs/2501.10013
CaFA: Cost-aware, Feasible Attacks With Database Constraints Against Neural Tabular Classifiers. (99%)
Matan Ben-Tov; Daniel Deutch; Nave Frost; Mahmood Sharif
  This work presents CaFA, a system for Cost-aware Feasible Attacks for
assessing the robustness of neural tabular classifiers against adversarial
examples realizable in the problem space, while minimizing adversaries' effort.
To this end, CaFA leverages TabPGD$-$an algorithm we set forth to generate
adversarial perturbations suitable for tabular data$-$ and incorporates
integrity constraints automatically mined by state-of-the-art database methods.
After producing adversarial examples in the feature space via TabPGD, CaFA
projects them on the mined constraints, leading, in turn, to better attack
realizability. We tested CaFA with three datasets and two architectures and
found, among others, that the constraints we use are of higher quality
(measured via soundness and completeness) than ones employed in prior work.
Moreover, CaFA achieves higher feasible success rates$-$i.e., it generates
adversarial examples that are often misclassified while satisfying
constraints$-$than prior attacks while simultaneously perturbing few features
with lower magnitudes, thus saving effort and improving inconspicuousness. We
open-source CaFA, hoping it will serve as a generic system enabling
machine-learning engineers to assess their models' robustness against
realizable attacks, thus advancing deployed models' trustworthiness.


http://arxiv.org/abs/2501.10606
Differentiable Adversarial Attacks for Marked Temporal Point Processes. (86%)
Pritish Chakraborty; Vinayak Gupta; Rahul R; Srikanta J. Bedathur; Abir De
  Marked temporal point processes (MTPPs) have been shown to be extremely
effective in modeling continuous time event sequences (CTESs). In this work, we
present adversarial attacks designed specifically for MTPP models. A key
criterion for a good adversarial attack is its imperceptibility. For objects
such as images or text, this is often achieved by bounding perturbation in some
fixed $L_p$ norm-ball. However, similarly minimizing distance norms between two
CTESs in the context of MTPPs is challenging due to their sequential nature and
varying time-scales and lengths. We address this challenge by first permuting
the events and then incorporating the additive noise to the arrival timestamps.
However, the worst case optimization of such adversarial attacks is a hard
combinatorial problem, requiring exploration across a permutation space that is
factorially large in the length of the input sequence. As a result, we propose
a novel differentiable scheme PERMTPP using which we can perform adversarial
attacks by learning to minimize the likelihood, while minimizing the distance
between two CTESs. Our experiments on four real-world datasets demonstrate the
offensive and defensive capabilities, and lower inference times of PERMTPP.


http://arxiv.org/abs/2501.10639
Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks. (62%)
Xin Yi; Yue Li; Dongsheng Shi; Linlin Wang; Xiaoling Wang; Liang He
  Ensuring safety alignment is a critical requirement for large language models
(LLMs), particularly given increasing deployment in real-world applications.
Despite considerable advancements, LLMs remain susceptible to jailbreak
attacks, which exploit system vulnerabilities to circumvent safety measures and
elicit harmful or inappropriate outputs. Furthermore, while adversarial
training-based defense methods have shown promise, a prevalent issue is the
unintended over-defense behavior, wherein models excessively reject benign
queries, significantly undermining their practical utility. To address these
limitations, we introduce LATPC, a Latent-space Adversarial Training with
Post-aware Calibration framework. LATPC dynamically identifies safety-critical
latent dimensions by contrasting harmful and benign inputs, enabling the
adaptive construction of targeted refusal feature removal attacks. This
mechanism allows adversarial training to concentrate on real-world jailbreak
tactics that disguise harmful queries as benign ones. During inference, LATPC
employs an efficient embedding-level calibration mechanism to minimize
over-defense behaviors with negligible computational overhead. Experimental
results across five types of disguise-based jailbreak attacks demonstrate that
LATPC achieves a superior balance between safety and utility compared to
existing defense frameworks. Further analysis demonstrates the effectiveness of
leveraging safety-critical dimensions in developing robust defense methods
against jailbreak attacks.


http://arxiv.org/abs/2501.13941
GaussMark: A Practical Approach for Structural Watermarking of Language Models. (2%)
Adam Block; Ayush Sekhari; Alexander Rakhlin
  Recent advances in Large Language Models (LLMs) have led to significant
improvements in natural language processing tasks, but their ability to
generate human-quality text raises significant ethical and operational concerns
in settings where it is important to recognize whether or not a given text was
generated by a human. Thus, recent work has focused on developing techniques
for watermarking LLM-generated text, i.e., introducing an almost imperceptible
signal that allows a provider equipped with a secret key to determine if given
text was generated by their model. Current watermarking techniques are often
not practical due to concerns with generation latency, detection time,
degradation in text quality, or robustness. Many of these drawbacks come from
the focus on token-level watermarking, which ignores the inherent structure of
text. In this work, we introduce a new scheme, GaussMark, that is simple and
efficient to implement, has formal statistical guarantees on its efficacy,
comes at no cost in generation latency, and embeds the watermark into the
weights of the model itself, providing a structural watermark. Our approach is
based on Gaussian independence testing and is motivated by recent empirical
observations that minor additive corruptions to LLM weights can result in
models of identical (or even improved) quality. We show that by adding a small
amount of Gaussian noise to the weights of a given LLM, we can watermark the
model in a way that is statistically detectable by a provider who retains the
secret key. We provide formal statistical bounds on the validity and power of
our procedure. Through an extensive suite of experiments, we demonstrate that
GaussMark is reliable, efficient, and relatively robust to corruptions such as
insertions, deletions, substitutions, and roundtrip translations and can be
instantiated with essentially no loss in model quality.


http://arxiv.org/abs/2501.09446
Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness. (98%)
Zeyu Wang; Cihang Xie; Brian Bartoldson; Bhavya Kailkhura
  This paper investigates the robustness of vision-language models against
adversarial visual perturbations and introduces a novel ``double visual
defense" to enhance this robustness. Unlike previous approaches that resort to
lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform
large-scale adversarial vision-language pre-training from scratch using
web-scale data. We then strengthen the defense by incorporating adversarial
visual instruction tuning. The resulting models from each stage, $\Delta$CLIP
and $\Delta^2$LLaVA, show substantially enhanced zero-shot robustness and set a
new state-of-the-art in adversarial defense for vision-language models. For
example, the adversarial robustness of $\Delta$CLIP surpasses that of the
previous best models on ImageNet-1k by ~20%. %For example, $\Delta$CLIP
surpasses the previous best models on ImageNet-1k by ~20% in terms of
adversarial robustness. Similarly, compared to prior art, $\Delta^2$LLaVA
brings a ~30% robustness improvement to image captioning task and a ~20%
robustness improvement to visual question answering task. Furthermore, our
models exhibit stronger zero-shot recognition capability, fewer hallucinations,
and superior reasoning performance compared to baselines. Our project page is
https://doublevisualdefense.github.io/.


http://arxiv.org/abs/2501.09609
Adversarial-Ensemble Kolmogorov Arnold Networks for Enhancing Indoor Wi-Fi Positioning: A Defensive Approach Against Spoofing and Signal Manipulation Attacks. (80%)
Mitul Goswami; Romit Chatterjee; Somnath Mahato; Prasant Kumar Pattnaik
  The research presents a study on enhancing the robustness of Wi-Fi-based
indoor positioning systems against adversarial attacks. The goal is to improve
the positioning accuracy and resilience of these systems under two attack
scenarios: Wi-Fi Spoofing and Signal Strength Manipulation. Three models are
developed and evaluated: a baseline model (M_Base), an adversarially trained
robust model (M_Rob), and an ensemble model (M_Ens). All models utilize a
Kolmogorov-Arnold Network (KAN) architecture. The robust model is trained with
adversarially perturbed data, while the ensemble model combines predictions
from both the base and robust models. Experimental results show that the robust
model reduces positioning error by approximately 10% compared to the baseline,
achieving 2.03 meters error under Wi-Fi spoofing and 2.00 meters under signal
strength manipulation. The ensemble model further outperforms with errors of
2.01 meters and 1.975 meters for the respective attack types. This analysis
highlights the effectiveness of adversarial training techniques in mitigating
attack impacts. The findings underscore the importance of considering
adversarial scenarios in developing indoor positioning systems, as improved
resilience can significantly enhance the accuracy and reliability of such
systems in mission-critical environments.


http://arxiv.org/abs/2501.09320
Cooperative Decentralized Backdoor Attacks on Vertical Federated Learning. (31%)
Seohyun Lee; Wenzhi Fang; Anindya Bijoy Das; Seyyedali Hosseinalipour; David J. Love; Christopher G. Brinton
  Federated learning (FL) is vulnerable to backdoor attacks, where adversaries
alter model behavior on target classification labels by embedding triggers into
data samples. While these attacks have received considerable attention in
horizontal FL, they are less understood for vertical FL (VFL), where devices
hold different features of the samples, and only the server holds the labels.
In this work, we propose a novel backdoor attack on VFL which (i) does not rely
on gradient information from the server and (ii) considers potential collusion
among multiple adversaries for sample selection and trigger embedding. Our
label inference model augments variational autoencoders with metric learning,
which adversaries can train locally. A consensus process over the adversary
graph topology determines which datapoints to poison. We further propose
methods for trigger splitting across the adversaries, with an intensity-based
implantation scheme skewing the server towards the trigger. Our convergence
analysis reveals the impact of backdoor perturbations on VFL indicated by a
stationarity gap for the trained model, which we verify empirically as well. We
conduct experiments comparing our attack with recent backdoor VFL approaches,
finding that ours obtains significantly higher success rates for the same main
task performance despite not using server information. Additionally, our
results verify the impact of collusion on attack performance.


http://arxiv.org/abs/2501.09798
Computing Optimization-Based Prompt Injections Against Closed-Weights Models By Misusing a Fine-Tuning API. (12%)
Andrey Labunets; Nishit V. Pandya; Ashish Hooda; Xiaohan Fu; Earlence Fernandes
  We surface a new threat to closed-weight Large Language Models (LLMs) that
enables an attacker to compute optimization-based prompt injections.
Specifically, we characterize how an attacker can leverage the loss-like
information returned from the remote fine-tuning interface to guide the search
for adversarial prompts. The fine-tuning interface is hosted by an LLM vendor
and allows developers to fine-tune LLMs for their tasks, thus providing
utility, but also exposes enough information for an attacker to compute
adversarial prompts. Through an experimental analysis, we characterize the
loss-like values returned by the Gemini fine-tuning API and demonstrate that
they provide a useful signal for discrete optimization of adversarial prompts
using a greedy search algorithm. Using the PurpleLlama prompt injection
benchmark, we demonstrate attack success rates between 65% and 82% on Google's
Gemini family of LLMs. These attacks exploit the classic utility-security
tradeoff - the fine-tuning interface provides a useful feature for developers
but also exposes the LLMs to powerful attacks.


http://arxiv.org/abs/2501.09328
Neural Honeytrace: A Robust Plug-and-Play Watermarking Framework against Model Extraction Attacks. (1%)
Yixiao Xu; Binxing Fang; Rui Wang; Yinghai Zhou; Shouling Ji; Yuan Liu; Mohan Li; Zhihong Tian
  Developing high-performance deep learning models is resource-intensive,
leading model owners to utilize Machine Learning as a Service (MLaaS) platforms
instead of publicly releasing their models. However, malicious users may
exploit query interfaces to execute model extraction attacks, reconstructing
the target model's functionality locally. While prior research has investigated
triggerable watermarking techniques for asserting ownership, existing methods
face significant challenges: (1) most approaches require additional training,
resulting in high overhead and limited flexibility, and (2) they often fail to
account for advanced attackers, leaving them vulnerable to adaptive attacks.
  In this paper, we propose Neural Honeytrace, a robust plug-and-play
watermarking framework against model extraction attacks. We first formulate a
watermark transmission model from an information-theoretic perspective,
providing an interpretable account of the principles and limitations of
existing triggerable watermarking. Guided by the model, we further introduce:
(1) a similarity-based training-free watermarking method for plug-and-play and
flexible watermarking, and (2) a distribution-based multi-step watermark
information transmission strategy for robust watermarking. Comprehensive
experiments on four datasets demonstrate that Neural Honeytrace outperforms
previous methods in efficiency and resisting adaptive attacks. Neural
Honeytrace reduces the average number of samples required for a worst-case
t-Test-based copyright claim from $12,000$ to $200$ with zero training cost.


http://arxiv.org/abs/2501.09086
Salient Information Preserving Adversarial Training Improves Clean and Robust Accuracy. (98%)
Timothy Redgrave; Adam Czajka
  In this work we introduce Salient Information Preserving Adversarial Training
(SIP-AT), an intuitive method for relieving the robustness-accuracy trade-off
incurred by traditional adversarial training. SIP-AT uses salient image regions
to guide the adversarial training process in such a way that fragile features
deemed meaningful by an annotator remain unperturbed during training, allowing
models to learn highly predictive non-robust features without sacrificing
overall robustness. This technique is compatible with both human-based and
automatically generated salience estimates, allowing SIP-AT to be used as a
part of human-driven model development without forcing SIP-AT to be reliant
upon additional human data. We perform experiments across multiple datasets and
architectures and demonstrate that SIP-AT is able to boost the clean accuracy
of models while maintaining a high degree of robustness against attacks at
multiple epsilon levels. We complement our central experiments with an
observational study measuring the rate at which human subjects successfully
identify perturbed images. This study helps build a more intuitive
understanding of adversarial attack strength and demonstrates the heightened
importance of low-epsilon robustness. Our results demonstrate the efficacy of
SIP-AT and provide valuable insight into the risks posed by adversarial samples
of various strengths.


http://arxiv.org/abs/2501.08862
ARMOR: Shielding Unlearnable Examples against Data Augmentation. (75%)
Xueluan Gong; Yuji Wang; Yanjiao Chen; Haocheng Dong; Yiming Li; Mengyuan Sun; Shuaike Li; Qian Wang; Chen Chen
  Private data, when published online, may be collected by unauthorized parties
to train deep neural networks (DNNs). To protect privacy, defensive noises can
be added to original samples to degrade their learnability by DNNs. Recently,
unlearnable examples are proposed to minimize the training loss such that the
model learns almost nothing. However, raw data are often pre-processed before
being used for training, which may restore the private information of protected
data. In this paper, we reveal the data privacy violation induced by data
augmentation, a commonly used data pre-processing technique to improve model
generalization capability, which is the first of its kind as far as we are
concerned. We demonstrate that data augmentation can significantly raise the
accuracy of the model trained on unlearnable examples from 21.3% to 66.1%. To
address this issue, we propose a defense framework, dubbed ARMOR, to protect
data privacy from potential breaches of data augmentation. To overcome the
difficulty of having no access to the model training process, we design a
non-local module-assisted surrogate model that better captures the effect of
data augmentation. In addition, we design a surrogate augmentation selection
strategy that maximizes distribution alignment between augmented and
non-augmented samples, to choose the optimal augmentation strategy for each
class. We also use a dynamic step size adjustment algorithm to enhance the
defensive noise generation process. Extensive experiments are conducted on 4
datasets and 5 data augmentation methods to verify the performance of ARMOR.
Comparisons with 6 state-of-the-art defense methods have demonstrated that
ARMOR can preserve the unlearnability of protected private data under data
augmentation. ARMOR reduces the test accuracy of the model trained on augmented
protected samples by as much as 60% more than baselines.


http://arxiv.org/abs/2501.09006
Improving Stability Estimates in Adversarial Explainable AI through Alternate Search Methods. (13%)
Christopher Burger; Charles Walter
  Advances in the effectiveness of machine learning models have come at the
cost of enormous complexity resulting in a poor understanding of how they
function. Local surrogate methods have been used to approximate the workings of
these complex models, but recent work has revealed their vulnerability to
adversarial attacks where the explanation produced is appreciably different
while the meaning and structure of the complex model's output remains similar.
This prior work has focused on the existence of these weaknesses but not on
their magnitude. Here we explore using an alternate search method with the goal
of finding minimum viable perturbations, the fewest perturbations necessary to
achieve a fixed similarity value between the original and altered text's
explanation. Intuitively, a method that requires fewer perturbations to expose
a given level of instability is inferior to one which requires more. This
nuance allows for superior comparisons of the stability of explainability
methods.


http://arxiv.org/abs/2501.10466
Improving the Efficiency of Self-Supervised Adversarial Training through Latent Clustering-Based Selection. (12%)
Somrita Ghosh; Yuelin Xu; Xiao Zhang
  Compared with standard learning, adversarially robust learning is widely
recognized to demand significantly more training examples. Recent works propose
the use of self-supervised adversarial training (SSAT) with external or
synthetically generated unlabeled data to enhance model robustness. However,
SSAT requires a substantial amount of extra unlabeled data, significantly
increasing memory usage and model training times. To address these challenges,
we propose novel methods to strategically select a small subset of unlabeled
data essential for SSAT and robustness improvement. Our selection prioritizes
data points near the model's decision boundary based on latent clustering-based
techniques, efficiently identifying a critical subset of unlabeled data with a
higher concentration of boundary-adjacent points. While focusing on
near-boundary data, our methods are designed to maintain a balanced ratio
between boundary and non-boundary data points to avoid overfitting. Our
experiments on image benchmarks show that integrating our selection strategies
into self-supervised adversarial training can largely reduce memory and
computational requirements while achieving high model robustness. In
particular, our latent clustering-based selection method with k-means is the
most effective, achieving nearly identical test-time robust accuracies with 5
to 10 times less external or generated unlabeled data when applied to image
benchmarks. Additionally, we validate the generalizability of our approach
across various application scenarios, including a real-world medical dataset
for COVID-19 chest X-ray classification.


http://arxiv.org/abs/2501.07922
VENOM: Text-driven Unrestricted Adversarial Example Generation with Diffusion Models. (99%)
Hui Kuurila-Zhang; Haoyu Chen; Guoying Zhao
  Adversarial attacks have proven effective in deceiving machine learning
models by subtly altering input images, motivating extensive research in recent
years. Traditional methods constrain perturbations within $l_p$-norm bounds,
but advancements in Unrestricted Adversarial Examples (UAEs) allow for more
complex, generative-model-based manipulations. Diffusion models now lead UAE
generation due to superior stability and image quality over GANs. However,
existing diffusion-based UAE methods are limited to using reference images and
face challenges in generating Natural Adversarial Examples (NAEs) directly from
random noise, often producing uncontrolled or distorted outputs. In this work,
we introduce VENOM, the first text-driven framework for high-quality
unrestricted adversarial examples generation through diffusion models. VENOM
unifies image content generation and adversarial synthesis into a single
reverse diffusion process, enabling high-fidelity adversarial examples without
sacrificing attack success rate (ASR). To stabilize this process, we
incorporate an adaptive adversarial guidance strategy with momentum, ensuring
that the generated adversarial examples $x^*$ align with the distribution
$p(x)$ of natural images. Extensive experiments demonstrate that VENOM achieves
superior ASR and image quality compared to prior methods, marking a significant
advancement in adversarial example generation and providing insights into model
vulnerabilities for improved defense development.


http://arxiv.org/abs/2501.08415
Cross-Modal Transferable Image-to-Video Attack on Video Quality Metrics. (99%)
Georgii Gotin; Ekaterina Shumitskaya; Anastasia Antsiferova; Dmitriy Vatolin
  Recent studies have revealed that modern image and video quality assessment
(IQA/VQA) metrics are vulnerable to adversarial attacks. An attacker can
manipulate a video through preprocessing to artificially increase its quality
score according to a certain metric, despite no actual improvement in visual
quality. Most of the attacks studied in the literature are white-box attacks,
while black-box attacks in the context of VQA have received less attention.
Moreover, some research indicates a lack of transferability of adversarial
examples generated for one model to another when applied to VQA. In this paper,
we propose a cross-modal attack method, IC2VQA, aimed at exploring the
vulnerabilities of modern VQA models. This approach is motivated by the
observation that the low-level feature spaces of images and videos are similar.
We investigate the transferability of adversarial perturbations across
different modalities; specifically, we analyze how adversarial perturbations
generated on a white-box IQA model with an additional CLIP module can
effectively target a VQA model. The addition of the CLIP module serves as a
valuable aid in increasing transferability, as the CLIP model is known for its
effective capture of low-level semantics. Extensive experiments demonstrate
that IC2VQA achieves a high success rate in attacking three black-box VQA
models. We compare our method with existing black-box attack strategies,
highlighting its superiority in terms of attack success within the same number
of iterations and levels of attack strength. We believe that the proposed
method will contribute to the deeper analysis of robust VQA metrics.


http://arxiv.org/abs/2501.08258
Towards an End-to-End (E2E) Adversarial Learning and Application in the Physical World. (97%)
Dudi Biton; Jacob Shams; Satoru Koda; Asaf Shabtai; Yuval Elovici; Ben Nassi
  The traditional learning process of patch-based adversarial attacks,
conducted in the digital domain and then applied in the physical domain (e.g.,
via printed stickers), may suffer from reduced performance due to adversarial
patches' limited transferability from the digital domain to the physical
domain. Given that previous studies have considered using projectors to apply
adversarial attacks, we raise the following question: can adversarial learning
(i.e., patch generation) be performed entirely in the physical domain with a
projector? In this work, we propose the Physical-domain Adversarial Patch
Learning Augmentation (PAPLA) framework, a novel end-to-end (E2E) framework
that converts adversarial learning from the digital domain to the physical
domain using a projector. We evaluate PAPLA across multiple scenarios,
including controlled laboratory settings and realistic outdoor environments,
demonstrating its ability to ensure attack success compared to conventional
digital learning-physical application (DL-PA) methods. We also analyze the
impact of environmental factors, such as projection surface color, projector
strength, ambient light, distance, and angle of the target object relative to
the camera, on the effectiveness of projected patches. Finally, we demonstrate
the feasibility of the attack against a parked car and a stop sign in a
real-world outdoor environment. Our results show that under specific
conditions, E2E adversarial learning in the physical domain eliminates the
transferability issue and ensures evasion by object detectors. Finally, we
provide insights into the challenges and opportunities of applying adversarial
learning in the physical domain and explain where such an approach is more
effective than using a sticker.


http://arxiv.org/abs/2501.08152
Energy Backdoor Attack to Deep Neural Networks. (64%)
Hanene F. Z. Brachemi Meftah; Wassim Hamidouche; Sid Ahmed Fezza; Olivier Déforges; Kassem Kallas
  The rise of deep learning (DL) has increased computing complexity and energy
use, prompting the adoption of application specific integrated circuits (ASICs)
for energy-efficient edge and mobile deployment. However, recent studies have
demonstrated the vulnerability of these accelerators to energy attacks. Despite
the development of various inference time energy attacks in prior research,
backdoor energy attacks remain unexplored. In this paper, we design an
innovative energy backdoor attack against deep neural networks (DNNs) operating
on sparsity-based accelerators. Our attack is carried out in two distinct
phases: backdoor injection and backdoor stealthiness. Experimental results
using ResNet-18 and MobileNet-V2 models trained on CIFAR-10 and Tiny ImageNet
datasets show the effectiveness of our proposed attack in increasing energy
consumption on trigger samples while preserving the model's performance for
clean/regular inputs. This demonstrates the vulnerability of DNNs to energy
backdoor attacks. The source code of our attack is available at:
https://github.com/hbrachemi/energy_backdoor.


http://arxiv.org/abs/2501.07927
Gandalf the Red: Adaptive Security for LLMs. (38%)
Niklas Pfister; Václav Volhejn; Manuel Knott; Santiago Arias; Julia Bazińska; Mykhailo Bichurin; Alan Commike; Janet Darling; Peter Dienes; Matthew Fiedler; David Haber; Matthias Kraft; Marco Lancini; Max Mathys; Damián Pascual-Ortiz; Jakub Podolak; Adrià Romero-López; Kyriacos Shiarlis; Andreas Signer; Zsolt Terek; Athanasios Theocharis; Daniel Timbrell; Samuel Trautwein; Samuel Watts; Yun-Han Wu; Mateo Rojas-Carulla
  Current evaluations of defenses against prompt attacks in large language
model (LLM) applications often overlook two critical factors: the dynamic
nature of adversarial behavior and the usability penalties imposed on
legitimate users by restrictive defenses. We propose D-SEC (Dynamic Security
Utility Threat Model), which explicitly separates attackers from legitimate
users, models multi-step interactions, and expresses the security-utility in an
optimizable form. We further address the shortcomings in existing evaluations
by introducing Gandalf, a crowd-sourced, gamified red-teaming platform designed
to generate realistic, adaptive attack. Using Gandalf, we collect and release a
dataset of 279k prompt attacks. Complemented by benign user data, our analysis
reveals the interplay between security and utility, showing that defenses
integrated in the LLM (e.g., system prompts) can degrade usability even without
blocking requests. We demonstrate that restricted application domains,
defense-in-depth, and adaptive defenses are effective strategies for building
secure and useful LLM applications.


http://arxiv.org/abs/2501.09039
Playing Devil's Advocate: Unmasking Toxicity and Vulnerabilities in Large Vision-Language Models. (2%)
Abdulkadir Erol; Trilok Padhi; Agnik Saha; Ugur Kursuncu; Mehmet Emin Aktas
  The rapid advancement of Large Vision-Language Models (LVLMs) has enhanced
capabilities offering potential applications from content creation to
productivity enhancement. Despite their innovative potential, LVLMs exhibit
vulnerabilities, especially in generating potentially toxic or unsafe
responses. Malicious actors can exploit these vulnerabilities to propagate
toxic content in an automated (or semi-) manner, leveraging the susceptibility
of LVLMs to deception via strategically crafted prompts without fine-tuning or
compute-intensive procedures. Despite the red-teaming efforts and inherent
potential risks associated with the LVLMs, exploring vulnerabilities of LVLMs
remains nascent and yet to be fully addressed in a systematic manner. This
study systematically examines the vulnerabilities of open-source LVLMs,
including LLaVA, InstructBLIP, Fuyu, and Qwen, using adversarial prompt
strategies that simulate real-world social manipulation tactics informed by
social theories. Our findings show that (i) toxicity and insulting are the most
prevalent behaviors, with the mean rates of 16.13% and 9.75%, respectively;
(ii) Qwen-VL-Chat, LLaVA-v1.6-Vicuna-7b, and InstructBLIP-Vicuna-7b are the
most vulnerable models, exhibiting toxic response rates of 21.50%, 18.30% and
17.90%, and insulting responses of 13.40%, 11.70% and 10.10%, respectively;
(iii) prompting strategies incorporating dark humor and multimodal toxic prompt
completion significantly elevated these vulnerabilities. Despite being
fine-tuned for safety, these models still generate content with varying degrees
of toxicity when prompted with adversarial inputs, highlighting the urgent need
for enhanced safety mechanisms and robust guardrails in LVLM development.


http://arxiv.org/abs/2501.07251
MOS-Attack: A Scalable Multi-objective Adversarial Attack Framework. (99%)
Ping Guo; Cheng Gong; Xi Lin; Fei Liu; Zhichao Lu; Qingfu Zhang; Zhenkun Wang
  Crafting adversarial examples is crucial for evaluating and enhancing the
robustness of Deep Neural Networks (DNNs), presenting a challenge equivalent to
maximizing a non-differentiable 0-1 loss function.
  However, existing single objective methods, namely adversarial attacks focus
on a surrogate loss function, do not fully harness the benefits of engaging
multiple loss functions, as a result of insufficient understanding of their
synergistic and conflicting nature.
  To overcome these limitations, we propose the Multi-Objective Set-based
Attack (MOS Attack), a novel adversarial attack framework leveraging multiple
loss functions and automatically uncovering their interrelations.
  The MOS Attack adopts a set-based multi-objective optimization strategy,
enabling the incorporation of numerous loss functions without additional
parameters.
  It also automatically mines synergistic patterns among various losses,
facilitating the generation of potent adversarial attacks with fewer
objectives.
  Extensive experiments have shown that our MOS Attack outperforms
single-objective attacks. Furthermore, by harnessing the identified synergistic
patterns, MOS Attack continues to show superior results with a reduced number
of loss functions.


http://arxiv.org/abs/2501.07493
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards. (89%)
Yangsibo Huang; Milad Nasr; Anastasios Angelopoulos; Nicholas Carlini; Wei-Lin Chiang; Christopher A. Choquette-Choo; Daphne Ippolito; Matthew Jagielski; Katherine Lee; Ken Ziyu Liu; Ion Stoica; Florian Tramer; Chiyuan Zhang
  It is now common to evaluate Large Language Models (LLMs) by having humans
manually vote to evaluate model outputs, in contrast to typical benchmarks that
evaluate knowledge or skill at some particular task. Chatbot Arena, the most
popular benchmark of this type, ranks models by asking users to select the
better response between two randomly selected models (without revealing which
model was responsible for the generations). These platforms are widely trusted
as a fair and accurate measure of LLM capabilities. In this paper, we show that
if bot protection and other defenses are not implemented, these voting-based
benchmarks are potentially vulnerable to adversarial manipulation.
Specifically, we show that an attacker can alter the leaderboard (to promote
their favorite model or demote competitors) at the cost of roughly a thousand
votes (verified in a simulated, offline version of Chatbot Arena). Our attack
consists of two steps: first, we show how an attacker can determine which model
was used to generate a given reply with more than $95\%$ accuracy; and then,
the attacker can use this information to consistently vote for (or against) a
target model. Working with the Chatbot Arena developers, we identify, propose,
and implement mitigations to improve the robustness of Chatbot Arena against
adversarial manipulation, which, based on our analysis, substantially increases
the cost of such attacks. Some of these defenses were present before our
collaboration, such as bot protection with Cloudflare, malicious user
detection, and rate limiting. Others, including reCAPTCHA and login are being
integrated to strengthen the security in Chatbot Arena.


http://arxiv.org/abs/2501.07275
Generating Poisoning Attacks against Ridge Regression Models with Categorical Features. (64%)
Monse Guedes-Ayala; Lars Schewe; Zeynep Suvak; Miguel Anjos
  Machine Learning (ML) models have become a very powerful tool to extract
information from large datasets and use it to make accurate predictions and
automated decisions. However, ML models can be vulnerable to external attacks,
causing them to underperform or deviate from their expected tasks. One way to
attack ML models is by injecting malicious data to mislead the algorithm during
the training phase, which is referred to as a poisoning attack. We can prepare
for such situations by designing anticipated attacks, which are later used for
creating and testing defence strategies. In this paper, we propose an algorithm
to generate strong poisoning attacks for a ridge regression model containing
both numerical and categorical features that explicitly models and poisons
categorical features. We model categorical features as SOS-1 sets and formulate
the problem of designing poisoning attacks as a bilevel optimization problem
that is nonconvex mixed-integer in the upper-level and unconstrained convex
quadratic in the lower-level. We present the mathematical formulation of the
problem, introduce a single-level reformulation based on the Karush-Kuhn-Tucker
(KKT) conditions of the lower level, find bounds for the lower-level variables
to accelerate solver performance, and propose a new algorithm to poison
categorical features. Numerical experiments show that our method improves the
mean squared error of all datasets compared to the previous benchmark in the
literature.


http://arxiv.org/abs/2501.07192
A4O: All Trigger for One sample. (2%)
Duc Anh Vu; Anh Tuan Tran; Cong Tran; Cuong Pham
  Backdoor attacks have become a critical threat to deep neural networks
(DNNs), drawing many research interests. However, most of the studied attacks
employ a single type of trigger. Consequently, proposed backdoor defenders
often rely on the assumption that triggers would appear in a unified way. In
this paper, we show that this naive assumption can create a loophole, allowing
more sophisticated backdoor attacks to bypass. We design a novel backdoor
attack mechanism that incorporates multiple types of backdoor triggers,
focusing on stealthiness and effectiveness. Our journey begins with the
intriguing observation that the performance of a backdoor attack in deep
learning models, as well as its detectability and removability, are all
proportional to the magnitude of the trigger. Based on this correlation, we
propose reducing the magnitude of each trigger type and combining them to
achieve a strong backdoor relying on the combined trigger while still staying
safely under the radar of defenders. Extensive experiments on three standard
datasets demonstrate that our method can achieve high attack success rates
(ASRs) while consistently bypassing state-of-the-art defenses.


http://arxiv.org/abs/2501.07670
A Survey of Early Exit Deep Neural Networks in NLP. (1%)
Divya Jyoti Bajpai; Manjesh Kumar Hanawal
  Deep Neural Networks (DNNs) have grown increasingly large in size to achieve
state of the art performance across a wide range of tasks. However, their high
computational requirements make them less suitable for resource-constrained
applications. Also, real-world datasets often consist of a mixture of easy and
complex samples, necessitating adaptive inference mechanisms that account for
sample difficulty. Early exit strategies offer a promising solution by enabling
adaptive inference, where simpler samples are classified using the initial
layers of the DNN, thereby accelerating the overall inference process. By
attaching classifiers at different layers, early exit methods not only reduce
inference latency but also improve the model robustness against adversarial
attacks. This paper presents a comprehensive survey of early exit methods and
their applications in NLP.


http://arxiv.org/abs/2501.07752
Towards the Pseudorandomness of Expander Random Walks for Read-Once ACC0 circuits. (1%)
Emile Anand
  Expander graphs are among the most useful combinatorial objects in
theoretical computer science. A line of work studies random walks on expander
graphs for their pseudorandomness against various classes of test functions,
including symmetric functions, read-only branching programs, permutation
branching programs, and $\mathrm{AC}^0$ circuits. The promising results of
pseudorandomness of expander random walks against $\mathrm{AC}^0$ circuits
indicate a robustness of expander random walks beyond symmetric functions,
motivating the question of whether expander random walks can fool more robust
\emph{asymmetric} complexity classes, such as $\mathrm{ACC}^0$. In this work,
we make progress towards this question by considering certain two-layered
circuit compositions of $\mathrm{MOD}[k]$ gates, where we show that these
family of circuits are fooled by expander random walks with total variation
distance error $O(\lambda)$, where $\lambda$ is the second largest eigenvalue
of the underlying expander graph. For $k\geq 3$, these circuits can be highly
asymmetric with complicated Fourier characters. In this context, our work takes
a step in the direction of fooling more complex asymmetric circuits.
Separately, drawing from the learning-theory literature, we construct an
explicit threshold circuit in the circuit family $\mathrm{TC}^0$, and show that
it \emph{is} fooled by expander random walks, providing an upper bound on the
set of functions fooled by expander random walks.


http://arxiv.org/abs/2501.07044
Protego: Detecting Adversarial Examples for Vision Transformers via Intrinsic Capabilities. (99%)
Jialin Wu; Kaikai Pan; Yanjiao Chen; Jiangyi Deng; Shengyuan Pang; Wenyuan Xu
  Transformer models have excelled in natural language tasks, prompting the
vision community to explore their implementation in computer vision problems.
However, these models are still influenced by adversarial examples. In this
paper, we investigate the attack capabilities of six common adversarial attacks
on three pretrained ViT models to reveal the vulnerability of ViT models. To
understand and analyse the bias in neural network decisions when the input is
adversarial, we use two visualisation techniques that are attention rollout and
grad attention rollout. To prevent ViT models from adversarial attack, we
propose Protego, a detection framework that leverages the transformer intrinsic
capabilities to detection adversarial examples of ViT models. Nonetheless, this
is challenging due to a diversity of attack strategies that may be adopted by
adversaries. Inspired by the attention mechanism, we know that the token of
prediction contains all the information from the input sample. Additionally,
the attention region for adversarial examples differs from that of normal
examples. Given these points, we can train a detector that achieves superior
performance than existing detection methods to identify adversarial examples.
Our experiments have demonstrated the high effectiveness of our detection
method. For these six adversarial attack methods, our detector's AUC scores all
exceed 0.95. Protego may advance investigations in metaverse security.


http://arxiv.org/abs/2501.06729
KeTS: Kernel-based Trust Segmentation against Model Poisoning Attacks. (98%)
Ankit Gangwal; Mauro Conti; Tommaso Pauselli
  Federated Learning (FL) enables multiple users to collaboratively train a
global model in a distributed manner without revealing their personal data.
However, FL remains vulnerable to model poisoning attacks, where malicious
actors inject crafted updates to compromise the global model's accuracy. These
vulnerabilities are particularly severe in non-homogeneous environments, where
clients exhibit varying proportions of class labels, resulting in heterogeneous
updates. In such settings, benign outliers are often misclassified as false
positives, while maliciously crafted uploads evade detection and are aggregated
at the server. Existing defense mechanisms struggle in such real-world
settings, resulting in significant declines in the global FL model's
performance.
  We propose a novel defense mechanism, Kernel-based Trust Segmentation (KeTS),
to counter model poisoning attacks. Unlike existing approaches, KeTS analyzes
the evolution of each client's updates and effectively segments malicious
clients using Kernel Density Estimation (KDE), even in the presence of benign
outliers. We thoroughly evaluate KeTS's performance against the six most
effective model poisoning attacks (i.e., Trim-Attack, Krum-Attack, Min-Max
attack, Min-Sum attack, and their variants) on two different datasets (i.e.,
MNIST and Fashion-MNIST) and compare its performance with three classical
robust schemes (i.e., Krum, Trim-Mean, and Median) and a state-of-the-art
defense (i.e., FLTrust). Our results show that KeTS outperforms the existing
defenses in every attack setting; beating the best-performing defense by an
overall average of >24% (on MNIST) and >14% (on Fashion-MNIST). A series of
further experiments (varying poisoning approaches, attacker population, etc.)
reveal the consistent and superior performance of KeTS under diverse
conditions.


http://arxiv.org/abs/2501.06736
ZOQO: Zero-Order Quantized Optimization. (1%)
Noga Bar; Raja Giryes
  The increasing computational and memory demands in deep learning present
significant challenges, especially in resource-constrained environments. We
introduce a zero-order quantized optimization (ZOQO) method designed for
training models with quantized parameters and operations. Our approach
leverages zero-order approximations of the gradient sign and adapts the
learning process to maintain the parameters' quantization without the need for
full-precision gradient calculations. We demonstrate the effectiveness of ZOQO
through experiments in fine-tuning of large language models and black-box
adversarial attacks. Despite the limitations of zero-order and quantized
operations training, our method achieves competitive performance compared to
full-precision methods, highlighting its potential for low-resource
environments.


http://arxiv.org/abs/2501.06646
RogueRFM: Attacking Refresh Management for Covert-Channel and Denial-of-Service. (10%)
Hritvik Taneja; Moinuddin Qureshi
  With lowering thresholds, transparently defending against Rowhammer within
DRAM is challenging due to the lack of time to perform mitigation. Commercially
deployed in-DRAM defenses like TRR that steal time from normal refreshes~(REF)
to perform mitigation have been proven ineffective against Rowhammer. In
response, a new Refresh Management (RFM) interface has been added to the DDR5
specifications. RFM provides dedicated time to an in-DRAM defense to perform
mitigation. Several recent works have used RFM for the intended purpose -
building better Rowhammer defenses. However, to the best of our knowledge, no
prior study has looked at the potential security implications of this new
feature if an attacker subjects it to intentional misuse.
  Our paper shows that RFM introduces new side effects in the system - the
activity of one bank causes interference with the operation of the other banks.
Thus, the latency of a bank becomes dependent on the activity of other banks.
We use these side effects to build two new attacks. First, a novel memory-based
covert channel, which has a bandwidth of up to 31.3 KB/s, and is also effective
even in a bank-partitioned system. Second, a new Denial-of-Service (DOS) attack
pattern that exploits the activity within a single bank to reduce the
performance of the other banks. Our experiments on SPEC2017, PARSEC, and LIGRA
workloads show a slowdown of up to 67\% when running alongside our DOS pattern.
We also discuss potential countermeasures for our attacks.


http://arxiv.org/abs/2501.06650
SafeSplit: A Novel Defense Against Client-Side Backdoor Attacks in Split Learning. (10%)
Phillip Rieger; Alessandro Pegoraro; Kavita Kumari; Tigist Abera; Jonathan Knauer; Ahmad-Reza Sadeghi
  Split Learning (SL) is a distributed deep learning approach enabling multiple
clients and a server to collaboratively train and infer on a shared deep neural
network (DNN) without requiring clients to share their private local data. The
DNN is partitioned in SL, with most layers residing on the server and a few
initial layers and inputs on the client side. This configuration allows
resource-constrained clients to participate in training and inference. However,
the distributed architecture exposes SL to backdoor attacks, where malicious
clients can manipulate local datasets to alter the DNN's behavior. Existing
defenses from other distributed frameworks like Federated Learning are not
applicable, and there is a lack of effective backdoor defenses specifically
designed for SL.
  We present SafeSplit, the first defense against client-side backdoor attacks
in Split Learning (SL). SafeSplit enables the server to detect and filter out
malicious client behavior by employing circular backward analysis after a
client's training is completed, iteratively reverting to a trained checkpoint
where the model under examination is found to be benign. It uses a two-fold
analysis to identify client-induced changes and detect poisoned models. First,
a static analysis in the frequency domain measures the differences in the
layer's parameters at the server. Second, a dynamic analysis introduces a novel
rotational distance metric that assesses the orientation shifts of the server's
layer parameters during training. Our comprehensive evaluation across various
data distributions, client counts, and attack scenarios demonstrates the high
efficacy of this dual analysis in mitigating backdoor attacks while preserving
model utility.


http://arxiv.org/abs/2501.05962
Effective faking of verbal deception detection with target-aligned adversarial attacks. (87%)
Bennett Kleinberg; Riccardo Loconte; Bruno Verschuere
  Background: Deception detection through analysing language is a promising
avenue using both human judgments and automated machine learning judgments. For
both forms of credibility assessment, automated adversarial attacks that
rewrite deceptive statements to appear truthful pose a serious threat. Methods:
We used a dataset of 243 truthful and 262 fabricated autobiographical stories
in a deception detection task for humans and machine learning models. A large
language model was tasked to rewrite deceptive statements so that they appear
truthful. In Study 1, humans who made a deception judgment or used the
detailedness heuristic and two machine learning models (a fine-tuned language
model and a simple n-gram model) judged original or adversarial modifications
of deceptive statements. In Study 2, we manipulated the target alignment of the
modifications, i.e. tailoring the attack to whether the statements would be
assessed by humans or computer models. Results: When adversarial modifications
were aligned with their target, human (d=-0.07 and d=-0.04) and machine
judgments (51% accuracy) dropped to the chance level. When the attack was not
aligned with the target, both human heuristics judgments (d=0.30 and d=0.36)
and machine learning predictions (63-78%) were significantly better than
chance. Conclusions: Easily accessible language models can effectively help
anyone fake deception detection efforts both by humans and machine learning
models. Robustness against adversarial modifications for humans and machines
depends on that target alignment. We close with suggestions on advancing
deception research with adversarial attack designs and techniques.


http://arxiv.org/abs/2501.05783
UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mapping. (81%)
Yanjie Li; Wenxuan Zhang; Kaisheng Liang; Bin Xiao
  In recent research, adversarial attacks on person detectors using patches or
static 3D model-based texture modifications have struggled with low success
rates due to the flexible nature of human movement. Modeling the 3D
deformations caused by various actions has been a major challenge. Fortunately,
advancements in Neural Radiance Fields (NeRF) for dynamic human modeling offer
new possibilities. In this paper, we introduce UV-Attack, a groundbreaking
approach that achieves high success rates even with extensive and unseen human
actions. We address the challenge above by leveraging dynamic-NeRF-based UV
mapping. UV-Attack can generate human images across diverse actions and
viewpoints, and even create novel actions by sampling from the SMPL parameter
space. While dynamic NeRF models are capable of modeling human bodies,
modifying clothing textures is challenging because they are embedded in neural
network parameters. To tackle this, UV-Attack generates UV maps instead of RGB
images and modifies the texture stacks. This approach enables real-time texture
edits and makes the attack more practical. We also propose a novel Expectation
over Pose Transformation loss (EoPT) to improve the evasion success rate on
unseen poses and views. Our experiments show that UV-Attack achieves a 92.75%
attack success rate against the FastRCNN model across varied poses in dynamic
video settings, significantly outperforming the state-of-the-art AdvCamou
attack, which only had a 28.50% ASR. Moreover, we achieve 49.5% ASR on the
latest YOLOv8 detector in black-box settings. This work highlights the
potential of dynamic NeRF-based UV mapping for creating more effective
adversarial attacks on person detectors, addressing key challenges in modeling
human movement and texture modification.


http://arxiv.org/abs/2501.05835
Fine-tuning is Not Fine: Mitigating Backdoor Attacks in GNNs with Limited Clean Data. (80%)
Jiale Zhang; Bosen Rao; Chengcheng Zhu; Xiaobing Sun; Qingming Li; Haibo Hu; Xiapu Luo; Qingqing Ye; Shouling Ji
  Graph Neural Networks (GNNs) have achieved remarkable performance through
their message-passing mechanism. However, recent studies have highlighted the
vulnerability of GNNs to backdoor attacks, which can lead the model to
misclassify graphs with attached triggers as the target class. The
effectiveness of recent promising defense techniques, such as fine-tuning or
distillation, is heavily contingent on having comprehensive knowledge of the
sufficient training dataset. Empirical studies have shown that fine-tuning
methods require a clean dataset of 20% to reduce attack accuracy to below 25%,
while distillation methods require a clean dataset of 15%. However, obtaining
such a large amount of clean data is commonly impractical.
  In this paper, we propose a practical backdoor mitigation framework, denoted
as GRAPHNAD, which can capture high-quality intermediate-layer representations
in GNNs to enhance the distillation process with limited clean data. To achieve
this, we address the following key questions: How to identify the appropriate
attention representations in graphs for distillation? How to enhance
distillation with limited data? By adopting the graph attention transfer
method, GRAPHNAD can effectively align the intermediate-layer attention
representations of the backdoored model with that of the teacher model, forcing
the backdoor neurons to transform into benign ones. Besides, we extract the
relation maps from intermediate-layer transformation and enforce the relation
maps of the backdoored model to be consistent with that of the teacher model,
thereby ensuring model accuracy while further reducing the influence of
backdoors. Extensive experimental results show that by fine-tuning a teacher
model with only 3% of the clean data, GRAPHNAD can reduce the attack success
rate to below 5%.


http://arxiv.org/abs/2501.05928
Towards Backdoor Stealthiness in Model Parameter Space. (61%)
Xiaoyun Xu; Zhuoran Liu; Stefanos Koffas; Stjepan Picek
  Recent research on backdoor stealthiness focuses mainly on indistinguishable
triggers in input space and inseparable backdoor representations in feature
space, aiming to circumvent backdoor defenses that examine these respective
spaces. However, existing backdoor attacks are typically designed to resist a
specific type of backdoor defense without considering the diverse range of
defense mechanisms. Based on this observation, we pose a natural question: Are
current backdoor attacks truly a real-world threat when facing diverse
practical defenses?
  To answer this question, we examine 12 common backdoor attacks that focus on
input-space or feature-space stealthiness and 17 diverse representative
defenses. Surprisingly, we reveal a critical blind spot: Backdoor attacks
designed to be stealthy in input and feature spaces can be mitigated by
examining backdoored models in parameter space. To investigate the underlying
causes behind this common vulnerability, we study the characteristics of
backdoor attacks in the parameter space. Notably, we find that input- and
feature-space attacks introduce prominent backdoor-related neurons in parameter
space, which are not thoroughly considered by current backdoor attacks. Taking
comprehensive stealthiness into account, we propose a novel supply-chain attack
called Grond. Grond limits the parameter changes by a simple yet effective
module, Adversarial Backdoor Injection (ABI), which adaptively increases the
parameter-space stealthiness during the backdoor injection. Extensive
experiments demonstrate that Grond outperforms all 12 backdoor attacks against
state-of-the-art (including adaptive) defenses on CIFAR-10, GTSRB, and a subset
of ImageNet. In addition, we show that ABI consistently improves the
effectiveness of common backdoor attacks.


http://arxiv.org/abs/2501.05991
An Attention-Guided Deep Learning Approach for Classifying 39 Skin Lesion Types. (1%)
Sauda Adiv Hanum; Ashim Dey; Muhammad Ashad Kabir
  The skin, as the largest organ of the human body, is vulnerable to a diverse
array of conditions collectively known as skin lesions, which encompass various
dermatoses. Diagnosing these lesions presents significant challenges for
medical practitioners due to the subtle visual differences that are often
imperceptible to the naked eye. While not all skin lesions are
life-threatening, certain types can act as early indicators of severe diseases,
including skin cancers, underscoring the critical need for timely and accurate
diagnostic methods. Deep learning algorithms have demonstrated remarkable
potential in facilitating the early detection and prognosis of skin lesions.
This study advances the field by curating a comprehensive and diverse dataset
comprising 39 categories of skin lesions, synthesized from five publicly
available datasets. Using this dataset, the performance of five
state-of-the-art deep learning models -- MobileNetV2, Xception, InceptionV3,
EfficientNetB1, and Vision Transformer - is rigorously evaluated. To enhance
the accuracy and robustness of these models, attention mechanisms such as the
Efficient Channel Attention (ECA) and the Convolutional Block Attention Module
(CBAM) are incorporated into their architectures. Comprehensive evaluation
across multiple performance metrics reveals that the Vision Transformer model
integrated with CBAM outperforms others, achieving an accuracy of 93.46%,
precision of 94%, recall of 93%, F1-score of 93%, and specificity of 93.67%.
These results underscore the significant potential of the proposed system in
supporting medical professionals with accurate and efficient prognostic tools
for diagnosing a broad spectrum of skin lesions. The dataset and code used in
this study can be found at
https://github.com/akabircs/Skin-Lesions-Classification.


http://arxiv.org/abs/2501.06264
HPAC-IDS: A Hierarchical Packet Attention Convolution for Intrusion Detection System. (99%)
Anass Grini; Btissam El Khamlichi; Abdellatif El Afia; Amal El Fallah-Seghrouchni
  This research introduces a robust detection system against malicious network
traffic, leveraging hierarchical structures and self-attention mechanisms. The
proposed system includes a Packet Segmenter that divides a given raw network
packet into fixed-size segments that are fed to the HPAC-IDS. The experiments
performed on CIC-IDS2017 dataset show that the system exhibits high accuracy
and low false positive rates while demonstrating resilience against diverse
adversarial methods like Fast Gradient Sign Method (FGSM), Projected Gradient
Descent (PGD), and Wasserstein GAN (WGAN). The model's ability to withstand
adversarial perturbations is attributed to the fusion of hierarchical attention
mechanisms and convolutional neural networks, resulting in a 0% to 10%
adversarial attack severity under tested adversarial attacks with different
segment sizes, surpassing the state-of-the-art model in detection performance
and adversarial attack robustness.


http://arxiv.org/abs/2501.05127
DiffAttack: Diffusion-based Timbre-reserved Adversarial Attack in Speaker Identification. (97%)
Qing Wang; Jixun Yao; Zhaokai Sun; Pengcheng Guo; Lei Xie; John H. L. Hansen
  Being a form of biometric identification, the security of the speaker
identification (SID) system is of utmost importance. To better understand the
robustness of SID systems, we aim to perform more realistic attacks in SID,
which are challenging for both humans and machines to detect. In this study, we
propose DiffAttack, a novel timbre-reserved adversarial attack approach that
exploits the capability of a diffusion-based voice conversion (DiffVC) model to
generate adversarial fake audio with distinct target speaker attribution. By
introducing adversarial constraints into the generative process of the
diffusion-based voice conversion model, we craft fake samples that effectively
mislead target models while preserving speaker-wise characteristics.
Specifically, inspired by the use of randomly sampled Gaussian noise in
conventional adversarial attacks and diffusion processes, we incorporate
adversarial constraints into the reverse diffusion process. These constraints
subtly guide the reverse diffusion process toward aligning with the target
speaker distribution. Our experiments on the LibriTTS dataset indicate that
DiffAttack significantly improves the attack success rate compared to vanilla
DiffVC and other methods. Moreover, objective and subjective evaluations
demonstrate that introducing adversarial constraints does not compromise the
speech quality generated by the DiffVC model.


http://arxiv.org/abs/2501.05588
Enforcing Fundamental Relations via Adversarial Attacks on Input Parameter Correlations. (92%)
Timo Saala; Lucie Flek; Alexander Jung; Akbar Karimi; Alexander Schmidt; Matthias Schott; Philipp Soldin; Christopher Wiebusch
  Correlations between input parameters play a crucial role in many scientific
classification tasks, since these are often related to fundamental laws of
nature. For example, in high energy physics, one of the common deep learning
use-cases is the classification of signal and background processes in particle
collisions. In many such cases, the fundamental principles of the correlations
between observables are often better understood than the actual distributions
of the observables themselves. In this work, we present a new adversarial
attack algorithm called Random Distribution Shuffle Attack (RDSA), emphasizing
the correlations between observables in the network rather than individual
feature characteristics. Correct application of the proposed novel attack can
result in a significant improvement in classification performance -
particularly in the context of data augmentation - when using the generated
adversaries within adversarial training. Given that correlations between input
features are also crucial in many other disciplines. We demonstrate the RDSA
effectiveness on six classification tasks, including two particle collision
challenges (using CERN Open Data), hand-written digit recognition (MNIST784),
human activity recognition (HAR), weather forecasting (Rain in Australia), and
ICU patient mortality (MIMIC-IV), demonstrating a general use case beyond
fundamental physics for this new type of adversarial attack algorithms.


http://arxiv.org/abs/2501.04985
SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs. (50%)
Muhammad Salman; Muhammad Ikram; Nardine Basta; Mohamed Ali Kaafar
  The increasing threat of SMS spam, driven by evolving adversarial techniques
and concept drift, calls for more robust and adaptive detection methods. In
this paper, we evaluate the potential of large language models (LLMs), both
open-source and commercial, for SMS spam detection, comparing their performance
across zero-shot, few-shot, fine-tuning, and chain-of-thought prompting
approaches. Using a comprehensive dataset of SMS messages, we assess the spam
detection capabilities of prominent LLMs such as GPT-4, DeepSeek, LLAMA-2, and
Mixtral. Our findings reveal that while zero-shot learning provides
convenience, it is unreliable for effective spam detection. Few-shot learning,
particularly with carefully selected examples, improves detection but exhibits
variability across models. Fine-tuning emerges as the most effective strategy,
with Mixtral achieving 98.6% accuracy and a balanced false positive and false
negative rate below 2%, meeting the criteria for robust spam detection.
Furthermore, we explore the resilience of these models to adversarial attacks,
finding that fine-tuning significantly enhances robustness against both
perceptible and imperceptible manipulations. Lastly, we investigate the impact
of concept drift and demonstrate that fine-tuned LLMs, especially when combined
with few-shot learning, can mitigate its effects, maintaining high performance
even on evolving spam datasets. This study highlights the importance of
fine-tuning and tailored learning strategies to deploy LLMs effectively for
real-world SMS spam detection


http://arxiv.org/abs/2501.05359
SC-Pro: Training-Free Framework for Defending Unsafe Image Synthesis Attack. (38%)
Junha Park; Jaehui Hwang; Ian Ryu; Hyungkeun Park; Jiyoon Kim; Jong-Seok Lee
  With advances in diffusion models, image generation has shown significant
performance improvements. This raises concerns about the potential abuse of
image generation, such as the creation of explicit or violent images, commonly
referred to as Not Safe For Work (NSFW) content. To address this, the Stable
Diffusion model includes several safety checkers to censor initial text prompts
and final output images generated from the model. However, recent research has
shown that these safety checkers have vulnerabilities against adversarial
attacks, allowing them to generate NSFW images. In this paper, we find that
these adversarial attacks are not robust to small changes in text prompts or
input latents. Based on this, we propose SC-Pro (Spherical or Circular
Probing), a training-free framework that easily defends against adversarial
attacks generating NSFW images. Moreover, we develop an approach that utilizes
one-step diffusion models for efficient NSFW detection (SC-Pro-o), further
reducing computational resources. We demonstrate the superiority of our method
in terms of performance and applicability.


http://arxiv.org/abs/2501.05015
On Measuring Unnoticeability of Graph Adversarial Attacks: Observations, New Measure, and Applications. (33%)
Hyeonsoo Jo; Hyunjin Hwang; Fanchen Bu; Soo Yong Lee; Chanyoung Park; Kijung Shin
  Adversarial attacks are allegedly unnoticeable. Prior studies have designed
attack noticeability measures on graphs, primarily using statistical tests to
compare the topology of original and (possibly) attacked graphs. However, we
observe two critical limitations in the existing measures. First, because the
measures rely on simple rules, attackers can readily enhance their attacks to
bypass them, reducing their attack "noticeability" and, yet, maintaining their
attack performance. Second, because the measures naively leverage global
statistics, such as degree distributions, they may entirely overlook attacks
until severe perturbations occur, letting the attacks be almost "totally
unnoticeable." To address the limitations, we introduce HideNSeek, a learnable
measure for graph attack noticeability. First, to mitigate the bypass problem,
HideNSeek learns to distinguish the original and (potential) attack edges using
a learnable edge scorer (LEO), which scores each edge on its likelihood of
being an attack. Second, to mitigate the overlooking problem, HideNSeek
conducts imbalance-aware aggregation of all the edge scores to obtain the final
noticeability score. Using six real-world graphs, we empirically demonstrate
that HideNSeek effectively alleviates the observed limitations, and LEO (i.e.,
our learnable edge scorer) outperforms eleven competitors in distinguishing
attack edges under five different attack methods. For an additional
application, we show that LEO boost the performance of robust GNNs by removing
attack-like edges.


http://arxiv.org/abs/2501.05168
KabaddiPy: A package to enable access to Professional Kabaddi Data. (1%)
Bhaskar Lalwani; Aniruddha Mukherjee
  Kabaddi, a contact team sport of Indian origin, has seen a dramatic rise in
global popularity, highlighted by the upcoming Kabaddi World Cup in 2025 with
over sixteen international teams participating, alongside flourishing national
leagues such as the Indian Pro Kabaddi League (230 million viewers) and the
British Kabaddi League. We present the first open-source Python module to make
Kabaddi statistical data easily accessible from multiple scattered sources
across the internet. The module was developed by systematically web-scraping
and collecting team-wise, player-wise and match-by-match data. The data has
been cleaned, organized, and categorized into team overviews and player
metrics, each filterable by season. The players are classified as raiders and
defenders, with their best strategies for attacking, counter-attacking, and
defending against different teams highlighted. Our module enables continuous
monitoring of exponentially growing data streams, aiding researchers to quickly
start building upon the data to answer critical questions, such as the impact
of player inclusion/exclusion on team performance, scoring patterns against
specific teams, and break down opponent gameplay. The data generated from
Kabaddi tournaments has been sparsely used, and coaches and players rely
heavily on intuition to make decisions and craft strategies. Our module can be
utilized to build predictive models, craft uniquely strategic gameplays to
target opponents and identify hidden correlations in the data. This open source
module has the potential to increase time-efficiency, encourage analytical
studies of Kabaddi gameplay and player dynamics and foster reproducible
research. The data and code are publicly available:
https://github.com/kabaddiPy/kabaddiPy


http://arxiv.org/abs/2501.05239
Is Your Autonomous Vehicle Safe? Understanding the Threat of Electromagnetic Signal Injection Attacks on Traffic Scene Perception. (1%)
Wenhao Liao; Sineng Yan; Youqian Zhang; Xinwei Zhai; Yuanyuan Wang; Eugene Yujun Fu
  Autonomous vehicles rely on camera-based perception systems to comprehend
their driving environment and make crucial decisions, thereby ensuring vehicles
to steer safely. However, a significant threat known as Electromagnetic Signal
Injection Attacks (ESIA) can distort the images captured by these cameras,
leading to incorrect AI decisions and potentially compromising the safety of
autonomous vehicles. Despite the serious implications of ESIA, there is limited
understanding of its impacts on the robustness of AI models across various and
complex driving scenarios. To address this gap, our research analyzes the
performance of different models under ESIA, revealing their vulnerabilities to
the attacks. Moreover, due to the challenges in obtaining real-world attack
data, we develop a novel ESIA simulation method and generate a simulated attack
dataset for different driving scenarios. Our research provides a comprehensive
simulation and evaluation framework, aiming to enhance the development of more
robust AI models and secure intelligent systems, ultimately contributing to the
advancement of safer and more reliable technology across various fields.


http://arxiv.org/abs/2501.05249
RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language Models. (1%)
Peizhuo Lv; Mengjie Sun; Hao Wang; Xiaofeng Wang; Shengzhi Zhang; Yuxuan Chen; Kai Chen; Limin Sun
  In recent years, tremendous success has been witnessed in Retrieval-Augmented
Generation (RAG), widely used to enhance Large Language Models (LLMs) in
domain-specific, knowledge-intensive, and privacy-sensitive tasks. However,
attackers may steal those valuable RAGs and deploy or commercialize them,
making it essential to detect Intellectual Property (IP) infringement. Most
existing ownership protection solutions, such as watermarks, are designed for
relational databases and texts. They cannot be directly applied to RAGs because
relational database watermarks require white-box access to detect IP
infringement, which is unrealistic for the knowledge base in RAGs. Meanwhile,
post-processing by the adversary's deployed LLMs typically destructs text
watermark information. To address those problems, we propose a novel black-box
"knowledge watermark" approach, named RAG-WM, to detect IP infringement of
RAGs. RAG-WM uses a multi-LLM interaction framework, comprising a Watermark
Generator, Shadow LLM & RAG, and Watermark Discriminator, to create watermark
texts based on watermark entity-relationship tuples and inject them into the
target RAG. We evaluate RAG-WM across three domain-specific and two
privacy-sensitive tasks on four benchmark LLMs. Experimental results show that
RAG-WM effectively detects the stolen RAGs in various deployed LLMs.
Furthermore, RAG-WM is robust against paraphrasing, unrelated content removal,
knowledge insertion, and knowledge expansion attacks. Lastly, RAG-WM can also
evade watermark detection approaches, highlighting its promising application in
detecting IP infringement of RAG systems.


http://arxiv.org/abs/2501.04861
LayerMix: Enhanced Data Augmentation through Fractal Integration for Robust Deep Learning. (86%)
Hafiz Mughees Ahmad; Dario Morle; Afshin Rahimi
  Deep learning models have demonstrated remarkable performance across various
computer vision tasks, yet their vulnerability to distribution shifts remains a
critical challenge. Despite sophisticated neural network architectures,
existing models often struggle to maintain consistent performance when
confronted with Out-of-Distribution (OOD) samples, including natural
corruptions, adversarial perturbations, and anomalous patterns. We introduce
LayerMix, an innovative data augmentation approach that systematically enhances
model robustness through structured fractal-based image synthesis. By
meticulously integrating structural complexity into training datasets, our
method generates semantically consistent synthetic samples that significantly
improve neural network generalization capabilities. Unlike traditional
augmentation techniques that rely on random transformations, LayerMix employs a
structured mixing pipeline that preserves original image semantics while
introducing controlled variability. Extensive experiments across multiple
benchmark datasets, including CIFAR-10, CIFAR-100, ImageNet-200, and
ImageNet-1K demonstrate LayerMixs superior performance in classification
accuracy and substantially enhances critical Machine Learning (ML) safety
metrics, including resilience to natural image corruptions, robustness against
adversarial attacks, improved model calibration and enhanced prediction
consistency. LayerMix represents a significant advancement toward developing
more reliable and adaptable artificial intelligence systems by addressing the
fundamental challenges of deep learning generalization. The code is available
at https://github.com/ahmadmughees/layermix.


http://arxiv.org/abs/2501.04802
Reproducing HotFlip for Corpus Poisoning Attacks in Dense Retrieval. (84%)
Yongkang Li; Panagiotis Eustratiadis; Evangelos Kanoulas
  HotFlip is a topical gradient-based word substitution method for attacking
language models. Recently, this method has been further applied to attack
retrieval systems by generating malicious passages that are injected into a
corpus, i.e., corpus poisoning. However, HotFlip is known to be computationally
inefficient, with the majority of time being spent on gradient accumulation for
each query-passage pair during the adversarial token generation phase, making
it impossible to generate an adequate number of adversarial passages in a
reasonable amount of time. Moreover, the attack method itself assumes access to
a set of user queries, a strong assumption that does not correspond to how
real-world adversarial attacks are usually performed. In this paper, we first
significantly boost the efficiency of HotFlip, reducing the adversarial
generation process from 4 hours per document to only 15 minutes, using the same
hardware. We further contribute experiments and analysis on two additional
tasks: (1) transfer-based black-box attacks, and (2) query-agnostic attacks.
Whenever possible, we provide comparisons between the original method and our
improved version. Our experiments demonstrate that HotFlip can effectively
attack a variety of dense retrievers, with an observed trend that its attack
performance diminishes against more advanced and recent methods. Interestingly,
we observe that while HotFlip performs poorly in a black-box setting,
indicating limited capacity for generalization, in query-agnostic scenarios its
performance is correlated to the volume of injected adversarial passages.


http://arxiv.org/abs/2501.04453
Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learning. (78%)
Bin Li; Xiaoye Miao; Yongheng Shang; Xinkui Zhao; Shuiguang Deng; Jianwei Yin
  Decentralized federated learning (DFL) is inherently vulnerable to poisoning
attacks, as malicious clients can transmit manipulated model gradients to
neighboring clients. Existing defense methods either reject suspicious
gradients per iteration or restart DFL aggregation after detecting all
malicious clients. They overlook the potential accuracy benefit from the
discarded malicious gradients. In this paper, we propose a novel gradient
purification defense, named GPD, that integrates seamlessly with existing DFL
aggregation to defend against poisoning attacks. It aims to mitigate the harm
in model gradients while retaining the benefit in model weights for enhancing
accuracy. For each benign client in GPD, a recording variable is designed to
track the historically aggregated gradients from one of its neighbors. It
allows benign clients to precisely detect malicious neighbors and swiftly
mitigate aggregated malicious gradients via historical consistency checks. Upon
mitigation, GPD optimizes model weights via aggregating gradients solely from
benign clients. This retains the previously beneficial portions from malicious
clients and exploits the contributions from benign clients, thereby
significantly enhancing the model accuracy. We analyze the convergence of GPD,
as well as its ability to harvest high accuracy. Extensive experiments over
three datasets demonstrate that, GPD is capable of mitigating poisoning attacks
under both iid and non-iid data distributions. It significantly outperforms
state-of-the-art defenses in terms of accuracy against various poisoning
attacks.


http://arxiv.org/abs/2501.04931
Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency. (38%)
Shiji Zhao; Ranjie Duan; Fengxiang Wang; Chi Chen; Caixin Kang; Jialing Tao; YueFeng Chen; Hui Xue; Xingxing Wei
  Multimodal Large Language Models (MLLMs) have achieved impressive performance
and have been put into practical use in commercial applications, but they still
have potential safety mechanism vulnerabilities. Jailbreak attacks are red
teaming methods that aim to bypass safety mechanisms and discover MLLMs'
potential risks. Existing MLLMs' jailbreak methods often bypass the model's
safety mechanism through complex optimization methods or carefully designed
image and text prompts. Despite achieving some progress, they have a low attack
success rate on commercial closed-source MLLMs. Unlike previous research, we
empirically find that there exists a Shuffle Inconsistency between MLLMs'
comprehension ability and safety ability for the shuffled harmful instruction.
That is, from the perspective of comprehension ability, MLLMs can understand
the shuffled harmful text-image instructions well. However, they can be easily
bypassed by the shuffled harmful instructions from the perspective of safety
ability, leading to harmful responses. Then we innovatively propose a
text-image jailbreak attack named SI-Attack. Specifically, to fully utilize the
Shuffle Inconsistency and overcome the shuffle randomness, we apply a
query-based black-box optimization method to select the most harmful shuffled
inputs based on the feedback of the toxic judge model. A series of experiments
show that SI-Attack can improve the attack's performance on three benchmarks.
In particular, SI-Attack can obviously improve the attack success rate for
commercial MLLMs such as GPT-4o or Claude-3.5-Sonnet.


http://arxiv.org/abs/2501.04527
Towards Fair Class-wise Robustness: Class Optimal Distribution Adversarial Training. (26%)
Hongxin Zhi; Hongtao Yu; Shaome Li; Xiuming Zhao; Yiteng Wu
  Adversarial training has proven to be a highly effective method for improving
the robustness of deep neural networks against adversarial attacks.
Nonetheless, it has been observed to exhibit a limitation in terms of robust
fairness, characterized by a significant disparity in robustness across
different classes. Recent efforts to mitigate this problem have turned to
class-wise reweighted methods. However, these methods suffer from a lack of
rigorous theoretical analysis and are limited in their exploration of the
weight space, as they mainly rely on existing heuristic algorithms or intuition
to compute weights. In addition, these methods fail to guarantee the
consistency of the optimization direction due to the decoupled optimization of
weights and the model parameters. They potentially lead to suboptimal weight
assignments and consequently, a suboptimal model. To address these problems,
this paper proposes a novel min-max training framework, Class Optimal
Distribution Adversarial Training (CODAT), which employs distributionally
robust optimization to fully explore the class-wise weight space, thus enabling
the identification of the optimal weight with theoretical guarantees.
Furthermore, we derive a closed-form optimal solution to the internal
maximization and then get a deterministic equivalent objective function, which
provides a theoretical basis for the joint optimization of weights and model
parameters. Meanwhile, we propose a fairness elasticity coefficient for the
evaluation of the algorithm with regard to both robustness and robust fairness.
Experimental results on various datasets show that the proposed method can
effectively improve the robust fairness of the model and outperform the
state-of-the-art approaches.


http://arxiv.org/abs/2501.03562
Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective. (81%)
Tianyang Duan; Zongyuan Zhang; Zheng Lin; Yue Gao; Ling Xiong; Yong Cui; Hongbin Liang; Xianhao Chen; Heming Cui; Dong Huang
  Deep Reinforcement Learning (DRL) suffers from uncertainties and inaccuracies
in the observation signal in realworld applications. Adversarial attack is an
effective method for evaluating the robustness of DRL agents. However, existing
attack methods targeting individual sampled actions have limited impacts on the
overall policy distribution, particularly in continuous action spaces. To
address these limitations, we propose the Distribution-Aware Projected Gradient
Descent attack (DAPGD). DAPGD uses distribution similarity as the gradient
perturbation input to attack the policy network, which leverages the entire
policy distribution rather than relying on individual samples. We utilize the
Bhattacharyya distance in DAPGD to measure policy similarity, enabling
sensitive detection of subtle but critical differences between probability
distributions. Our experiment results demonstrate that DAPGD achieves SOTA
results compared to the baselines in three robot navigation tasks, achieving an
average 22.03% higher reward drop compared to the best baseline.


http://arxiv.org/abs/2501.02968
FlipedRAG: Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Models. (99%)
Zhuo Chen; Yuyang Gong; Miaokun Chen; Haotan Liu; Qikai Cheng; Fan Zhang; Wei Lu; Xiaozhong Liu; Jiawei Liu
  Retrieval-Augmented Generation (RAG) addresses hallucination and real-time
constraints by dynamically retrieving relevant information from a knowledge
database to supplement the LLMs' input. When presented with a query, RAG
selects the most semantically similar texts from its knowledge bases and uses
them as context for the LLMs to generate more accurate responses. RAG also
creates a new attack surface, especially since RAG databases are frequently
sourced from public domains. While existing studies have predominantly focused
on optimizing RAG's performance and efficiency, emerging research has begun
addressing the security concerns associated with RAG. However, these works have
some limitations, typically focusing on either white-box methodologies or
heuristic-based black-box attacks. Furthermore, prior research has mainly
targeted simple factoid question answering, which is neither practically
challenging nor resistant to correction. In this paper, we unveil a more
realistic and threatening scenario: opinion manipulation for controversial
topics against RAG. Particularly, we propose a novel RAG black-box attack
method, termed FlipedRAG, which is transfer-based. By leveraging instruction
engineering, we obtain partial retrieval model outputs from black-box RAG
system, facilitating the training of surrogate models to enhance the
effectiveness of opinion manipulation attack. Extensive experimental results
confirms that our approach significantly enhances the average success rate of
opinion manipulation by 16.7%. It achieves an average of a 50% directional
change in the opinion polarity of RAG responses across four themes.
Additionally, it induces a 20% shift in user cognition. Furthermore, we discuss
the efficacy of potential defense mechanisms and conclude that they are
insufficient in mitigating this type of attack, highlighting the urgent need to
develop novel defensive strategies.


http://arxiv.org/abs/2501.03507
An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning. (3%)
Fatemeh Ghofrani; Pooyan Jamshidi
  Self-supervised learning (SSL) has significantly advanced image
representation learning, yet efficiency challenges persist, particularly with
adversarial training. Many SSL methods require extensive epochs to achieve
convergence, a demand further amplified in adversarial settings. To address
this inefficiency, we revisit the robust EMP-SSL framework, emphasizing the
importance of increasing the number of crops per image to accelerate learning.
Unlike traditional contrastive learning, robust EMP-SSL leverages multi-crop
sampling, integrates an invariance term and regularization, and reduces
training epochs, enhancing time efficiency. Evaluated with both standard linear
classifiers and multi-patch embedding aggregation, robust EMP-SSL provides new
insights into SSL evaluation strategies.
  Our results show that robust crop-based EMP-SSL not only accelerates
convergence but also achieves a superior balance between clean accuracy and
adversarial robustness, outperforming multi-crop embedding aggregation.
Additionally, we extend this approach with free adversarial training in
Multi-Crop SSL, introducing the Cost-Free Adversarial Multi-Crop
Self-Supervised Learning (CF-AMC-SSL) method. CF-AMC-SSL demonstrates the
effectiveness of free adversarial training in reducing training time while
simultaneously improving clean accuracy and adversarial robustness. These
findings underscore the potential of CF-AMC-SSL for practical SSL applications.
Our code is publicly available at https://github.com/softsys4ai/CF-AMC-SSL.


http://arxiv.org/abs/2501.03301
Rethinking Byzantine Robustness in Federated Recommendation from Sparse Aggregation Perspective. (2%)
Zhongjian Zhang; Mengmei Zhang; Xiao Wang; Lingjuan Lyu; Bo Yan; Junping Du; Chuan Shi
  To preserve user privacy in recommender systems, federated recommendation
(FR) based on federated learning (FL) emerges, keeping the personal data on the
local client and updating a model collaboratively. Unlike FL, FR has a unique
sparse aggregation mechanism, where the embedding of each item is updated by
only partial clients, instead of full clients in a dense aggregation of general
FL. Recently, as an essential principle of FL, model security has received
increasing attention, especially for Byzantine attacks, where malicious clients
can send arbitrary updates. The problem of exploring the Byzantine robustness
of FR is particularly critical since in the domains applying FR, e.g.,
e-commerce, malicious clients can be injected easily by registering new
accounts. However, existing Byzantine works neglect the unique sparse
aggregation of FR, making them unsuitable for our problem. Thus, we make the
first effort to investigate Byzantine attacks on FR from the perspective of
sparse aggregation, which is non-trivial: it is not clear how to define
Byzantine robustness under sparse aggregations and design Byzantine attacks
under limited knowledge/capability. In this paper, we reformulate the Byzantine
robustness under sparse aggregation by defining the aggregation for a single
item as the smallest execution unit. Then we propose a family of effective
attack strategies, named Spattack, which exploit the vulnerability in sparse
aggregation and are categorized along the adversary's knowledge and capability.
Extensive experimental results demonstrate that Spattack can effectively
prevent convergence and even break down defenses under a few malicious clients,
raising alarms for securing FR systems.


http://arxiv.org/abs/2501.02860
Seeing the Whole in the Parts in Self-Supervised Representation Learning. (1%)
Arthur Aubret; Céline Teulière; Jochen Triesch
  Recent successes in self-supervised learning (SSL) model spatial
co-occurrences of visual features either by masking portions of an image or by
aggressively cropping it. Here, we propose a new way to model spatial
co-occurrences by aligning local representations (before pooling) with a global
image representation. We present CO-SSL, a family of instance discrimination
methods and show that it outperforms previous methods on several datasets,
including ImageNet-1K where it achieves 71.5% of Top-1 accuracy with 100
pre-training epochs. CO-SSL is also more robust to noise corruption, internal
corruption, small adversarial attacks, and large training crop sizes. Our
analysis further indicates that CO-SSL learns highly redundant local
representations, which offers an explanation for its robustness. Overall, our
work suggests that aligning local and global representations may be a powerful
principle of unsupervised category learning.


http://arxiv.org/abs/2501.02450
GCP: Guarded Collaborative Perception with Spatial-Temporal Aware Malicious Agent Detection. (67%)
Yihang Tao; Senkang Hu; Yue Hu; Haonan An; Hangcheng Cao; Yuguang Fang
  Collaborative perception significantly enhances autonomous driving safety by
extending each vehicle's perception range through message sharing among
connected and autonomous vehicles. Unfortunately, it is also vulnerable to
adversarial message attacks from malicious agents, resulting in severe
performance degradation. While existing defenses employ
hypothesis-and-verification frameworks to detect malicious agents based on
single-shot outliers, they overlook temporal message correlations, which can be
circumvented by subtle yet harmful perturbations in model input and output
spaces. This paper reveals a novel blind area confusion (BAC) attack that
compromises existing single-shot outlier-based detection methods. As a
countermeasure, we propose GCP, a Guarded Collaborative Perception framework
based on spatial-temporal aware malicious agent detection, which maintains
single-shot spatial consistency through a confidence-scaled spatial concordance
loss, while simultaneously examining temporal anomalies by reconstructing
historical bird's eye view motion flows in low-confidence regions. We also
employ a joint spatial-temporal Benjamini-Hochberg test to synthesize
dual-domain anomaly results for reliable malicious agent detection. Extensive
experiments demonstrate GCP's superior performance under diverse attack
scenarios, achieving up to 34.69% improvements in AP@0.5 compared to the
state-of-the-art CP defense strategies under BAC attacks, while maintaining
consistent 5-8% improvements under other typical attacks. Code will be released
at https://github.com/CP-Security/GCP.git.


http://arxiv.org/abs/2501.02629
Layer-Level Self-Exposure and Patch: Affirmative Token Mitigation for Jailbreak Attack Defense. (67%)
Yang Ouyang; Hengrui Gu; Shuhang Lin; Wenyue Hua; Jie Peng; Bhavya Kailkhura; Meijun Gao; Tianlong Chen; Kaixiong Zhou
  As large language models (LLMs) are increasingly deployed in diverse
applications, including chatbot assistants and code generation, aligning their
behavior with safety and ethical standards has become paramount. However,
jailbreak attacks, which exploit vulnerabilities to elicit unintended or
harmful outputs, threaten LLMs' safety significantly. In this paper, we
introduce Layer-AdvPatcher, a novel methodology designed to defend against
jailbreak attacks by utilizing an unlearning strategy to patch specific layers
within LLMs through self-augmented datasets. Our insight is that certain
layer(s), tend to produce affirmative tokens when faced with harmful prompts.
By identifying these layers and adversarially exposing them to generate more
harmful data, one can understand their inherent and diverse vulnerabilities to
attacks. With these exposures, we then "unlearn" these issues, reducing the
impact of affirmative tokens and hence minimizing jailbreak risks while keeping
the model's responses to safe queries intact. We conduct extensive experiments
on two models, four benchmark datasets, and multiple state-of-the-art jailbreak
attacks to demonstrate the efficacy of our approach. Results indicate that our
framework reduces the harmfulness and attack success rate of jailbreak attacks
without compromising utility for benign queries compared to recent defense
methods. Our code is publicly available at:
https://github.com/oyy2000/LayerAdvPatcher


http://arxiv.org/abs/2501.02654
Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks. (54%)
Yang Wang; Chenghua Lin
  vulnerability of deep learning models to adversarial attacks. While various
defence mechanisms have been proposed, there is a lack of comprehensive
benchmarks that evaluate these defences across diverse datasets, models, and
tasks. In this work, we address this gap by presenting an extensive benchmark
for textual adversarial defence that significantly expands upon previous work.
Our benchmark incorporates a wide range of datasets, evaluates state-of-the-art
defence mechanisms, and extends the assessment to include critical tasks such
as single-sentence classification, similarity and paraphrase identification,
natural language inference, and commonsense reasoning. This work not only
serves as a valuable resource for researchers and practitioners in the field of
adversarial robustness but also identifies key areas for future research in
textual adversarial defence. By establishing a new standard for benchmarking in
this domain, we aim to accelerate progress towards more robust and reliable
natural language processing systems.


http://arxiv.org/abs/2501.02704
Persistence of Backdoor-based Watermarks for Neural Networks: A Comprehensive Evaluation. (5%)
Anh Tu Ngo; Chuan Song Heng; Nandish Chattopadhyay; Anupam Chattopadhyay
  Deep Neural Networks (DNNs) have gained considerable traction in recent years
due to the unparalleled results they gathered. However, the cost behind
training such sophisticated models is resource intensive, resulting in many to
consider DNNs to be intellectual property (IP) to model owners. In this era of
cloud computing, high-performance DNNs are often deployed all over the internet
so that people can access them publicly. As such, DNN watermarking schemes,
especially backdoor-based watermarks, have been actively developed in recent
years to preserve proprietary rights. Nonetheless, there lies much uncertainty
on the robustness of existing backdoor watermark schemes, towards both
adversarial attacks and unintended means such as fine-tuning neural network
models. One reason for this is that no complete guarantee of robustness can be
assured in the context of backdoor-based watermark. In this paper, we
extensively evaluate the persistence of recent backdoor-based watermarks within
neural networks in the scenario of fine-tuning, we propose/develop a novel
data-driven idea to restore watermark after fine-tuning without exposing the
trigger set. Our empirical results show that by solely introducing training
data after fine-tuning, the watermark can be restored if model parameters do
not shift dramatically during fine-tuning. Depending on the types of trigger
samples used, trigger accuracy can be reinstated to up to 100%. Our study
further explores how the restoration process works using loss landscape
visualization, as well as the idea of introducing training data in fine-tuning
stage to alleviate watermark vanishing.


http://arxiv.org/abs/2501.02232
Distillation-Enhanced Physical Adversarial Attacks. (96%)
Wei Liu; Yonglin Wu; Chaoqun Li; Zhuodong Liu; Huanqian Yan
  The study of physical adversarial patches is crucial for identifying
vulnerabilities in AI-based recognition systems and developing more robust deep
learning models. While recent research has focused on improving patch
stealthiness for greater practical applicability, achieving an effective
balance between stealth and attack performance remains a significant challenge.
To address this issue, we propose a novel physical adversarial attack method
that leverages knowledge distillation. Specifically, we first define a stealthy
color space tailored to the target environment to ensure smooth blending. Then,
we optimize an adversarial patch in an unconstrained color space, which serves
as the 'teacher' patch. Finally, we use an adversarial knowledge distillation
module to transfer the teacher patch's knowledge to the 'student' patch,
guiding the optimization of the stealthy patch. Experimental results show that
our approach improves attack performance by 20%, while maintaining stealth,
highlighting its practical value.


http://arxiv.org/abs/2501.02406
Zero-Shot Statistical Tests for LLM-Generated Text Detection using Finite Sample Concentration Inequalities. (83%)
Tara Radvand; Mojtaba Abdolmaleki; Mohamed Mostagir; Ambuj Tewari
  Verifying the provenance of content is crucial to the function of many
organizations, e.g., educational institutions, social media platforms, firms,
etc. This problem is becoming increasingly challenging as text generated by
Large Language Models (LLMs) becomes almost indistinguishable from
human-generated content. In addition, many institutions utilize in-house LLMs
and want to ensure that external, non-sanctioned LLMs do not produce content
within the institution. In this paper, we answer the following question: Given
a piece of text, can we identify whether it was produced by a particular LLM or
not? We model LLM-generated text as a sequential stochastic process with
complete dependence on history. We then design zero-shot statistical tests to
(i) distinguish between text generated by two different known sets of LLMs $A$
(non-sanctioned) and $B$ (in-house), and (ii) identify whether text was
generated by a known LLM or generated by any unknown model, e.g., a human or
some other language generation process. We prove that the type I and type II
errors of our test decrease exponentially with the length of the text. For
that, we show that if $B$ generates the text, then except with an exponentially
small probability in string length, the log-perplexity of the string under $A$
converges to the average cross-entropy of $B$ and $A$. We then present
experiments using LLMs with white-box access to support our theoretical results
and empirically examine the robustness of our results to black-box settings and
adversarial attacks. In the black-box setting, our method achieves an average
TPR of 82.5\% at a fixed FPR of 5\%. Under adversarial perturbations, our
minimum TPR is 48.6\% at the same FPR threshold. Both results outperform all
non-commercial baselines. See
https://github.com/TaraRadvand74/llm-text-detection for code, data, and an
online demo of the project.


http://arxiv.org/abs/2501.03272
Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models. (75%)
Peihai Jiang; Xixiang Lyu; Yige Li; Jing Ma
  Supervised fine-tuning has become the predominant method for adapting large
pretrained models to downstream tasks. However, recent studies have revealed
that these models are vulnerable to backdoor attacks, where even a small number
of malicious samples can successfully embed backdoor triggers into the model.
While most existing defense methods focus on post-training backdoor defense,
efficiently defending against backdoor attacks during training phase remains
largely unexplored. To address this gap, we propose a novel defense method
called Backdoor Token Unlearning (BTU), which proactively detects and
neutralizes trigger tokens during the training stage. Our work is based on two
key findings: 1) backdoor learning causes distinctive differences between
backdoor token parameters and clean token parameters in word embedding layers,
and 2) the success of backdoor attacks heavily depends on backdoor token
parameters. The BTU defense leverages these properties to identify aberrant
embedding parameters and subsequently removes backdoor behaviors using a
fine-grained unlearning technique. Extensive evaluations across three datasets
and four types of backdoor attacks demonstrate that BTU effectively defends
against these threats while preserving the model's performance on primary
tasks. Our code is available at https://github.com/XDJPH/BTU.


http://arxiv.org/abs/2501.02373
BADTV: Unveiling Backdoor Threats in Third-Party Task Vectors. (38%)
Chia-Yi Hsu; Yu-Lin Tsai; Yu Zhe; Yan-Lun Chen; Chih-Hsun Lin; Chia-Mu Yu; Yang Zhang; Chun-Ying Huang; Jun Sakuma
  Task arithmetic in large-scale pre-trained models enables agile adaptation to
diverse downstream tasks without extensive retraining. By leveraging task
vectors (TVs), users can perform modular updates through simple arithmetic
operations like addition and subtraction. Yet, this flexibility presents new
security challenges. In this paper, we investigate how TVs are vulnerable to
backdoor attacks, revealing how malicious actors can exploit them to compromise
model integrity. By creating composite backdoors that are designed
asymmetrically, we introduce BadTV, a backdoor attack specifically crafted to
remain effective simultaneously under task learning, forgetting, and analogy
operations. Extensive experiments show that BadTV achieves near-perfect attack
success rates across diverse scenarios, posing a serious threat to models
relying on task arithmetic. We also evaluate current defenses, finding they
fail to detect or mitigate BadTV. Our results highlight the urgent need for
robust countermeasures to secure TVs in real-world deployments.


http://arxiv.org/abs/2501.02042
Towards Robust and Accurate Stability Estimation of Local Surrogate Models in Text-based Explainable AI. (98%)
Christopher Burger; Charles Walter; Thai Le; Lingwei Chen
  Recent work has investigated the concept of adversarial attacks on
explainable AI (XAI) in the NLP domain with a focus on examining the
vulnerability of local surrogate methods such as Lime to adversarial
perturbations or small changes on the input of a machine learning (ML) model.
In such attacks, the generated explanation is manipulated while the meaning and
structure of the original input remain similar under the ML model. Such attacks
are especially alarming when XAI is used as a basis for decision making (e.g.,
prescribing drugs based on AI medical predictors) or for legal action (e.g.,
legal dispute involving AI software). Although weaknesses across many XAI
methods have been shown to exist, the reasons behind why remain little
explored. Central to this XAI manipulation is the similarity measure used to
calculate how one explanation differs from another. A poor choice of similarity
measure can lead to erroneous conclusions about the stability or adversarial
robustness of an XAI method. Therefore, this work investigates a variety of
similarity measures designed for text-based ranked lists referenced in related
work to determine their comparative suitability for use. We find that many
measures are overly sensitive, resulting in erroneous estimates of stability.
We then propose a weighting scheme for text-based data that incorporates the
synonymity between the features within an explanation, providing more accurate
estimates of the actual weakness of XAI methods to adversarial examples.


http://arxiv.org/abs/2501.02147
Exploring Secure Machine Learning Through Payload Injection and FGSM Attacks on ResNet-50. (93%)
Umesh Yadav; Suman Niraula; Gaurav Kumar Gupta; Bicky Yadav
  This paper investigates the resilience of a ResNet-50 image classification
model under two prominent security threats: Fast Gradient Sign Method (FGSM)
adversarial attacks and malicious payload injection. Initially, the model
attains a 53.33% accuracy on clean images. When subjected to FGSM
perturbations, its overall accuracy remains unchanged; however, the model's
confidence in incorrect predictions notably increases. Concurrently, a payload
injection scheme is successfully executed in 93.33% of the tested samples,
revealing how stealthy attacks can manipulate model predictions without
degrading visual quality. These findings underscore the vulnerability of even
high-performing neural networks and highlight the urgency of developing more
robust defense mechanisms for security-critical applications.


http://arxiv.org/abs/2501.01818
Rerouting LLM Routers. (82%)
Avital Shafran; Roei Schuster; Thomas Ristenpart; Vitaly Shmatikov
  LLM routers aim to balance quality and cost of generation by classifying
queries and routing them to a cheaper or more expensive LLM depending on their
complexity. Routers represent one type of what we call LLM control planes:
systems that orchestrate use of one or more LLMs. In this paper, we investigate
routers' adversarial robustness.
  We first define LLM control plane integrity, i.e., robustness of LLM
orchestration to adversarial inputs, as a distinct problem in AI safety. Next,
we demonstrate that an adversary can generate query-independent token sequences
we call ``confounder gadgets'' that, when added to any query, cause LLM routers
to send the query to a strong LLM.
  Our quantitative evaluation shows that this attack is successful both in
white-box and black-box settings against a variety of open-source and
commercial routers, and that confounding queries do not affect the quality of
LLM responses. Finally, we demonstrate that gadgets can be effective while
maintaining low perplexity, thus perplexity-based filtering is not an effective
defense. We finish by investigating alternative defenses.


http://arxiv.org/abs/2501.02135
AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs. (69%)
Sanjoy Chowdhury; Sayan Nag; Subhrajyoti Dasgupta; Yaoting Wang; Mohamed Elhoseiny; Ruohan Gao; Dinesh Manocha
  With the rapid advancement of Multi-modal Large Language Models (MLLMs),
several diagnostic benchmarks have recently been developed to assess these
models' multi-modal reasoning proficiency. However, these benchmarks are
restricted to assessing primarily the visual aspect and do not examine the
holistic audio-visual (AV) understanding. Moreover, currently, there are no
benchmarks that investigate the capabilities of AVLLMs to calibrate their
responses when presented with perturbed inputs. To this end, we introduce
Audio-Visual Trustworthiness assessment Benchmark (AVTrustBench), comprising
600K samples spanning over 9 meticulously crafted tasks, evaluating the
capabilities of AVLLMs across three distinct dimensions: Adversarial attack,
Compositional reasoning, and Modality-specific dependency. Using our benchmark
we extensively evaluate 13 state-of-the-art AVLLMs. The findings reveal that
the majority of existing models fall significantly short of achieving
human-like comprehension, offering valuable insights for future research
directions. To alleviate the limitations in the existing approaches, we further
propose a robust, model-agnostic calibrated audio-visual preference
optimization based training strategy CAVPref, obtaining a gain up to 30.19%
across all 9 tasks. We will publicly release our code and benchmark to
facilitate future research in this direction.


http://arxiv.org/abs/2501.01913
Mingling with the Good to Backdoor Federated Learning. (12%)
Nuno Neves
  Federated learning (FL) is a decentralized machine learning technique that
allows multiple entities to jointly train a model while preserving dataset
privacy. However, its distributed nature has raised various security concerns,
which have been addressed by increasingly sophisticated defenses. These
protections utilize a range of data sources and metrics to, for example, filter
out malicious model updates, ensuring that the impact of attacks is minimized
or eliminated.
  This paper explores the feasibility of designing a generic attack method
capable of installing backdoors in FL while evading a diverse array of
defenses. Specifically, we focus on an attacker strategy called MIGO, which
aims to produce model updates that subtly blend with legitimate ones. The
resulting effect is a gradual integration of a backdoor into the global model,
often ensuring its persistence long after the attack concludes, while
generating enough ambiguity to hinder the effectiveness of defenses.
  MIGO was employed to implant three types of backdoors across five datasets
and different model architectures. The results demonstrate the significant
threat posed by these backdoors, as MIGO consistently achieved exceptionally
high backdoor accuracy (exceeding 90%) while maintaining the utility of the
main task. Moreover, MIGO exhibited strong evasion capabilities against ten
defenses, including several state-of-the-art methods. When compared to four
other attack strategies, MIGO consistently outperformed them across most
configurations. Notably, even in extreme scenarios where the attacker controls
just 0.1% of the clients, the results indicate that successful backdoor
insertion is possible if the attacker can persist for a sufficient number of
rounds.


http://arxiv.org/abs/2501.02182
AdaMixup: A Dynamic Defense Framework for Membership Inference Attack Mitigation. (10%)
Ying Chen; Jiajing Chen; Yijie Weng; ChiaHua Chang; Dezhi Yu; Guanbiao Lin
  Membership inference attacks have emerged as a significant privacy concern in
the training of deep learning models, where attackers can infer whether a data
point was part of the training set based on the model's outputs. To address
this challenge, we propose a novel defense mechanism, AdaMixup. AdaMixup
employs adaptive mixup techniques to enhance the model's robustness against
membership inference attacks by dynamically adjusting the mixup strategy during
training. This method not only improves the model's privacy protection but also
maintains high performance. Experimental results across multiple datasets
demonstrate that AdaMixup significantly reduces the risk of membership
inference attacks while achieving a favorable trade-off between defensive
efficiency and model accuracy. This research provides an effective solution for
data privacy protection and lays the groundwork for future advancements in
mixup training methods.


http://arxiv.org/abs/2501.01741
How Toxic Can You Get? Search-based Toxicity Testing for Large Language Models. (9%)
Simone Corbo; Luca Bancale; Gennaro Valeria De; Livia Lestingi; Vincenzo Scotti; Matteo Camilli
  Language is a deep-rooted means of perpetration of stereotypes and
discrimination. Large Language Models (LLMs), now a pervasive technology in our
everyday lives, can cause extensive harm when prone to generating toxic
responses. The standard way to address this issue is to align the LLM, which,
however, dampens the issue without constituting a definitive solution.
Therefore, testing LLM even after alignment efforts remains crucial for
detecting any residual deviations with respect to ethical standards. We present
EvoTox, an automated testing framework for LLMs' inclination to toxicity,
providing a way to quantitatively assess how much LLMs can be pushed towards
toxic responses even in the presence of alignment. The framework adopts an
iterative evolution strategy that exploits the interplay between two LLMs, the
System Under Test (SUT) and the Prompt Generator steering SUT responses toward
higher toxicity. The toxicity level is assessed by an automated oracle based on
an existing toxicity classifier. We conduct a quantitative and qualitative
empirical evaluation using four state-of-the-art LLMs as evaluation subjects
having increasing complexity (7-13 billion parameters). Our quantitative
evaluation assesses the cost-effectiveness of four alternative versions of
EvoTox against existing baseline methods, based on random search, curated
datasets of toxic prompts, and adversarial attacks. Our qualitative assessment
engages human evaluators to rate the fluency of the generated prompts and the
perceived toxicity of the responses collected during the testing sessions.
Results indicate that the effectiveness, in terms of detected toxicity level,
is significantly higher than the selected baseline methods (effect size up to
1.0 against random search and up to 0.99 against adversarial attacks).
Furthermore, EvoTox yields a limited cost overhead (from 22% to 35% on
average).


http://arxiv.org/abs/2501.01620
Adaptive Meta-learning-based Adversarial Training for Robust Automatic Modulation Classification. (99%)
Amirmohammad Bamdad; Ali Owfi; Fatemeh Afghah
  DL-based automatic modulation classification (AMC) models are highly
susceptible to adversarial attacks, where even minimal input perturbations can
cause severe misclassifications. While adversarially training an AMC model
based on an adversarial attack significantly increases its robustness against
that attack, the AMC model will still be defenseless against other adversarial
attacks. The theoretically infinite possibilities for adversarial perturbations
mean that an AMC model will inevitably encounter new unseen adversarial attacks
if it is ever to be deployed to a real-world communication system. Moreover,
the computational limitations and challenges of obtaining new data in real-time
will not allow a full training process for the AMC model to adapt to the new
attack when it is online. To this end, we propose a meta-learning-based
adversarial training framework for AMC models that substantially enhances
robustness against unseen adversarial attacks and enables fast adaptation to
these attacks using just a few new training samples, if any are available. Our
results demonstrate that this training framework provides superior robustness
and accuracy with much less online training time than conventional adversarial
training of AMC models, making it highly efficient for real-world deployment.


http://arxiv.org/abs/2501.01106
AIM: Additional Image Guided Generation of Transferable Adversarial Attacks. (99%)
Teng Li; Xingjun Ma; Yu-Gang Jiang
  Transferable adversarial examples highlight the vulnerability of deep neural
networks (DNNs) to imperceptible perturbations across various real-world
applications. While there have been notable advancements in untargeted
transferable attacks, targeted transferable attacks remain a significant
challenge. In this work, we focus on generative approaches for targeted
transferable attacks. Current generative attacks focus on reducing overfitting
to surrogate models and the source data domain, but they often overlook the
importance of enhancing transferability through additional semantics. To
address this issue, we introduce a novel plug-and-play module into the general
generator architecture to enhance adversarial transferability. Specifically, we
propose a \emph{Semantic Injection Module} (SIM) that utilizes the semantics
contained in an additional guiding image to improve transferability. The
guiding image provides a simple yet effective method to incorporate target
semantics from the target class to create targeted and highly transferable
attacks. Additionally, we propose new loss formulations that can integrate the
semantic injection module more effectively for both targeted and untargeted
attacks. We conduct comprehensive experiments under both targeted and
untargeted attack settings to demonstrate the efficacy of our proposed
approach.


http://arxiv.org/abs/2501.01090
HoneypotNet: Backdoor Attacks Against Model Extraction. (93%)
Yixu Wang; Tianle Gu; Yan Teng; Yingchun Wang; Xingjun Ma
  Model extraction attacks are one type of inference-time attacks that
approximate the functionality and performance of a black-box victim model by
launching a certain number of queries to the model and then leveraging the
model's predictions to train a substitute model. These attacks pose severe
security threats to production models and MLaaS platforms and could cause
significant monetary losses to the model owners. A body of work has proposed to
defend machine learning models against model extraction attacks, including both
active defense methods that modify the model's outputs or increase the query
overhead to avoid extraction and passive defense methods that detect malicious
queries or leverage watermarks to perform post-verification. In this work, we
introduce a new defense paradigm called attack as defense which modifies the
model's output to be poisonous such that any malicious users that attempt to
use the output to train a substitute model will be poisoned. To this end, we
propose a novel lightweight backdoor attack method dubbed HoneypotNet that
replaces the classification layer of the victim model with a honeypot layer and
then fine-tunes the honeypot layer with a shadow model (to simulate model
extraction) via bi-level optimization to modify its output to be poisonous
while remaining the original performance. We empirically demonstrate on four
commonly used benchmark datasets that HoneypotNet can inject backdoors into
substitute models with a high success rate. The injected backdoor not only
facilitates ownership verification but also disrupts the functionality of
substitute models, serving as a significant deterrent to model extraction
attacks.


http://arxiv.org/abs/2501.01516
Improving Robustness Estimates in Natural Language Explainable AI though Synonymity Weighted Similarity Measures. (87%)
Christopher Burger
  Explainable AI (XAI) has seen a surge in recent interest with the
proliferation of powerful but intractable black-box models. Moreover, XAI has
come under fire for techniques that may not offer reliable explanations. As
many of the methods in XAI are themselves models, adversarial examples have
been prominent in the literature surrounding the effectiveness of XAI, with the
objective of these examples being to alter the explanation while maintaining
the output of the original model. For explanations in natural language, it is
natural to use measures found in the domain of information retrieval for use
with ranked lists to guide the adversarial XAI process. We show that the
standard implementation of these measures are poorly suited for the comparison
of explanations in adversarial XAI and amend them by using information that is
discarded, the synonymity of perturbed words. This synonymity weighting
produces more accurate estimates of the actual weakness of XAI methods to
adversarial examples.


http://arxiv.org/abs/2501.01263
Stealthy Backdoor Attack to Real-world Models in Android Apps. (80%)
Jiali Wei; Ming Fan; Xicheng Zhang; Wenjing Jiao; Haijun Wang; Ting Liu
  Powered by their superior performance, deep neural networks (DNNs) have found
widespread applications across various domains. Many deep learning (DL) models
are now embedded in mobile apps, making them more accessible to end users
through on-device DL. However, deploying on-device DL to users' smartphones
simultaneously introduces several security threats. One primary threat is
backdoor attacks. Extensive research has explored backdoor attacks for several
years and has proposed numerous attack approaches. However, few studies have
investigated backdoor attacks on DL models deployed in the real world, or they
have shown obvious deficiencies in effectiveness and stealthiness. In this
work, we explore more effective and stealthy backdoor attacks on real-world DL
models extracted from mobile apps. Our main justification is that imperceptible
and sample-specific backdoor triggers generated by DNN-based steganography can
enhance the efficacy of backdoor attacks on real-world models. We first confirm
the effectiveness of steganography-based backdoor attacks on four
state-of-the-art DNN models. Subsequently, we systematically evaluate and
analyze the stealthiness of the attacks to ensure they are difficult to
perceive. Finally, we implement the backdoor attacks on real-world models and
compare our approach with three baseline methods. We collect 38,387 mobile
apps, extract 89 DL models from them, and analyze these models to obtain the
prerequisite model information for the attacks. After identifying the target
models, our approach achieves an average of 12.50% higher attack success rate
than DeepPayload while better maintaining the normal performance of the models.
Extensive experimental results demonstrate that our method enables more
effective, robust, and stealthy backdoor attacks on real-world models.


http://arxiv.org/abs/2501.01529
SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers. (45%)
Bhavna Gopal; Huanrui Yang; Mark Horton; Yiran Chen
  Vision transformers (ViTs) have become essential backbones in advanced
computer vision applications and multi-modal foundation models. Despite their
strengths, ViTs remain vulnerable to adversarial perturbations, comparable to
or even exceeding the vulnerability of convolutional neural networks (CNNs).
Furthermore, the large parameter count and complex architecture of ViTs make
them particularly prone to adversarial overfitting, often compromising both
clean and adversarial accuracy.
  This paper mitigates adversarial overfitting in ViTs through a novel,
layer-selective fine-tuning approach: SAFER. Instead of optimizing the entire
model, we identify and selectively fine-tune a small subset of layers most
susceptible to overfitting, applying sharpness-aware minimization to these
layers while freezing the rest of the model. Our method consistently enhances
both clean and adversarial accuracy over baseline approaches. Typical
improvements are around 5%, with some cases achieving gains as high as 20%
across various ViT architectures and datasets.


http://arxiv.org/abs/2501.01606
Test Input Validation for Vision-based DL Systems: An Active Learning Approach. (1%)
Delaram Ghobari; Mohammad Hossein Amini; Dai Quoc Tran; Seunghee Park; Shiva Nejati; Mehrdad Sabetzadeh
  Testing deep learning (DL) systems requires extensive and diverse, yet valid,
test inputs. While synthetic test input generation methods, such as metamorphic
testing, are widely used for DL testing, they risk introducing invalid inputs
that do not accurately reflect real-world scenarios. Invalid test inputs can
lead to misleading results. Hence, there is a need for automated validation of
test inputs to ensure effective assessment of DL systems. In this paper, we
propose a test input validation approach for vision-based DL systems. Our
approach uses active learning to balance the trade-off between accuracy and the
manual effort required for test input validation. Further, by employing
multiple image-comparison metrics, it achieves better results in classifying
valid and invalid test inputs compared to methods that rely on single metrics.
We evaluate our approach using an industrial and a public-domain dataset. Our
evaluation shows that our multi-metric, active learning-based approach produces
several optimal accuracy-effort trade-offs, including those deemed practical
and desirable by our industry partner. Furthermore, provided with the same
level of manual effort, our approach is significantly more accurate than two
state-of-the-art test input validation methods, achieving an average accuracy
of 97%. Specifically, the use of multiple metrics, rather than a single metric,
results in an average improvement of at least 5.4% in overall accuracy compared
to the state-of-the-art baselines. Incorporating an active learning loop for
test input validation yields an additional 7.5% improvement in average
accuracy, bringing the overall average improvement of our approach to at least
12.9% compared to the baselines.


http://arxiv.org/abs/2501.01558
Predicting the Performance of Black-box LLMs through Self-Queries. (1%)
Dylan Sam; Marc Finzi; J. Zico Kolter
  As large language models (LLMs) are increasingly relied on in AI systems,
predicting when they make mistakes is crucial. While a great deal of work in
the field uses internal representations to interpret model behavior, these
representations are inaccessible when given solely black-box access through an
API. In this paper, we extract features of LLMs in a black-box manner by using
follow-up prompts and taking the probabilities of different responses as
representations to train reliable predictors of model behavior. We demonstrate
that training a linear model on these low-dimensional representations produces
reliable and generalizable predictors of model performance at the instance
level (e.g., if a particular generation correctly answers a question).
Remarkably, these can often outperform white-box linear predictors that operate
over a model's hidden state or the full distribution over its vocabulary. In
addition, we demonstrate that these extracted features can be used to evaluate
more nuanced aspects of a language model's state. For instance, they can be
used to distinguish between a clean version of GPT-4o-mini and a version that
has been influenced via an adversarial system prompt that answers
question-answering tasks incorrectly or introduces bugs into generated code.
Furthermore, they can reliably distinguish between different model
architectures and sizes, enabling the detection of misrepresented models
provided through an API (e.g., identifying if GPT-3.5 is supplied instead of
GPT-4o-mini).


http://arxiv.org/abs/2501.01015
Boosting Adversarial Transferability with Spatial Adversarial Alignment. (99%)
Zhaoyu Chen; Haijing Guo; Kaixun Jiang; Jiyuan Fu; Xinyu Zhou; Dingkang Yang; Hao Tang; Bo Li; Wenqiang Zhang
  Deep neural networks are vulnerable to adversarial examples that exhibit
transferability across various models. Numerous approaches are proposed to
enhance the transferability of adversarial examples, including advanced
optimization, data augmentation, and model modifications. However, these
methods still show limited transferability, particularly in cross-architecture
scenarios, such as from CNN to ViT. To achieve high transferability, we propose
a technique termed Spatial Adversarial Alignment (SAA), which employs an
alignment loss and leverages a witness model to fine-tune the surrogate model.
Specifically, SAA consists of two key parts: spatial-aware alignment and
adversarial-aware alignment. First, we minimize the divergences of features
between the two models in both global and local regions, facilitating spatial
alignment. Second, we introduce a self-adversarial strategy that leverages
adversarial examples to impose further constraints, aligning features from an
adversarial perspective. Through this alignment, the surrogate model is trained
to concentrate on the common features extracted by the witness model. This
facilitates adversarial attacks on these shared features, thereby yielding
perturbations that exhibit enhanced transferability. Extensive experiments on
various architectures on ImageNet show that aligned surrogate models based on
SAA can provide higher transferable adversarial examples, especially in
cross-architecture attacks.


http://arxiv.org/abs/2501.01025
Towards Adversarially Robust Deep Metric Learning. (99%)
Xiaopeng Ke
  Deep Metric Learning (DML) has shown remarkable successes in many domains by
taking advantage of powerful deep neural networks. Deep neural networks are
prone to adversarial attacks and could be easily fooled by adversarial
examples. The current progress on this robustness issue is mainly about deep
classification models but pays little attention to DML models. Existing works
fail to thoroughly inspect the robustness of DML and neglect an important DML
scenario, the clustering-based inference. In this work, we first point out the
robustness issue of DML models in clustering-based inference scenarios. We find
that, for the clustering-based inference, existing defenses designed DML are
unable to be reused and the adaptions of defenses designed for deep
classification models cannot achieve satisfactory robustness performance. To
alleviate the hazard of adversarial examples, we propose a new defense, the
Ensemble Adversarial Training (EAT), which exploits ensemble learning and
adversarial training. EAT promotes the diversity of the ensemble, encouraging
each model in the ensemble to have different robustness features, and employs a
self-transferring mechanism to make full use of the robustness statistics of
the whole ensemble in the update of every single model. We evaluate the EAT
method on three widely-used datasets with two popular model architectures. The
results show that the proposed EAT method greatly outperforms the adaptions of
defenses designed for deep classification models.


http://arxiv.org/abs/2501.01042
Image-based Multimodal Models as Intruders: Transferable Multimodal Attacks on Video-based MLLMs. (99%)
Linhao Huang; Xue Jiang; Zhiqiang Wang; Wentao Mo; Xi Xiao; Bo Han; Yongjie Yin; Feng Zheng
  Video-based multimodal large language models (V-MLLMs) have shown
vulnerability to adversarial examples in video-text multimodal tasks. However,
the transferability of adversarial videos to unseen models--a common and
practical real world scenario--remains unexplored. In this paper, we pioneer an
investigation into the transferability of adversarial video samples across
V-MLLMs. We find that existing adversarial attack methods face significant
limitations when applied in black-box settings for V-MLLMs, which we attribute
to the following shortcomings: (1) lacking generalization in perturbing video
features, (2) focusing only on sparse key-frames, and (3) failing to integrate
multimodal information. To address these limitations and deepen the
understanding of V-MLLM vulnerabilities in black-box scenarios, we introduce
the Image-to-Video MLLM (I2V-MLLM) attack. In I2V-MLLM, we utilize an
image-based multimodal model (IMM) as a surrogate model to craft adversarial
video samples. Multimodal interactions and temporal information are integrated
to disrupt video representations within the latent space, improving adversarial
transferability. In addition, a perturbation propagation technique is
introduced to handle different unknown frame sampling strategies. Experimental
results demonstrate that our method can generate adversarial examples that
exhibit strong transferability across different V-MLLMs on multiple video-text
multimodal tasks. Compared to white-box attacks on these models, our black-box
attacks (using BLIP-2 as surrogate model) achieve competitive performance, with
average attack success rates of 55.48% on MSVD-QA and 58.26% on MSRVTT-QA for
VideoQA tasks, respectively. Our code will be released upon acceptance.


http://arxiv.org/abs/2501.00745
Dynamics of Adversarial Attacks on Large Language Model-Based Search Engines. (76%)
Xiyang Hu
  The increasing integration of Large Language Model (LLM) based search engines
has transformed the landscape of information retrieval. However, these systems
are vulnerable to adversarial attacks, especially ranking manipulation attacks,
where attackers craft webpage content to manipulate the LLM's ranking and
promote specific content, gaining an unfair advantage over competitors. In this
paper, we study the dynamics of ranking manipulation attacks. We frame this
problem as an Infinitely Repeated Prisoners' Dilemma, where multiple players
strategically decide whether to cooperate or attack. We analyze the conditions
under which cooperation can be sustained, identifying key factors such as
attack costs, discount rates, attack success rates, and trigger strategies that
influence player behavior. We identify tipping points in the system dynamics,
demonstrating that cooperation is more likely to be sustained when players are
forward-looking. However, from a defense perspective, we find that simply
reducing attack success probabilities can, paradoxically, incentivize attacks
under certain conditions. Furthermore, defensive measures to cap the upper
bound of attack success rates may prove futile in some scenarios. These
insights highlight the complexity of securing LLM-based systems. Our work
provides a theoretical foundation and practical insights for understanding and
mitigating their vulnerabilities, while emphasizing the importance of adaptive
security strategies and thoughtful ecosystem design.


http://arxiv.org/abs/2501.00879
TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generation. (16%)
Huichi Zhou; Kin-Hei Lee; Zhonghao Zhan; Yue Chen; Zhenhao Li; Zhaoyang Wang; Hamed Haddadi; Emine Yilmaz
  Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by
integrating external knowledge sources, enabling more accurate and contextually
relevant responses tailored to user queries. These systems, however, remain
susceptible to corpus poisoning attacks, which can severely impair the
performance of LLMs. To address this challenge, we propose TrustRAG, a robust
framework that systematically filters malicious and irrelevant content before
it is retrieved for generation. Our approach employs a two-stage defense
mechanism. The first stage implements a cluster filtering strategy to detect
potential attack patterns. The second stage employs a self-assessment process
that harnesses the internal capabilities of LLMs to detect malicious documents
and resolve inconsistencies. TrustRAG provides a plug-and-play, training-free
module that integrates seamlessly with any open- or closed-source language
model. Extensive experiments demonstrate that TrustRAG delivers substantial
improvements in retrieval accuracy, efficiency, and attack resistance.


http://arxiv.org/abs/2501.00973
Defense Strategies for Autonomous Multi-agent Systems: Ensuring Safety and Resilience Under Exponentially Unbounded FDI Attacks. (2%)
Yichao Wang; Mohamadamin Rajabinezhad; Dimitra Panagou; Shan Zuo
  False data injection attacks pose a significant threat to autonomous
multi-agent systems (MASs). Existing attack-resilient control strategies
generally have strict assumptions on the attack signals and overlook safety
constraints, such as collision avoidance. In practical applications, leader
agents equipped with advanced sensors or weaponry span a safe region to guide
heterogeneous follower agents, ensuring coordinated operations while addressing
collision avoidance to prevent financial losses and mission failures. This
letter addresses these gaps by introducing and solving the safety-aware and
attack-resilient (SAAR) control problem under exponentially unbounded false
data injection (EU-FDI) attacks. Specifically, a novel attack-resilient
observer layer (OL) is first designed to defend against EU-FDI attacks on the
OL. Then, an attack-resilient compensational signal is designed to mitigate the
adverse effects caused by the EU-FDI attack on control input layer (CIL).
Finally, a SAAR controller is designed by solving a quadratic programming (QP)
problem integrating control barrier function (CBF) certified collision-free
safety constraints. Rigorous Lyapunov-based stability analysis certifies the
SAAR controller's effectiveness in ensuring both safety and resilience. This
study also pioneers a three-dimensional (3D) simulation of the SAAR containment
control problem for heterogeneous MASs, demonstrating its applicability in
realistic multi-agent scenarios.


http://arxiv.org/abs/2501.00824
How Breakable Is Privacy: Probing and Resisting Model Inversion Attacks in Collaborative Inference. (2%)
Rongke Liu
  Collaborative inference (CI) improves computational efficiency for edge
devices by transmitting intermediate features to cloud models. However, this
process inevitably exposes feature representations to model inversion attacks
(MIAs), enabling unauthorized data reconstruction. Despite extensive research,
there is no established criterion for assessing the difficulty of MIA
implementation, leaving a fundamental question unanswered: \textit{What factors
truly and verifiably determine the attack's success in CI?} Moreover, existing
defenses lack the theoretical foundation described above, making it challenging
to regulate feature information effectively while ensuring privacy and
minimizing computational overhead. These shortcomings introduce three key
challenges: theoretical gap, methodological limitation, and practical
constraint.
  To overcome these challenges, we propose the first theoretical criterion to
assess MIA difficulty in CI, identifying mutual information, entropy, and
effective information volume as key influencing factors. The validity of this
criterion is demonstrated by using the mutual information neural estimator.
Building on this insight, we propose SiftFunnel, a privacy-preserving framework
to resist MIA while maintaining usability. Specifically, we incorporate linear
and non-linear correlation constraints alongside label smoothing to suppress
redundant information transmission, effectively balancing privacy and
usability. To enhance deployability, the edge model adopts a funnel-shaped
structure with attention mechanisms, strengthening privacy while reducing
computational and storage burdens. Experiments show that, compared to
state-of-the-art defense, SiftFunnel increases reconstruction error by
$\sim$30\%, lowers mutual and effective information metrics by $\geq$50\%, and
reduces edge burdens by almost $20\times$, while maintaining comparable
usability.


http://arxiv.org/abs/2501.00707
Everywhere Attack: Attacking Locally and Globally to Boost Targeted Transferability. (99%)
Hui Zeng; Sanshuai Cui; Biwei Chen; Anjie Peng
  Adversarial examples' (AE) transferability refers to the phenomenon that AEs
crafted with one surrogate model can also fool other models. Notwithstanding
remarkable progress in untargeted transferability, its targeted counterpart
remains challenging. This paper proposes an everywhere scheme to boost targeted
transferability. Our idea is to attack a victim image both globally and
locally. We aim to optimize 'an army of targets' in every local image region
instead of the previous works that optimize a high-confidence target in the
image. Specifically, we split a victim image into non-overlap blocks and
jointly mount a targeted attack on each block. Such a strategy mitigates
transfer failures caused by attention inconsistency between surrogate and
victim models and thus results in stronger transferability. Our approach is
method-agnostic, which means it can be easily combined with existing
transferable attacks for even higher transferability. Extensive experiments on
ImageNet demonstrate that the proposed approach universally improves the
state-of-the-art targeted attacks by a clear margin, e.g., the transferability
of the widely adopted Logit attack can be improved by 28.8%-300%.We also
evaluate the crafted AEs on a real-world platform: Google Cloud Vision. Results
further support the superiority of the proposed method.


http://arxiv.org/abs/2501.00537
Extending XReason: Formal Explanations for Adversarial Detection. (82%)
Amira Jemaa; Adnan Rashid; Sofiene Tahar
  Explainable Artificial Intelligence (XAI) plays an important role in
improving the transparency and reliability of complex machine learning models,
especially in critical domains such as cybersecurity. Despite the prevalence of
heuristic interpretation methods such as SHAP and LIME, these techniques often
lack formal guarantees and may produce inconsistent local explanations. To
fulfill this need, few tools have emerged that use formal methods to provide
formal explanations. Among these, XReason uses a SAT solver to generate formal
instance-level explanation for XGBoost models. In this paper, we extend the
XReason tool to support LightGBM models as well as class-level explanations.
Additionally, we implement a mechanism to generate and detect adversarial
examples in XReason. We evaluate the efficiency and accuracy of our approach on
the CICIDS-2017 dataset, a widely used benchmark for detecting network attacks.


http://arxiv.org/abs/2412.20987
RobustBlack: Challenging Black-Box Adversarial Attacks on State-of-the-Art Defenses. (99%)
Mohamed Djilani; Salah Ghamizi; Maxime Cordy
  Although adversarial robustness has been extensively studied in white-box
settings, recent advances in black-box attacks (including transfer- and
query-based approaches) are primarily benchmarked against weak defenses,
leaving a significant gap in the evaluation of their effectiveness against more
recent and moderate robust models (e.g., those featured in the Robustbench
leaderboard). In this paper, we question this lack of attention from black-box
attacks to robust models. We establish a framework to evaluate the
effectiveness of recent black-box attacks against both top-performing and
standard defense mechanisms, on the ImageNet dataset. Our empirical evaluation
reveals the following key findings: (1) the most advanced black-box attacks
struggle to succeed even against simple adversarially trained models; (2)
robust models that are optimized to withstand strong white-box attacks, such as
AutoAttack, also exhibits enhanced resilience against black-box attacks; and
(3) robustness alignment between the surrogate models and the target model
plays a key factor in the success rate of transfer-based attacks


http://arxiv.org/abs/2412.21164
Adversarial Attack and Defense for LoRa Device Identification and Authentication via Deep Learning. (99%)
Yalin E. Sagduyu; Tugba Erpek
  LoRa provides long-range, energy-efficient communications in Internet of
Things (IoT) applications that rely on Low-Power Wide-Area Network (LPWAN)
capabilities. Despite these merits, concerns persist regarding the security of
LoRa networks, especially in situations where device identification and
authentication are imperative to secure the reliable access to the LoRa
networks. This paper explores a deep learning (DL) approach to tackle these
concerns, focusing on two critical tasks, namely (i) identifying LoRa devices
and (ii) classifying them to legitimate and rogue devices. Deep neural networks
(DNNs), encompassing both convolutional and feedforward neural networks, are
trained for these tasks using actual LoRa signal data. In this setting, the
adversaries may spoof rogue LoRa signals through the kernel density estimation
(KDE) method based on legitimate device signals that are received by the
adversaries. Two cases are considered, (i) training two separate classifiers,
one for each of the two tasks, and (ii) training a multi-task classifier for
both tasks. The vulnerabilities of the resulting DNNs to manipulations in input
samples are studied in form of untargeted and targeted adversarial attacks
using the Fast Gradient Sign Method (FGSM). Individual and common perturbations
are considered against single-task and multi-task classifiers for the LoRa
signal analysis. To provide resilience against such attacks, a defense approach
is presented by increasing the robustness of classifiers with adversarial
training. Results quantify how vulnerable LoRa signal classification tasks are
to adversarial attacks and emphasize the need to fortify IoT applications
against these subtle yet effective threats.


http://arxiv.org/abs/2412.20768
Sample Correlation for Fingerprinting Deep Face Recognition. (98%)
Jiyang Guan; Jian Liang; Yanbo Wang; Ran He
  Face recognition has witnessed remarkable advancements in recent years,
thanks to the development of deep learning techniques.However, an off-the-shelf
face recognition model as a commercial service could be stolen by model
stealing attacks, posing great threats to the rights of the model owner.Model
fingerprinting, as a model stealing detection method, aims to verify whether a
suspect model is stolen from the victim model, gaining more and more attention
nowadays.Previous methods always utilize transferable adversarial examples as
the model fingerprint, but this method is known to be sensitive to adversarial
defense and transfer learning techniques.To address this issue, we consider the
pairwise relationship between samples instead and propose a novel yet simple
model stealing detection method based on SAmple Correlation (SAC).Specifically,
we present SAC-JC that selects JPEG compressed samples as model inputs and
calculates the correlation matrix among their model outputs.Extensive results
validate that SAC successfully defends against various model stealing attacks
in deep face recognition, encompassing face verification and face emotion
recognition, exhibiting the highest performance in terms of AUC, p-value and F1
score.Furthermore, we extend our evaluation of SAC-JC to object recognition
datasets including Tiny-ImageNet and CIFAR10, which also demonstrates the
superior performance of SAC-JC to previous methods.The code will be available
at \url{https://github.com/guanjiyang/SAC_JC}.


http://arxiv.org/abs/2412.20807
Two Heads Are Better Than One: Averaging along Fine-Tuning to Improve Targeted Transferability. (93%)
Hui Zeng; Sanshuai Cui; Biwei Chen; Anjie Peng
  With much longer optimization time than that of untargeted attacks
notwithstanding, the transferability of targeted attacks is still far from
satisfactory. Recent studies reveal that fine-tuning an existing adversarial
example (AE) in feature space can efficiently boost its targeted
transferability. However, existing fine-tuning schemes only utilize the
endpoint and ignore the valuable information in the fine-tuning trajectory.
Noting that the vanilla fine-tuning trajectory tends to oscillate around the
periphery of a flat region of the loss surface, we propose averaging over the
fine-tuning trajectory to pull the crafted AE towards a more centered region.
We compare the proposed method with existing fine-tuning schemes by integrating
them with state-of-the-art targeted attacks in various attacking scenarios.
Experimental results uphold the superiority of the proposed method in boosting
targeted transferability. The code is available at github.com/zengh5/Avg_FT.


http://arxiv.org/abs/2412.20953
GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Search. (76%)
Matan Ben-Tov; Mahmood Sharif
  Dense embedding-based text retrieval$\unicode{x2013}$retrieval of relevant
passages from corpora via deep learning encodings$\unicode{x2013}$has emerged
as a powerful method attaining state-of-the-art search results and popularizing
the use of Retrieval Augmented Generation (RAG). Still, like other search
methods, embedding-based retrieval may be susceptible to search-engine
optimization (SEO) attacks, where adversaries promote malicious content by
introducing adversarial passages to corpora. To faithfully assess and gain
insights into the susceptibility of such systems to SEO, this work proposes the
GASLITE attack, a mathematically principled gradient-based search method for
generating adversarial passages without relying on the corpus content or
modifying the model. Notably, GASLITE's passages (1) carry adversary-chosen
information while (2) achieving high retrieval ranking for a selected query
distribution when inserted to corpora. We use GASLITE to extensively evaluate
retrievers' robustness, testing nine advanced models under varied threat
models, while focusing on realistic adversaries targeting queries on a specific
concept (e.g., a public figure). We found GASLITE consistently outperformed
baselines by $\geq$140% success rate, in all settings. Particularly,
adversaries using GASLITE require minimal effort to manipulate search
results$\unicode{x2013}$by injecting a negligible amount of adversarial
passages ($\leq$0.0001% of the corpus), they could make them visible in the
top-10 results for 61-100% of unseen concept-specific queries against most
evaluated models. Inspecting variance in retrievers' robustness, we identify
key factors that may contribute to models' susceptibility to SEO, including
specific properties in the embedding space's geometry.


http://arxiv.org/abs/2412.20756
Unsupervised dense retrieval with conterfactual contrastive learning. (54%)
Haitian Chen; Qingyao Ai; Xiao Wang; Yiqun Liu; Fen Lin; Qin Liu
  Efficiently retrieving a concise set of candidates from a large document
corpus remains a pivotal challenge in Information Retrieval (IR). Neural
retrieval models, particularly dense retrieval models built with transformers
and pretrained language models, have been popular due to their superior
performance. However, criticisms have also been raised on their lack of
explainability and vulnerability to adversarial attacks. In response to these
challenges, we propose to improve the robustness of dense retrieval models by
enhancing their sensitivity of fine-graned relevance signals. A model achieving
sensitivity in this context should exhibit high variances when documents' key
passages determining their relevance to queries have been modified, while
maintaining low variances for other changes in irrelevant passages. This
sensitivity allows a dense retrieval model to produce robust results with
respect to attacks that try to promote documents without actually increasing
their relevance. It also makes it possible to analyze which part of a document
is actually relevant to a query, and thus improve the explainability of the
retrieval model. Motivated by causality and counterfactual analysis, we propose
a series of counterfactual regularization methods based on game theory and
unsupervised learning with counterfactual passages. Experiments show that, our
method can extract key passages without reliance on the passage-level relevance
annotations. Moreover, the regularized dense retrieval models exhibit
heightened robustness against adversarial attacks, surpassing the
state-of-the-art anti-attack methods.


http://arxiv.org/abs/2412.21123
ExpShield: Safeguarding Web Text from Unauthorized Crawling and Language Modeling Exploitation. (10%)
Ruixuan Liu; Toan Tran; Tianhao Wang; Hongsheng Hu; Shuo Wang; Li Xiong
  As large language models (LLMs) increasingly depend on web-scraped datasets,
concerns arise over their potential to generate verbatim training content with
copyrighted or private information. However, current protections against web
crawling or sample-specific memorization are inherently limited, as they
require compliance from crawlers (e.g., respecting robots.txt) or model
trainers (e.g., applying differential privacy). To empower data owners with
direct control, we propose ExpShiled, a proactive self-defense mechanism that
mitigates sample-specific memorization via imperceptible text perturbations.
This approach requires no external collaboration while maintaining original
readability. To evaluate individual-level defense efficacy, we first propose
the metric of instance exploitation: a zero value indicates perfect defense,
achieved when a protected text's log-perplexity ranking aligns with its
counterfactual untrained ranking. We then reveal and validate the memorization
trigger hypothesis, demonstrating that a model's memorization of a specific
text sample stems primarily from its outlier tokens. Leveraging this insight,
we design targeted perturbations that (1) prioritize inherent trigger tokens
and (2) introduce artificial trigger tokens as pitfalls to disrupt memorization
on the protected sample. Experiments validate our defense across model scales,
languages, vision-to-language tasks, and fine-tuning methods. Even with privacy
backdoors, the Membership Inference Attack (MIA) AUC drops from 0.95 to 0.55,
and instance exploitation approaches zero. This suggests that compared to the
ideal no-misuse scenario, the risk of exposing a text instance remains nearly
unchanged despite its inclusion in training data.


http://arxiv.org/abs/2412.21016
Automated Robustness Testing for LLM-based NLP Software. (2%)
Mingxuan Xiao; Yan Xiao; Shunhui Ji; Hanbo Cai; Lei Xue; Pengcheng Zhang
  Benefiting from the advancements in LLMs, NLP software has undergone rapid
development. Such software is widely employed in various safety-critical tasks,
such as financial sentiment analysis, toxic content moderation, and log
generation. To our knowledge, there are no known automated robustness testing
methods specifically designed for LLM-based NLP software. Given the complexity
of LLMs and the unpredictability of real-world inputs (including prompts and
examples), it is essential to examine the robustness of overall inputs to
ensure the safety of such software.
  To this end, this paper introduces the first AutOmated Robustness Testing
frAmework, AORTA, which reconceptualizes the testing process into a
combinatorial optimization problem. Existing testing methods designed for
DNN-based software can be applied to LLM-based software by AORTA, but their
effectiveness is limited. To address this, we propose a novel testing method
for LLM-based software within AORTA called Adaptive Beam Search. ABS is
tailored for the expansive feature space of LLMs and improves testing
effectiveness through an adaptive beam width and the capability for
backtracking.
  We successfully embed 18 test methods in the designed framework AORTA and
compared the test validity of ABS with three datasets and five threat models.
ABS facilitates a more comprehensive and accurate robustness assessment before
software deployment, with an average test success rate of 86.138%. Compared to
the currently best-performing baseline PWWS, ABS significantly reduces the
computational overhead by up to 3441.895 seconds per successful test case and
decreases the number of queries by 218.762 times on average. Furthermore, test
cases generated by ABS exhibit greater naturalness and transferability.


http://arxiv.org/abs/2412.20804
DELA: A Novel Approach for Detecting Errors Induced by Large Atomic Condition Numbers. (1%)
Youshuai Tan; Zhanwei Zhang; Jinfu Chen; Zishuo Ding; Jifeng Xuan; Weiyi Shang
  Numerical programs form the foundation of modern science and engineering,
providing essential solutions to complex mathematical problems. Therefore,
errors in numerical results would lead to harmful consequences, especially in
safety-critical applications. Since only a few inputs may lead to substantial
errors for numerical programs, it is essential to determine whether a given
input could result in a significant error. Existing researchers tend to use the
results of high-precision programs to assess whether there is a substantial
error, which introduces three main challenges: difficulty of implementation,
existence of potential faults in the detection of numerical errors, and long
execution time.
  To address these limitations, we propose a novel approach named DELA. Our
approach is based on the observation that most numerical errors stem from large
condition numbers in atomic operations (such as subtraction), which then
propagate and accumulate. DELA injects small perturbations into the results of
individual atomic operations within the program and compares the outcomes of
the original program with the perturbed version to detect errors. We evaluate
DELA with datasets from ATOMU and HSED, as well as data from a complex linear
system-solving program. Experimental results demonstrate that we can detect all
the significant errors that were reported by prior research. DELA shows strong
alignment with high-precision programs of ATOMU and HSED, with average Pearson
and Spearman correlations of 0.86 and 0.61. Additionally, DELA effectively
detects significant errors in complex programs, achieving correlation scores of
0.9763 and 0.8993. More importantly, in experiments with ATOMU and HSED, DELA's
perturbed programs run within only 0.13% of the time needed by high-precision
versions; while for the linear system-solving programs, DELA is 73.46 times
faster than the high-precision programs.


http://arxiv.org/abs/2412.20392
Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning. (95%)
Zhifang Zhang; Shuo He; Bingquan Shen; Lei Feng
  Multimodal contrastive learning models (e.g., CLIP) can learn high-quality
representations from large-scale image-text datasets, yet they exhibit
significant vulnerabilities to backdoor attacks, raising serious safety
concerns. In this paper, we disclose that CLIP's vulnerabilities primarily stem
from its excessive encoding of class-irrelevant features, which can compromise
the model's visual feature resistivity to input perturbations, making it more
susceptible to capturing the trigger patterns inserted by backdoor attacks.
Inspired by this finding, we propose Repulsive Visual Prompt Tuning (RVPT), a
novel defense approach that employs specially designed deep visual prompt
tuning and feature-repelling loss to eliminate excessive class-irrelevant
features while simultaneously optimizing cross-entropy loss to maintain clean
accuracy. Unlike existing multimodal backdoor defense methods that typically
require the availability of poisoned data or involve fine-tuning the entire
model, RVPT leverages few-shot downstream clean samples and only tunes a small
number of parameters. Empirical results demonstrate that RVPT tunes only 0.27\%
of the parameters relative to CLIP, yet it significantly outperforms
state-of-the-art baselines, reducing the attack success rate from 67.53\% to
2.76\% against SoTA attacks and effectively generalizing its defensive
capabilities across multiple datasets.


http://arxiv.org/abs/2412.20670
Prototypical Distillation and Debiased Tuning for Black-box Unsupervised Domain Adaptation. (10%)
Jian Liang; Lijun Sheng; Hongmin Liu; Ran He
  Unsupervised domain adaptation aims to transfer knowledge from a related,
label-rich source domain to an unlabeled target domain, thereby circumventing
the high costs associated with manual annotation. Recently, there has been
growing interest in source-free domain adaptation, a paradigm in which only a
pre-trained model, rather than the labeled source data, is provided to the
target domain. Given the potential risk of source data leakage via model
inversion attacks, this paper introduces a novel setting called black-box
domain adaptation, where the source model is accessible only through an API
that provides the predicted label along with the corresponding confidence value
for each query. We develop a two-step framework named $\textbf{Pro}$totypical
$\textbf{D}$istillation and $\textbf{D}$ebiased tun$\textbf{ing}$
($\textbf{ProDDing}$). In the first step, ProDDing leverages both the raw
predictions from the source model and prototypes derived from the target domain
as teachers to distill a customized target model. In the second step, ProDDing
keeps fine-tuning the distilled model by penalizing logits that are biased
toward certain classes. Empirical results across multiple benchmarks
demonstrate that ProDDing outperforms existing black-box domain adaptation
methods. Moreover, in the case of hard-label black-box domain adaptation, where
only predicted labels are available, ProDDing achieves significant improvements
over these methods. Code will be available at
\url{https://github.com/tim-learn/ProDDing/}.


http://arxiv.org/abs/2501.00066
On Adversarial Robustness of Language Models in Transfer Learning. (10%)
Bohdan Turbal; Anastasiia Mazur; Jiaxu Zhao; Mykola Pechenizkiy
  We investigate the adversarial robustness of LLMs in transfer learning
scenarios. Through comprehensive experiments on multiple datasets (MBIB Hate
Speech, MBIB Political Bias, MBIB Gender Bias) and various model architectures
(BERT, RoBERTa, GPT-2, Gemma, Phi), we reveal that transfer learning, while
improving standard performance metrics, often leads to increased vulnerability
to adversarial attacks. Our findings demonstrate that larger models exhibit
greater resilience to this phenomenon, suggesting a complex interplay between
model size, architecture, and adaptation methods. Our work highlights the
crucial need for considering adversarial robustness in transfer learning
scenarios and provides insights into maintaining model security without
compromising performance. These findings have significant implications for the
development and deployment of LLMs in real-world applications where both
performance and robustness are paramount.


http://arxiv.org/abs/2412.20529
Attacks on the neural network and defense methods. (2%)
A. Korenev; G. Belokrylov; B. Lodonova; A. Novokhrestov
  This article will discuss the use of attacks on a neural network trained on
audio data, as well as possible methods of protection against these attacks.
FGSM, PGD and CW attacks, as well as data poisoning, will be considered. Within
the framework of protection, Art-IBM and advertorch libraries will be
considered. The obtained accuracy metrics within the framework of attack
applications are presented


http://arxiv.org/abs/2412.20476
Cut the Deadwood Out: Post-Training Model Purification with Selective Module Substitution. (2%)
Yao Tong; Weijun Li; Xuanli He; Haolan Zhan; Qiongkai Xu
  The success of DNNs often depends on training with large-scale datasets, but
building such datasets is both expensive and challenging. Consequently, public
datasets from open-source platforms like HuggingFace have become popular,
posing significant risks of data poisoning attacks. Existing backdoor defenses
in NLP primarily focus on identifying and removing poisoned samples; however,
purifying a backdoored model with these sample-cleaning approaches typically
requires expensive retraining. Therefore, we propose Greedy Module Substitution
(GMS), which identifies and substitutes ''deadwood'' modules (i.e., components
critical to backdoor pathways) in a backdoored model to purify it. Our method
relaxes the common dependency of prior model purification methods on clean
datasets or clean auxiliary models. When applied to RoBERTa-large under
backdoor attacks, GMS demonstrates strong effectiveness across various
settings, particularly against widely recognized challenging attacks like LWS,
achieving a post-purification attack success rate (ASR) of 9.7% on SST-2
compared to 58.8% for the best baseline approach.


http://arxiv.org/abs/2412.20025
A Robust Adversarial Ensemble with Causal (Feature Interaction) Interpretations for Image Classification. (99%)
Chunheng Zhao; Pierluigi Pisu; Gurcan Comert; Negash Begashaw; Varghese Vaidyan; Nina Christine Hubig
  Deep learning-based discriminative classifiers, despite their remarkable
success, remain vulnerable to adversarial examples that can mislead model
predictions. While adversarial training can enhance robustness, it fails to
address the intrinsic vulnerability stemming from the opaque nature of these
black-box models. We present a deep ensemble model that combines discriminative
features with generative models to achieve both high accuracy and adversarial
robustness. Our approach integrates a bottom-level pre-trained discriminative
network for feature extraction with a top-level generative classification
network that models adversarial input distributions through a deep latent
variable model. Using variational Bayes, our model achieves superior robustness
against white-box adversarial attacks without adversarial training. Extensive
experiments on CIFAR-10 and CIFAR-100 demonstrate our model's superior
adversarial robustness. Through evaluations using counterfactual metrics and
feature interaction-based metrics, we establish correlations between model
interpretability and adversarial robustness. Additionally, preliminary results
on Tiny-ImageNet validate our approach's scalability to more complex datasets,
offering a practical solution for developing robust image classification
models.


http://arxiv.org/abs/2412.20087
On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs. (13%)
Atmane Ayoub Mansour Bahar; Ahmad Samer Wazan
  This research investigates the effectiveness of established vulnerability
metrics, such as the Common Vulnerability Scoring System (CVSS), in evaluating
attacks against Large Language Models (LLMs), with a focus on Adversarial
Attacks (AAs). The study explores the influence of both general and specific
metric factors in determining vulnerability scores, providing new perspectives
on potential enhancements to these metrics.
  This study adopts a quantitative approach, calculating and comparing the
coefficient of variation of vulnerability scores across 56 adversarial attacks
on LLMs. The attacks, sourced from various research papers, and obtained
through online databases, were evaluated using multiple vulnerability metrics.
Scores were determined by averaging the values assessed by three distinct LLMs.
The results indicate that existing scoring-systems yield vulnerability scores
with minimal variation across different attacks, suggesting that many of the
metric factors are inadequate for assessing adversarial attacks on LLMs. This
is particularly true for context-specific factors or those with predefined
value sets, such as those in CVSS. These findings support the hypothesis that
current vulnerability metrics, especially those with rigid values, are limited
in evaluating AAs on LLMs, highlighting the need for the development of more
flexible, generalized metrics tailored to such attacks.
  This research offers a fresh analysis of the effectiveness and applicability
of established vulnerability metrics, particularly in the context of
Adversarial Attacks on Large Language Models, both of which have gained
significant attention in recent years. Through extensive testing and
calculations, the study underscores the limitations of these metrics and opens
up new avenues for improving and refining vulnerability assessment frameworks
specifically tailored for LLMs.


http://arxiv.org/abs/2412.20086
MAFT: Efficient Model-Agnostic Fairness Testing for Deep Neural Networks via Zero-Order Gradient Search. (9%)
Zhaohui Wang; Min Zhang; Jingran Yang; Bojie Shao; Min Zhang
  Deep neural networks (DNNs) have shown powerful performance in various
applications and are increasingly being used in decision-making systems.
However, concerns about fairness in DNNs always persist. Some efficient
white-box fairness testing methods about individual fairness have been
proposed. Nevertheless, the development of black-box methods has stagnated, and
the performance of existing methods is far behind that of white-box methods. In
this paper, we propose a novel black-box individual fairness testing method
called Model-Agnostic Fairness Testing (MAFT). By leveraging MAFT,
practitioners can effectively identify and address discrimination in DL models,
regardless of the specific algorithm or architecture employed. Our approach
adopts lightweight procedures such as gradient estimation and attribute
perturbation rather than non-trivial procedures like symbol execution,
rendering it significantly more scalable and applicable than existing methods.
We demonstrate that MAFT achieves the same effectiveness as state-of-the-art
white-box methods whilst improving the applicability to large-scale networks.
Compared to existing black-box approaches, our approach demonstrates
distinguished performance in discovering fairness violations w.r.t
effectiveness (approximately 14.69 times) and efficiency (approximately 32.58
times).


http://arxiv.org/abs/2501.00055
LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models. (2%)
Miao Yu; Junfeng Fang; Yingjie Zhou; Xing Fan; Kun Wang; Shirui Pan; Qingsong Wen
  While safety-aligned large language models (LLMs) are increasingly used as
the cornerstone for powerful systems such as multi-agent frameworks to solve
complex real-world problems, they still suffer from potential adversarial
queries, such as jailbreak attacks, which attempt to induce harmful content.
Researching attack methods allows us to better understand the limitations of
LLM and make trade-offs between helpfulness and safety. However, existing
jailbreak attacks are primarily based on opaque optimization techniques (e.g.
token-level gradient descent) and heuristic search methods like LLM refinement,
which fall short in terms of transparency, transferability, and computational
cost. In light of these limitations, we draw inspiration from the evolution and
infection processes of biological viruses and propose LLM-Virus, a jailbreak
attack method based on evolutionary algorithm, termed evolutionary jailbreak.
LLM-Virus treats jailbreak attacks as both an evolutionary and transfer
learning problem, utilizing LLMs as heuristic evolutionary operators to ensure
high attack efficiency, transferability, and low time cost. Our experimental
results on multiple safety benchmarks show that LLM-Virus achieves competitive
or even superior performance compared to existing attack methods.


http://arxiv.org/abs/2412.20154
Defending Against Network Attacks for Secure AI Agent Migration in Vehicular Metaverses. (1%)
Xinru Wen; Jinbo Wen; Ming Xiao; Jiawen Kang; Tao Zhang; Xiaohuan Li; Chuanxi Chen; Dusit Niyato
  Vehicular metaverses, blending traditional vehicular networks with metaverse
technology, are expected to revolutionize fields such as autonomous driving. As
virtual intelligent assistants in vehicular metaverses, Artificial Intelligence
(AI) agents powered by large language models can create immersive 3D virtual
spaces for passengers to enjoy on-broad vehicular applications and services. To
provide users with seamless and engaging virtual interactions, resource-limited
vehicles offload AI agents to RoadSide Units (RSUs) with adequate communication
and computational capabilities. Due to the mobility of vehicles and the limited
coverage of RSUs, AI agents need to migrate from one RSU to another RSU.
However, potential network attacks pose significant challenges to ensuring
reliable and efficient AI agent migration. In this paper, we first explore
specific network attacks including traffic-based attacks (i.e., DDoS attacks)
and infrastructure-based attacks (i.e., malicious RSU attacks). Then, we model
the AI agent migration process as a Partially Observable Markov Decision
Process (POMDP) and apply multi-agent proximal policy optimization algorithms
to mitigate DDoS attacks. In addition, we propose a trust assessment mechanism
to counter malicious RSU attacks. Numerical results validate that the proposed
solutions effectively defend against these network attacks and reduce the total
latency of AI agent migration by approximately 43.3%.


http://arxiv.org/abs/2412.19947
Standard-Deviation-Inspired Regularization for Improving Adversarial Robustness. (99%)
Olukorede Fakorede; Modeste Atsague; Jin Tian
  Adversarial Training (AT) has been demonstrated to improve the robustness of
deep neural networks (DNNs) against adversarial attacks. AT is a min-max
optimization procedure where in adversarial examples are generated to train a
more robust DNN. The inner maximization step of AT increases the losses of
inputs with respect to their actual classes. The outer minimization involves
minimizing the losses on the adversarial examples obtained from the inner
maximization. This work proposes a standard-deviation-inspired (SDI)
regularization term to improve adversarial robustness and generalization. We
argue that the inner maximization in AT is similar to minimizing a modified
standard deviation of the model's output probabilities. Moreover, we suggest
that maximizing this modified standard deviation can complement the outer
minimization of the AT framework. To support our argument, we experimentally
show that the SDI measure can be used to craft adversarial examples.
Additionally, we demonstrate that combining the SDI regularization term with
existing AT variants enhances the robustness of DNNs against stronger attacks,
such as CW and Auto-attack, and improves generalization.


http://arxiv.org/abs/2412.20006
Adversarial Robustness for Deep Learning-based Wildfire Detection Models. (98%)
Ryo Ide; Lei Yang
  Smoke detection using Deep Neural Networks (DNNs) is an effective approach
for early wildfire detection. However, because smoke is temporally and
spatially anomalous, there are limitations in collecting sufficient training
data. This raises overfitting and bias concerns in existing DNN-based wildfire
detection models. Thus, we introduce WARP (Wildfire Adversarial Robustness
Procedure), the first model-agnostic framework for evaluating the adversarial
robustness of DNN-based wildfire detection models. WARP addresses limitations
in smoke image diversity using global and local adversarial attack methods. The
global attack method uses image-contextualized Gaussian noise, while the local
attack method uses patch noise injection, tailored to address critical aspects
of wildfire detection. Leveraging WARP's model-agnostic capabilities, we assess
the adversarial robustness of real-time Convolutional Neural Networks (CNNs)
and Transformers. The analysis revealed valuable insights into the models'
limitations. Specifically, the global attack method demonstrates that the
Transformer model has more than 70\% precision degradation than the CNN against
global noise. In contrast, the local attack method shows that both models are
susceptible to cloud image injections when detecting smoke-positive instances,
suggesting a need for model improvements through data augmentation. WARP's
comprehensive robustness analysis contributed to the development of
wildfire-specific data augmentation strategies, marking a step toward
practicality.


http://arxiv.org/abs/2412.19747
Enhancing Adversarial Robustness of Deep Neural Networks Through Supervised Contrastive Learning. (96%)
Longwei Wang; Navid Nayyem; Abdullah Rakin
  Adversarial attacks exploit the vulnerabilities of convolutional neural
networks by introducing imperceptible perturbations that lead to
misclassifications, exposing weaknesses in feature representations and decision
boundaries. This paper presents a novel framework combining supervised
contrastive learning and margin-based contrastive loss to enhance adversarial
robustness. Supervised contrastive learning improves the structure of the
feature space by clustering embeddings of samples within the same class and
separating those from different classes. Margin-based contrastive loss,
inspired by support vector machines, enforces explicit constraints to create
robust decision boundaries with well-defined margins. Experiments on the
CIFAR-100 dataset with a ResNet-18 backbone demonstrate robustness performance
improvements in adversarial accuracy under Fast Gradient Sign Method attacks.


http://arxiv.org/abs/2412.19523
Attribution for Enhanced Explanation with Transferable Adversarial eXploration. (92%)
Zhiyu Zhu; Jiayu Zhang; Zhibo Jin; Huaming Chen; Jianlong Zhou; Fang Chen
  The interpretability of deep neural networks is crucial for understanding
model decisions in various applications, including computer vision.
AttEXplore++, an advanced framework built upon AttEXplore, enhances attribution
by incorporating transferable adversarial attack methods such as MIG and GRA,
significantly improving the accuracy and robustness of model explanations. We
conduct extensive experiments on five models, including CNNs (Inception-v3,
ResNet-50, VGG16) and vision transformers (MaxViT-T, ViT-B/16), using the
ImageNet dataset. Our method achieves an average performance improvement of
7.57\% over AttEXplore and 32.62\% compared to other state-of-the-art
interpretability algorithms. Using insertion and deletion scores as evaluation
metrics, we show that adversarial transferability plays a vital role in
enhancing attribution results. Furthermore, we explore the impact of
randomness, perturbation rate, noise amplitude, and diversity probability on
attribution performance, demonstrating that AttEXplore++ provides more stable
and reliable explanations across various models. We release our code at:
https://anonymous.4open.science/r/ATTEXPLOREP-8435/


http://arxiv.org/abs/2412.19354
Federated Hybrid Training and Self-Adversarial Distillation: Towards Robust Edge Networks. (45%)
Yu Qiao; Apurba Adhikary; Kitae Kim; Eui-Nam Huh; Zhu Han; Choong Seon Hong
  Federated learning (FL) is a distributed training technology that enhances
data privacy in mobile edge networks by allowing data owners to collaborate
without transmitting raw data to the edge server. However, data heterogeneity
and adversarial attacks pose challenges to develop an unbiased and robust
global model for edge deployment. To address this, we propose Federated hyBrid
Adversarial training and self-adversarial disTillation (FedBAT), a new
framework designed to improve both robustness and generalization of the global
model. FedBAT seamlessly integrates hybrid adversarial training and
self-adversarial distillation into the conventional FL framework from data
augmentation and feature distillation perspectives. From a data augmentation
perspective, we propose hybrid adversarial training to defend against
adversarial attacks by balancing accuracy and robustness through a weighted
combination of standard and adversarial training. From a feature distillation
perspective, we introduce a novel augmentation-invariant adversarial
distillation method that aligns local adversarial features of augmented images
with their corresponding unbiased global clean features. This alignment can
effectively mitigate bias from data heterogeneity while enhancing both the
robustness and generalization of the global model. Extensive experimental
results across multiple datasets demonstrate that FedBAT yields comparable or
superior performance gains in improving robustness while maintaining accuracy
compared to several baselines.


http://arxiv.org/abs/2412.19394
An Engorgio Prompt Makes Large Language Model Babble on. (16%)
Jianshuo Dong; Ziyuan Zhang; Qingjie Zhang; Tianwei Zhang; Hao Wang; Hewu Li; Qi Li; Chao Zhang; Ke Xu; Han Qiu
  Auto-regressive large language models (LLMs) have yielded impressive
performance in many real-world tasks. However, the new paradigm of these LLMs
also exposes novel threats. In this paper, we explore their vulnerability to
inference cost attacks, where a malicious user crafts Engorgio prompts to
intentionally increase the computation cost and latency of the inference
process. We design Engorgio, a novel methodology, to efficiently generate
adversarial Engorgio prompts to affect the target LLM's service availability.
Engorgio has the following two technical contributions. (1) We employ a
parameterized distribution to track LLMs' prediction trajectory. (2) Targeting
the auto-regressive nature of LLMs' inference process, we propose novel loss
functions to stably suppress the appearance of the <EOS> token, whose
occurrence will interrupt the LLM's generation process. We conduct extensive
experiments on 13 open-sourced LLMs with parameters ranging from 125M to 30B.
The results show that Engorgio prompts can successfully induce LLMs to generate
abnormally long outputs (i.e., roughly 2-13$\times$ longer to reach 90%+ of the
output length limit) in a white-box scenario and our real-world experiment
demonstrates Engergio's threat to LLM service with limited computing resources.
The code is released at: https://github.com/jianshuod/Engorgio-prompt.


http://arxiv.org/abs/2412.19311
xSRL: Safety-Aware Explainable Reinforcement Learning -- Safety as a Product of Explainability. (2%)
Risal Shahriar Shefin; Md Asifur Rahman; Thai Le; Sarra Alqahtani
  Reinforcement learning (RL) has shown great promise in simulated
environments, such as games, where failures have minimal consequences. However,
the deployment of RL agents in real-world systems such as autonomous vehicles,
robotics, UAVs, and medical devices demands a higher level of safety and
transparency, particularly when facing adversarial threats. Safe RL algorithms
have been developed to address these concerns by optimizing both task
performance and safety constraints. However, errors are inevitable, and when
they occur, it is essential that the RL agents can also explain their actions
to human operators. This makes trust in the safety mechanisms of RL systems
crucial for effective deployment. Explainability plays a key role in building
this trust by providing clear, actionable insights into the agent's
decision-making process, ensuring that safety-critical decisions are well
understood. While machine learning (ML) has seen significant advances in
interpretability and visualization, explainability methods for RL remain
limited. Current tools fail to address the dynamic, sequential nature of RL and
its needs to balance task performance with safety constraints over time. The
re-purposing of traditional ML methods, such as saliency maps, is inadequate
for safety-critical RL applications where mistakes can result in severe
consequences. To bridge this gap, we propose xSRL, a framework that integrates
both local and global explanations to provide a comprehensive understanding of
RL agents' behavior. xSRL also enables developers to identify policy
vulnerabilities through adversarial attacks, offering tools to debug and patch
agents without retraining. Our experiments and user studies demonstrate xSRL's
effectiveness in increasing safety in RL systems, making them more reliable and
trustworthy for real-world deployment. Code is available at
https://github.com/risal-shefin/xSRL.


http://arxiv.org/abs/2412.18815
Distortion-Aware Adversarial Attacks on Bounding Boxes of Object Detectors. (99%)
Pham Phuc; Son Vuong; Khang Nguyen; Tuan Dang
  Deep learning-based object detection has become ubiquitous in the last decade
due to its high accuracy in many real-world applications. With this growing
trend, these models are interested in being attacked by adversaries, with most
of the results being on classifiers, which do not match the context of
practical object detection. In this work, we propose a novel method to fool
object detectors, expose the vulnerability of state-of-the-art detectors, and
promote later works to build more robust detectors to adversarial examples. Our
method aims to generate adversarial images by perturbing object confidence
scores during training, which is crucial in predicting confidence for each
class in the testing phase. Herein, we provide a more intuitive technique to
embed additive noises based on detected objects' masks and the training loss
with distortion control over the original image by leveraging the gradient of
iterative images. To verify the proposed method, we perform adversarial attacks
against different object detectors, including the most recent state-of-the-art
models like YOLOv8, Faster R-CNN, RetinaNet, and Swin Transformer. We also
evaluate our technique on MS COCO 2017 and PASCAL VOC 2012 datasets and analyze
the trade-off between success attack rate and image distortion. Our experiments
show that the achievable success attack rate is up to $100$\% and up to $98$\%
when performing white-box and black-box attacks, respectively. The source code
and relevant documentation for this work are available at the following link:
https://github.com/anonymous20210106/attack_detector


http://arxiv.org/abs/2412.18844
Improving Integrated Gradient-based Transferable Adversarial Examples by Refining the Integration Path. (98%)
Yuchen Ren; Zhengyu Zhao; Chenhao Lin; Bo Yang; Lu Zhou; Zhe Liu; Chao Shen
  Transferable adversarial examples are known to cause threats in practical,
black-box attack scenarios. A notable approach to improving transferability is
using integrated gradients (IG), originally developed for model
interpretability. In this paper, we find that existing IG-based attacks have
limited transferability due to their naive adoption of IG in model
interpretability. To address this limitation, we focus on the IG integration
path and refine it in three aspects: multiplicity, monotonicity, and diversity,
supported by theoretical analyses. We propose the Multiple Monotonic
Diversified Integrated Gradients (MuMoDIG) attack, which can generate highly
transferable adversarial examples on different CNN and ViT models and defenses.
Experiments validate that MuMoDIG outperforms the latest IG-based attack by up
to 37.3\% and other state-of-the-art attacks by 8.4\%. In general, our study
reveals that migrating established techniques to improve transferability may
require non-trivial efforts. Code is available at
\url{https://github.com/RYC-98/MuMoDIG}.


http://arxiv.org/abs/2412.19015
Imperceptible Adversarial Attacks on Point Clouds Guided by Point-to-Surface Field. (83%)
Keke Tang; Weiyao Ke; Weilong Peng; Xiaofei Wang; Ziyong Du; Zhize Wu; Peican Zhu; Zhihong Tian
  Adversarial attacks on point clouds are crucial for assessing and improving
the adversarial robustness of 3D deep learning models. Traditional solutions
strictly limit point displacement during attacks, making it challenging to
balance imperceptibility with adversarial effectiveness. In this paper, we
attribute the inadequate imperceptibility of adversarial attacks on point
clouds to deviations from the underlying surface. To address this, we introduce
a novel point-to-surface (P2S) field that adjusts adversarial perturbation
directions by dragging points back to their original underlying surface.
Specifically, we use a denoising network to learn the gradient field of the
logarithmic density function encoding the shape's surface, and apply a
distance-aware adjustment to perturbation directions during attacks, thereby
enhancing imperceptibility. Extensive experiments show that adversarial attacks
guided by our P2S field are more imperceptible, outperforming state-of-the-art
methods.


http://arxiv.org/abs/2412.18886
Adversarial Training for Graph Neural Networks via Graph Subspace Energy Optimization. (75%)
Ganlin Liu; Ziling Liang; Xiaowei Huang; Xinping Yi; Shi Jin
  Despite impressive capability in learning over graph-structured data, graph
neural networks (GNN) suffer from adversarial topology perturbation in both
training and inference phases. While adversarial training has demonstrated
remarkable effectiveness in image classification tasks, its suitability for GNN
models has been doubted until a recent advance that shifts the focus from
transductive to inductive learning. Still, GNN robustness in the inductive
setting is under-explored, and it calls for deeper understanding of GNN
adversarial training. To this end, we propose a new concept of graph subspace
energy (GSE) -- a generalization of graph energy that measures graph stability
-- of the adjacency matrix, as an indicator of GNN robustness against topology
perturbations. To further demonstrate the effectiveness of such concept, we
propose an adversarial training method with the perturbed graphs generated by
maximizing the GSE regularization term, referred to as AT-GSE. To deal with the
local and global topology perturbations raised respectively by LRBCD and PRBCD,
we employ randomized SVD (RndSVD) and Nystrom low-rank approximation to favor
the different aspects of the GSE terms. An extensive set of experiments shows
that AT-GSE outperforms consistently the state-of-the-art GNN adversarial
training methods over different homophily and heterophily datasets in terms of
adversarial accuracy, whilst more surprisingly achieving a superior clean
accuracy on non-perturbed graphs.


http://arxiv.org/abs/2412.18952
Bridging Interpretability and Robustness Using LIME-Guided Model Refinement. (69%)
Navid Nayyem; Abdullah Rakin; Longwei Wang
  This paper explores the intricate relationship between interpretability and
robustness in deep learning models. Despite their remarkable performance across
various tasks, deep learning models often exhibit critical vulnerabilities,
including susceptibility to adversarial attacks, over-reliance on spurious
correlations, and a lack of transparency in their decision-making processes. To
address these limitations, we propose a novel framework that leverages Local
Interpretable Model-Agnostic Explanations (LIME) to systematically enhance
model robustness. By identifying and mitigating the influence of irrelevant or
misleading features, our approach iteratively refines the model, penalizing
reliance on these features during training. Empirical evaluations on multiple
benchmark datasets demonstrate that LIME-guided refinement not only improves
interpretability but also significantly enhances resistance to adversarial
perturbations and generalization to out-of-distribution data.


http://arxiv.org/abs/2412.18975
Injecting Bias into Text Classification Models using Backdoor Attacks. (15%)
A. Dilara Yavuz; M. Emre Gursoy
  The rapid growth of natural language processing (NLP) and pre-trained
language models have enabled accurate text classification in a variety of
settings. However, text classification models are susceptible to backdoor
attacks, where an attacker embeds a trigger into the victim model to make the
model predict attacker-desired labels in targeted scenarios. In this paper, we
propose to utilize backdoor attacks for a new purpose: bias injection. We
develop a backdoor attack in which a subset of the training dataset is poisoned
to associate strong male actors with negative sentiment. We execute our attack
on two popular text classification datasets (IMDb and SST) and seven different
models ranging from traditional Doc2Vec-based models to LSTM networks and
modern transformer-based BERT and RoBERTa models. Our results show that the
reduction in backdoored models' benign classification accuracy is limited,
implying that our attacks remain stealthy, whereas the models successfully
learn to associate strong male actors with negative sentiment (100% attack
success rate with >= 3% poison rate). Attacks on BERT and RoBERTa are
particularly more stealthy and effective, demonstrating an increased risk of
using modern and larger models. We also measure the generalizability of our
bias injection by proposing two metrics: (i) U-BBSR which uses previously
unseen words when measuring attack success, and (ii) P-BBSR which measures
attack success using paraphrased test samples. U-BBSR and P-BBSR results show
that the bias injected by our attack can go beyond memorizing a trigger phrase.


http://arxiv.org/abs/2412.18791
Protective Perturbations against Unauthorized Data Usage in Diffusion-based Image Generation. (10%)
Sen Peng; Jijia Yang; Mingyue Wang; Jianfei He; Xiaohua Jia
  Diffusion-based text-to-image models have shown immense potential for various
image-related tasks. However, despite their prominence and popularity,
customizing these models using unauthorized data also brings serious privacy
and intellectual property issues. Existing methods introduce protective
perturbations based on adversarial attacks, which are applied to the
customization samples. In this systematization of knowledge, we present a
comprehensive survey of protective perturbation methods designed to prevent
unauthorized data usage in diffusion-based image generation. We establish the
threat model and categorize the downstream tasks relevant to these methods,
providing a detailed analysis of their designs. We also propose a completed
evaluation framework for these perturbation techniques, aiming to advance
research in this field.


http://arxiv.org/abs/2412.19037
CL-attack: Textual Backdoor Attacks via Cross-Lingual Triggers. (9%)
Jingyi Zheng; Tianyi Hu; Tianshuo Cong; Xinlei He
  Backdoor attacks significantly compromise the security of large language
models by triggering them to output specific and controlled content. Currently,
triggers for textual backdoor attacks fall into two categories: fixed-token
triggers and sentence-pattern triggers. However, the former are typically easy
to identify and filter, while the latter, such as syntax and style, do not
apply to all original samples and may lead to semantic shifts. In this paper,
inspired by cross-lingual (CL) prompts of LLMs in real-world scenarios, we
propose a higher-dimensional trigger method at the paragraph level, namely
CL-attack. CL-attack injects the backdoor by using texts with specific
structures that incorporate multiple languages, thereby offering greater
stealthiness and universality compared to existing backdoor attack techniques.
Extensive experiments on different tasks and model architectures demonstrate
that CL-attack can achieve nearly 100% attack success rate with a low poisoning
rate in both classification and generation tasks. We also empirically show that
the CL-attack is more robust against current major defense methods compared to
baseline backdoor attacks. Additionally, to mitigate CL-attack, we further
develop a new defense called TranslateDefense, which can partially mitigate the
impact of CL-attack.


http://arxiv.org/abs/2412.18706
SurvAttack: Black-Box Attack On Survival Models through Ontology-Informed EHR Perturbation. (99%)
Mohsen Nayebi Kerdabadi; Arya Hadizadeh Moghaddam; Bin Liu; Mei Liu; Zijun Yao
  Survival analysis (SA) models have been widely studied in mining electronic
health records (EHRs), particularly in forecasting the risk of critical
conditions for prioritizing high-risk patients. However, their vulnerability to
adversarial attacks is much less explored in the literature. Developing
black-box perturbation algorithms and evaluating their impact on
state-of-the-art survival models brings two benefits to medical applications.
First, it can effectively evaluate the robustness of models in pre-deployment
testing. Also, exploring how subtle perturbations would result in significantly
different outcomes can provide counterfactual insights into the clinical
interpretation of model prediction. In this work, we introduce SurvAttack, a
novel black-box adversarial attack framework leveraging subtle clinically
compatible, and semantically consistent perturbations on longitudinal EHRs to
degrade survival models' predictive performance. We specifically develop a
greedy algorithm to manipulate medical codes with various adversarial actions
throughout a patient's medical history. Then, these adversarial actions are
prioritized using a composite scoring strategy based on multi-aspect
perturbation quality, including saliency, perturbation stealthiness, and
clinical meaningfulness. The proposed adversarial EHR perturbation algorithm is
then used in an efficient SA-specific strategy to attack a survival model when
estimating the temporal ranking of survival urgency for patients. To
demonstrate the significance of our work, we conduct extensive experiments,
including baseline comparisons, explainability analysis, and case studies. The
experimental results affirm our research's effectiveness in illustrating the
vulnerabilities of patient survival models, model interpretation, and
ultimately contributing to healthcare quality.


http://arxiv.org/abs/2412.18718
Evaluating the Adversarial Robustness of Detection Transformers. (99%)
Amirhossein Nazeri; Chunheng Zhao; Pierluigi Pisu
  Robust object detection is critical for autonomous driving and mobile
robotics, where accurate detection of vehicles, pedestrians, and obstacles is
essential for ensuring safety. Despite the advancements in object detection
transformers (DETRs), their robustness against adversarial attacks remains
underexplored. This paper presents a comprehensive evaluation of DETR model and
its variants under both white-box and black-box adversarial attacks, using the
MS-COCO and KITTI datasets to cover general and autonomous driving scenarios.
We extend prominent white-box attack methods (FGSM, PGD, and CW) to assess DETR
vulnerability, demonstrating that DETR models are significantly susceptible to
adversarial attacks, similar to traditional CNN-based detectors. Our extensive
transferability analysis reveals high intra-network transferability among DETR
variants, but limited cross-network transferability to CNN-based models.
Additionally, we propose a novel untargeted attack designed specifically for
DETR, exploiting its intermediate loss functions to induce misclassification
with minimal perturbations. Visualizations of self-attention feature maps
provide insights into how adversarial attacks affect the internal
representations of DETR models. These findings reveal critical vulnerabilities
in detection transformers under standard adversarial attacks, emphasizing the
need for future research to enhance the robustness of transformer-based object
detectors in safety-critical applications.


http://arxiv.org/abs/2412.18196
Robustness-aware Automatic Prompt Optimization. (98%)
Zeru Shi; Zhenting Wang; Yongye Su; Weidi Luo; Fan Yang; Yongfeng Zhang
  The performance of Large Language Models (LLMs) is based on the quality of
the prompts and the semantic and structural integrity information of the input
data. However, current prompt generation methods primarily focus on generating
prompts for clean input data, often overlooking the impact of perturbed inputs
on prompt performance. To address this limitation, we propose BATprompt (By
Adversarial Training prompt), a novel method for prompt generation designed to
withstand input perturbations (such as typos in the input). Inspired by
adversarial training techniques, BATprompt demonstrates strong performance on a
variety of perturbed tasks through a two-step process: adversarial perturbation
and iterative optimization on unperturbed input via LLM. Unlike conventional
adversarial attack methods, BATprompt avoids reliance on real gradients or
model parameters. Instead, it leverages the advanced reasoning, language
understanding and self reflection capabilities of LLMs to simulate gradients,
guiding the generation of adversarial perturbations and optimizing prompt
performance. In our experiments, we evaluate BATprompt on multiple datasets
across both language understanding and generation tasks. The results indicate
that BATprompt outperforms existing prompt generation methods, delivering
superior robustness and performance under diverse perturbation scenarios.


http://arxiv.org/abs/2412.18770
Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models. (98%)
Yu-An Liu; Ruqing Zhang; Jiafeng Guo; Rijke Maarten de; Yixing Fan; Xueqi Cheng
  Neural ranking models (NRMs) have been shown to be highly effective in terms
of retrieval performance. Unfortunately, they have also displayed a higher
degree of sensitivity to attacks than previous generation models. To help
expose and address this lack of robustness, we introduce a novel ranking attack
framework named Attack-in-the-Chain, which tracks interactions between large
language models (LLMs) and NRMs based on chain-of-thought (CoT) prompting to
generate adversarial examples under black-box settings. Our approach starts by
identifying anchor documents with higher ranking positions than the target
document as nodes in the reasoning chain. We then dynamically assign the number
of perturbation words to each node and prompt LLMs to execute attacks. Finally,
we verify the attack performance of all nodes at each reasoning step and
proceed to generate the next reasoning step. Empirical results on two web
search benchmarks show the effectiveness of our method.


http://arxiv.org/abs/2412.18218
On the Effectiveness of Adversarial Training on Malware Classifiers. (96%)
Hamid Bostani; Jacopo Cortellazzi; Daniel Arp; Fabio Pierazzi; Veelasha Moonsamy; Lorenzo Cavallaro
  Adversarial Training (AT) has been widely applied to harden learning-based
classifiers against adversarial evasive attacks. However, its effectiveness in
identifying and strengthening vulnerable areas of the model's decision space
while maintaining high performance on clean data of malware classifiers remains
an under-explored area. In this context, the robustness that AT achieves has
often been assessed against unrealistic or weak adversarial attacks, which
negatively affect performance on clean data and are arguably no longer threats.
Previous work seems to suggest robustness is a task-dependent property of AT.
We instead argue it is a more complex problem that requires exploring AT and
the intertwined roles played by certain factors within data, feature
representations, classifiers, and robust optimization settings, as well as
proper evaluation factors, such as the realism of evasion attacks, to gain a
true sense of AT's effectiveness. In our paper, we address this gap by
systematically exploring the role such factors have in hardening malware
classifiers through AT. Contrary to recent prior work, a key observation of our
research and extensive experiments confirm the hypotheses that all such factors
influence the actual effectiveness of AT, as demonstrated by the varying
degrees of success from our empirical analysis. We identify five evaluation
pitfalls that affect state-of-the-art studies and summarize our insights in ten
takeaways to draw promising research directions toward better understanding the
factors' settings under which adversarial training works at best.


http://arxiv.org/abs/2412.18262
Efficient Contrastive Explanations on Demand. (82%)
Yacine Izza; Joao Marques-Silva
  Recent work revealed a tight connection between adversarial robustness and
restricted forms of symbolic explanations, namely distance-based (formal)
explanations. This connection is significant because it represents a first step
towards making the computation of symbolic explanations as efficient as
deciding the existence of adversarial examples, especially for highly complex
machine learning (ML) models. However, a major performance bottleneck remains,
because of the very large number of features that ML models may possess, in
particular for deep neural networks. This paper proposes novel algorithms to
compute the so-called contrastive explanations for ML models with a large
number of features, by leveraging on adversarial robustness. Furthermore, the
paper also proposes novel algorithms for listing explanations and finding
smallest contrastive explanations. The experimental results demonstrate the
performance gains achieved by the novel algorithms proposed in this paper.


http://arxiv.org/abs/2412.18507
An Empirical Analysis of Federated Learning Models Subject to Label-Flipping Adversarial Attack. (73%)
Kunal Bhatnagar; Sagana Chattanathan; Angela Dang; Bhargav Eranki; Ronnit Rana; Charan Sridhar; Siddharth Vedam; Angie Yao; Mark Stamp
  In this paper, we empirically analyze adversarial attacks on selected
federated learning models. The specific learning models considered are
Multinominal Logistic Regression (MLR), Support Vector Classifier (SVC),
Multilayer Perceptron (MLP), Convolution Neural Network (CNN), %Recurrent
Neural Network (RNN), Random Forest, XGBoost, and Long Short-Term Memory
(LSTM). For each model, we simulate label-flipping attacks, experimenting
extensively with 10 federated clients and 100 federated clients. We vary the
percentage of adversarial clients from 10% to 100% and, simultaneously, the
percentage of labels flipped by each adversarial client is also varied from 10%
to 100%. Among other results, we find that models differ in their inherent
robustness to the two vectors in our label-flipping attack, i.e., the
percentage of adversarial clients, and the percentage of labels flipped by each
adversarial client. We discuss the potential practical implications of our
results.


http://arxiv.org/abs/2412.18370
Unveiling the Threat of Fraud Gangs to Graph Neural Networks: Multi-Target Graph Injection Attacks Against GNN-Based Fraud Detectors. (41%)
Jinhyeok Choi; Heehyeon Kim; Joyce Jiyoung Whang
  Graph neural networks (GNNs) have emerged as an effective tool for fraud
detection, identifying fraudulent users, and uncovering malicious behaviors.
However, attacks against GNN-based fraud detectors and their risks have rarely
been studied, thereby leaving potential threats unaddressed. Recent findings
suggest that frauds are increasingly organized as gangs or groups. In this
work, we design attack scenarios where fraud gangs aim to make their fraud
nodes misclassified as benign by camouflaging their illicit activities in
collusion. Based on these scenarios, we study adversarial attacks against
GNN-based fraud detectors by simulating attacks of fraud gangs in three
real-world fraud cases: spam reviews, fake news, and medical insurance frauds.
We define these attacks as multi-target graph injection attacks and propose
MonTi, a transformer-based Multi-target one-Time graph injection attack model.
MonTi simultaneously generates attributes and edges of all attack nodes with a
transformer encoder, capturing interdependencies between attributes and edges
more effectively than most existing graph injection attack methods that
generate these elements sequentially. Additionally, MonTi adaptively allocates
the degree budget for each attack node to explore diverse injection structures
involving target, candidate, and attack nodes, unlike existing methods that fix
the degree budget across all attack nodes. Experiments show that MonTi
outperforms the state-of-the-art graph injection attack methods on five
real-world graphs.


http://arxiv.org/abs/2412.18171
Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models. (38%)
Xiaomeng Hu; Pin-Yu Chen; Tsung-Yi Ho
  Large Language Models (LLMs) are increasingly being integrated into services
such as ChatGPT to provide responses to user queries. To mitigate potential
harm and prevent misuse, there have been concerted efforts to align the LLMs
with human values and legal compliance by incorporating various techniques,
such as Reinforcement Learning from Human Feedback (RLHF), into the training of
the LLMs. However, recent research has exposed that even aligned LLMs are
susceptible to adversarial manipulations known as Jailbreak Attacks. To address
this challenge, this paper proposes a method called Token Highlighter to
inspect and mitigate the potential jailbreak threats in the user query. Token
Highlighter introduced a concept called Affirmation Loss to measure the LLM's
willingness to answer the user query. It then uses the gradient of Affirmation
Loss for each token in the user query to locate the jailbreak-critical tokens.
Further, Token Highlighter exploits our proposed Soft Removal technique to
mitigate the jailbreak effects of critical tokens via shrinking their token
embeddings. Experimental results on two aligned LLMs (LLaMA-2 and Vicuna-V1.5)
demonstrate that the proposed method can effectively defend against a variety
of Jailbreak Attacks while maintaining competent performance on benign
questions of the AlpacaEval benchmark. In addition, Token Highlighter is a
cost-effective and interpretable defense because it only needs to query the
protected LLM once to compute the Affirmation Loss and can highlight the
critical tokens upon refusal.


http://arxiv.org/abs/2412.18365
Hypergraph Attacks via Injecting Homogeneous Nodes into Elite Hyperedges. (22%)
Meixia He; Peican Zhu; Keke Tang; Yangming Guo
  Recent studies have shown that Hypergraph Neural Networks (HGNNs) are
vulnerable to adversarial attacks. Existing approaches focus on hypergraph
modification attacks guided by gradients, overlooking node spanning in the
hypergraph and the group identity of hyperedges, thereby resulting in limited
attack performance and detectable attacks. In this manuscript, we present a
novel framework, i.e., Hypergraph Attacks via Injecting Homogeneous Nodes into
Elite Hyperedges (IE-Attack), to tackle these challenges. Initially, utilizing
the node spanning in the hypergraph, we propose the elite hyperedges sampler to
identify hyperedges to be injected. Subsequently, a node generator utilizing
Kernel Density Estimation (KDE) is proposed to generate the homogeneous node
with the group identity of hyperedges. Finally, by injecting the homogeneous
node into elite hyperedges, IE-Attack improves the attack performance and
enhances the imperceptibility of attacks. Extensive experiments are conducted
on five authentic datasets to validate the effectiveness of IE-Attack and the
corresponding superiority to state-of-the-art methods.


http://arxiv.org/abs/2412.18409
Re-assessing ImageNet: How aligned is its single-label assumption with its multi-label nature? (4%)
Esla Timothy Anzaku; Seyed Amir Mousavi; Messem Arnout Van; Neve Wesley De
  ImageNet, an influential dataset in computer vision, is traditionally
evaluated using single-label classification, which assumes that an image can be
adequately described by a single concept or label. However, this approach may
not fully capture the complex semantics within the images available in
ImageNet, potentially hindering the development of models that effectively
learn these intricacies. This study critically examines the prevalent
single-label benchmarking approach and advocates for a shift to multi-label
benchmarking for ImageNet. This shift would enable a more comprehensive
assessment of the capabilities of deep neural network (DNN) models. We analyze
the effectiveness of pre-trained state-of-the-art DNNs on ImageNet and one of
its variants, ImageNetV2. Studies in the literature have reported unexpected
accuracy drops of 11% to 14% on ImageNetV2. Our findings show that these
reported declines are largely attributable to a characteristic of the dataset
that has not received sufficient attention -- the proportion of images with
multiple labels. Taking this characteristic into account, the results of our
experiments provide evidence that there is no substantial degradation in
effectiveness on ImageNetV2. Furthermore, we acknowledge that ImageNet
pre-trained models exhibit some capability at capturing the multi-label nature
of the dataset even though they were trained under the single-label assumption.
Consequently, we propose a new evaluation approach to augment existing
approaches that assess this capability. Our findings highlight the importance
of considering the multi-label nature of the ImageNet dataset during
benchmarking. Failing to do so could lead to incorrect conclusions regarding
the effectiveness of DNNs and divert research efforts from addressing other
substantial challenges related to the reliability and robustness of these
models.


http://arxiv.org/abs/2412.17544
Retention Score: Quantifying Jailbreak Risks for Vision Language Models. (99%)
Zaitang Li; Pin-Yu Chen; Tsung-Yi Ho
  The emergence of Vision-Language Models (VLMs) is a significant advancement
in integrating computer vision with Large Language Models (LLMs) to enhance
multi-modal machine learning capabilities. However, this progress has also made
VLMs vulnerable to sophisticated adversarial attacks, raising concerns about
their reliability. The objective of this paper is to assess the resilience of
VLMs against jailbreak attacks that can compromise model safety compliance and
result in harmful outputs. To evaluate a VLM's ability to maintain its
robustness against adversarial input perturbations, we propose a novel metric
called the \textbf{Retention Score}. Retention Score is a multi-modal
evaluation metric that includes Retention-I and Retention-T scores for
quantifying jailbreak risks in visual and textual components of VLMs. Our
process involves generating synthetic image-text pairs using a conditional
diffusion model. These pairs are then predicted for toxicity score by a VLM
alongside a toxicity judgment classifier. By calculating the margin in toxicity
scores, we can quantify the robustness of the VLM in an attack-agnostic manner.
Our work has four main contributions. First, we prove that Retention Score can
serve as a certified robustness metric. Second, we demonstrate that most VLMs
with visual components are less robust against jailbreak attacks than the
corresponding plain VLMs. Additionally, we evaluate black-box VLM APIs and find
that the security settings in Google Gemini significantly affect the score and
robustness. Moreover, the robustness of GPT4V is similar to the medium settings
of Gemini. Finally, our approach offers a time-efficient alternative to
existing adversarial attack methods and provides consistent model robustness
rankings when evaluated on VLMs including MiniGPT-4, InstructBLIP, and LLaVA.


http://arxiv.org/abs/2412.17614
Emerging Security Challenges of Large Language Models. (73%)
Herve Debar; Sven Dietrich; Pavel Laskov; Emil C. Lupu; Eirini Ntoutsi
  Large language models (LLMs) have achieved record adoption in a short period
of time across many different sectors including high importance areas such as
education [4] and healthcare [23]. LLMs are open-ended models trained on
diverse data without being tailored for specific downstream tasks, enabling
broad applicability across various domains. They are commonly used for text
generation, but also widely used to assist with code generation [3], and even
analysis of security information, as Microsoft Security Copilot demonstrates
[18]. Traditional Machine Learning (ML) models are vulnerable to adversarial
attacks [9]. So the concerns on the potential security implications of such
wide scale adoption of LLMs have led to the creation of this working group on
the security of LLMs. During the Dagstuhl seminar on "Network Attack Detection
and Defense - AI-Powered Threats and Responses", the working group discussions
focused on the vulnerability of LLMs to adversarial attacks, rather than their
potential use in generating malware or enabling cyberattacks. Although we note
the potential threat represented by the latter, the role of the LLMs in such
uses is mostly as an accelerator for development, similar to what it is in
benign use. To make the analysis more specific, the working group employed
ChatGPT as a concrete example of an LLM and addressed the following points,
which also form the structure of this report: 1. How do LLMs differ in
vulnerabilities from traditional ML models? 2. What are the attack objectives
in LLMs? 3. How complex it is to assess the risks posed by the vulnerabilities
of LLMs? 4. What is the supply chain in LLMs, how data flow in and out of
systems and what are the security implications? We conclude with an overview of
open challenges and outlook.


http://arxiv.org/abs/2412.17531
Double Landmines: Invisible Textual Backdoor Attacks based on Dual-Trigger. (68%)
Yang Hou; Qiuling Yue; Lujia Chai; Guozhao Liao; Wenbao Han; Wei Ou
  At present, all textual backdoor attack methods are based on single triggers:
for example, inserting specific content into the text to activate the backdoor;
or changing the abstract text features. The former is easier to be identified
by existing defense strategies due to its obvious characteristics; the latter,
although improved in invisibility, has certain shortcomings in terms of attack
performance, construction of poisoned datasets, and selection of the final
poisoning rate. On this basis, this paper innovatively proposes a Dual-Trigger
backdoor attack based on syntax and mood, and optimizes the construction of the
poisoned dataset and the selection strategy of the final poisoning rate. A
large number of experimental results show that this method significantly
outperforms the previous methods based on abstract features in attack
performance, and achieves comparable attack performance (almost 100% attack
success rate) with the insertion-based method. In addition, the two trigger
mechanisms included in this method can be activated independently in the
application phase of the model, which not only improves the flexibility of the
trigger style, but also enhances its robustness against defense strategies.
These results profoundly reveal that textual backdoor attacks are extremely
harmful and provide a new perspective for security protection in this field.


http://arxiv.org/abs/2412.17888
Stability Bounds for the Unfolded Forward-Backward Algorithm. (38%)
Emilie Chouzenoux; Valle Cecile Della; Jean-Christophe Pesquet
  We consider a neural network architecture designed to solve inverse problems
where the degradation operator is linear and known. This architecture is
constructed by unrolling a forward-backward algorithm derived from the
minimization of an objective function that combines a data-fidelity term, a
Tikhonov-type regularization term, and a potentially nonsmooth convex penalty.
The robustness of this inversion method to input perturbations is analyzed
theoretically. Ensuring robustness complies with the principles of inverse
problem theory, as it ensures both the continuity of the inversion method and
the resilience to small noise - a critical property given the known
vulnerability of deep neural networks to adversarial perturbations. A key
novelty of our work lies in examining the robustness of the proposed network to
perturbations in its bias, which represents the observed data in the inverse
problem. Additionally, we provide numerical illustrations of the analytical
Lipschitz bounds derived in our analysis.


http://arxiv.org/abs/2412.18123
AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models. (2%)
Yiming Wang; Jiahao Chen; Qingming Li; Xing Yang; Shouling Ji
  As text-to-image (T2I) models continue to advance and gain widespread
adoption, their associated safety issues are becoming increasingly prominent.
Malicious users often exploit these models to generate Not-Safe-for-Work (NSFW)
images using harmful or adversarial prompts, highlighting the critical need for
robust safeguards to ensure the integrity and compliance of model outputs.
Current internal safeguards frequently degrade image quality, while external
detection methods often suffer from low accuracy and inefficiency.
  In this paper, we introduce AEIOU, a defense framework that is Adaptable,
Efficient, Interpretable, Optimizable, and Unified against NSFW prompts in T2I
models. AEIOU extracts NSFW features from the hidden states of the model's text
encoder, utilizing the separable nature of these features to detect NSFW
prompts. The detection process is efficient, requiring minimal inference time.
AEIOU also offers real-time interpretation of results and supports optimization
through data augmentation techniques. The framework is versatile, accommodating
various T2I architectures. Our extensive experiments show that AEIOU
significantly outperforms both commercial and open-source moderation tools,
achieving over 95% accuracy across all datasets and improving efficiency by at
least tenfold. It effectively counters adaptive attacks and excels in few-shot
and multi-label scenarios.


http://arxiv.org/abs/2412.17740
Sensitivity Curve Maximization: Attacking Robust Aggregators in Distributed Learning. (1%)
Christian A. Schroth; Stefan Vlaski; Abdelhak M. Zoubir
  In distributed learning agents aim at collaboratively solving a global
learning problem. It becomes more and more likely that individual agents are
malicious or faulty with an increasing size of the network. This leads to a
degeneration or complete breakdown of the learning process. Classical
aggregation schemes are prone to breakdown at small contamination rates,
therefore robust aggregation schemes are sought for. While robust aggregation
schemes can generally tolerate larger contamination rates, many have been shown
to be susceptible to carefully crafted malicious attacks. In this work, we show
how the sensitivity curve (SC), a classical tool from robust statistics, can be
used to systematically derive optimal attack patterns against arbitrary robust
aggregators, in most cases rendering them ineffective. We show the
effectiveness of the proposed attack in multiple simulations.


http://arxiv.org/abs/2412.17522
DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak. (1%)
Hao Wang; Hao Li; Junda Zhu; Xinyuan Wang; Chengwei Pan; MinLie Huang; Lei Sha
  Large Language Models (LLMs) are susceptible to generating harmful content
when prompted with carefully crafted inputs, a vulnerability known as LLM
jailbreaking. As LLMs become more powerful, studying jailbreak methods is
critical to enhancing security and aligning models with human values.
Traditionally, jailbreak techniques have relied on suffix addition or prompt
templates, but these methods suffer from limited attack diversity. This paper
introduces DiffusionAttacker, an end-to-end generative approach for jailbreak
rewriting inspired by diffusion models. Our method employs a
sequence-to-sequence (seq2seq) text diffusion model as a generator,
conditioning on the original prompt and guiding the denoising process with a
novel attack loss. Unlike previous approaches that use autoregressive LLMs to
generate jailbreak prompts, which limit the modification of already generated
tokens and restrict the rewriting space, DiffusionAttacker utilizes a seq2seq
diffusion model, allowing more flexible token modifications. This approach
preserves the semantic content of the original prompt while producing harmful
content. Additionally, we leverage the Gumbel-Softmax technique to make the
sampling process from the diffusion model's output distribution differentiable,
eliminating the need for iterative token search. Extensive experiments on
Advbench and Harmbench demonstrate that DiffusionAttacker outperforms previous
methods across various evaluation metrics, including attack success rate (ASR),
fluency, and diversity.


http://arxiv.org/abs/2412.17038
ErasableMask: A Robust and Erasable Privacy Protection Scheme against Black-box Face Recognition Models. (99%)
Sipeng Shen; Yunming Zhang; Dengpan Ye; Xiuwen Shi; Long Tang; Haoran Duan; Ziyi Liu
  While face recognition (FR) models have brought remarkable convenience in
face verification and identification, they also pose substantial privacy risks
to the public. Existing facial privacy protection schemes usually adopt
adversarial examples to disrupt face verification of FR models. However, these
schemes often suffer from weak transferability against black-box FR models and
permanently damage the identifiable information that cannot fulfill the
requirements of authorized operations such as forensics and authentication. To
address these limitations, we propose ErasableMask, a robust and erasable
privacy protection scheme against black-box FR models. Specifically, via
rethinking the inherent relationship between surrogate FR models, ErasableMask
introduces a novel meta-auxiliary attack, which boosts black-box
transferability by learning more general features in a stable and balancing
optimization strategy. It also offers a perturbation erasion mechanism that
supports the erasion of semantic perturbations in protected face without
degrading image quality. To further improve performance, ErasableMask employs a
curriculum learning strategy to mitigate optimization conflicts between
adversarial attack and perturbation erasion. Extensive experiments on the
CelebA-HQ and FFHQ datasets demonstrate that ErasableMask achieves the
state-of-the-art performance in transferability, achieving over 72% confidence
on average in commercial FR systems. Moreover, ErasableMask also exhibits
outstanding perturbation erasion performance, achieving over 90% erasion
success rate.


http://arxiv.org/abs/2412.16955
NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors. (99%)
Ziqi Zhou; Bowen Li; Yufei Song; Zhifei Yu; Shengshan Hu; Wei Wan; Leo Yu Zhang; Dezhong Yao; Hai Jin
  With the advancement of deep learning, object detectors (ODs) with various
architectures have achieved significant success in complex scenarios like
autonomous driving. Previous adversarial attacks against ODs have been focused
on designing customized attacks targeting their specific structures (e.g., NMS
and RPN), yielding some results but simultaneously constraining their
scalability. Moreover, most efforts against ODs stem from image-level attacks
originally designed for classification tasks, resulting in redundant
computations and disturbances in object-irrelevant areas (e.g., background).
Consequently, how to design a model-agnostic efficient attack to
comprehensively evaluate the vulnerabilities of ODs remains challenging and
unresolved. In this paper, we propose NumbOD, a brand-new spatial-frequency
fusion attack against various ODs, aimed at disrupting object detection within
images. We directly leverage the features output by the OD without relying on
its internal structures to craft adversarial examples. Specifically, we first
design a dual-track attack target selection strategy to select high-quality
bounding boxes from OD outputs for targeting. Subsequently, we employ
directional perturbations to shift and compress predicted boxes and change
classification results to deceive ODs. Additionally, we focus on manipulating
the high-frequency components of images to confuse ODs' attention on critical
objects, thereby enhancing the attack efficiency. Our extensive experiments on
nine ODs and two datasets show that NumbOD achieves powerful attack performance
and high stealthiness.


http://arxiv.org/abs/2412.16958
Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature. (99%)
Yichen Wang; Yuxuan Chou; Ziqi Zhou; Hangtao Zhang; Wei Wan; Shengshan Hu; Minghui Li
  As deep neural networks (DNNs) are widely applied in the physical world, many
researches are focusing on physical-world adversarial examples (PAEs), which
introduce perturbations to inputs and cause the model's incorrect outputs.
However, existing PAEs face two challenges: unsatisfactory attack performance
(i.e., poor transferability and insufficient robustness to environment
conditions), and difficulty in balancing attack effectiveness with
stealthiness, where better attack effectiveness often makes PAEs more
perceptible.
  In this paper, we explore a novel perturbation-based method to overcome the
challenges. For the first challenge, we introduce a strategy Deceptive RF
injection based on robust features (RFs) that are predictive, robust to
perturbations, and consistent across different models. Specifically, it
improves the transferability and robustness of PAEs by covering RFs of other
classes onto the predictive features in clean images. For the second challenge,
we introduce another strategy Adversarial Semantic Pattern Minimization, which
removes most perturbations and retains only essential adversarial patterns in
AEsBased on the two strategies, we design our method Robust Feature Coverage
Attack (RFCoA), comprising Robust Feature Disentanglement and Adversarial
Feature Fusion. In the first stage, we extract target class RFs in feature
space. In the second stage, we use attention-based feature fusion to overlay
these RFs onto predictive features of clean images and remove unnecessary
perturbations. Experiments show our method's superior transferability,
robustness, and stealthiness compared to existing state-of-the-art methods.
Additionally, our method's effectiveness can extend to Large Vision-Language
Models (LVLMs), indicating its potential applicability to more complex tasks.


http://arxiv.org/abs/2412.16893
Preventing Non-intrusive Load Monitoring Privacy Invasion: A Precise Adversarial Attack Scheme for Networked Smart Meters. (98%)
Jialing He; Jiacheng Wang; Ning Wang; Shangwei Guo; Liehuang Zhu; Dusit Niyato; Tao Xiang
  Smart grid, through networked smart meters employing the non-intrusive load
monitoring (NILM) technique, can considerably discern the usage patterns of
residential appliances. However, this technique also incurs privacy leakage. To
address this issue, we propose an innovative scheme based on adversarial attack
in this paper. The scheme effectively prevents NILM models from violating
appliance-level privacy, while also ensuring accurate billing calculation for
users. To achieve this objective, we overcome two primary challenges. First, as
NILM models fall under the category of time-series regression models, direct
application of traditional adversarial attacks designed for classification
tasks is not feasible. To tackle this issue, we formulate a novel adversarial
attack problem tailored specifically for NILM and providing a theoretical
foundation for utilizing the Jacobian of the NILM model to generate
imperceptible perturbations. Leveraging the Jacobian, our scheme can produce
perturbations, which effectively misleads the signal prediction of NILM models
to safeguard users' appliance-level privacy. The second challenge pertains to
fundamental utility requirements, where existing adversarial attack schemes
struggle to achieve accurate billing calculation for users. To handle this
problem, we introduce an additional constraint, mandating that the sum of added
perturbations within a billing period must be precisely zero. Experimental
validation on real-world power datasets REDD and UK-DALE demonstrates the
efficacy of our proposed solutions, which can significantly amplify the
discrepancy between the output of the targeted NILM model and the actual power
signal of appliances, and enable accurate billing at the same time.
Additionally, our solutions exhibit transferability, making the generated
perturbation signal from one target model applicable to other diverse NILM
models.


http://arxiv.org/abs/2412.16905
A Backdoor Attack Scheme with Invisible Triggers Based on Model Architecture Modification. (45%)
Yuan Ma; Xu Ma; Jiankang Wei; Jinmeng Tang; Xiaoyu Zhang; Yilun Lyu; Kehao Chen; Jingtong Huang
  Machine learning systems are vulnerable to backdoor attacks, where attackers
manipulate model behavior through data tampering or architectural
modifications. Traditional backdoor attacks involve injecting malicious samples
with specific triggers into the training data, causing the model to produce
targeted incorrect outputs in the presence of the corresponding triggers. More
sophisticated attacks modify the model's architecture directly, embedding
backdoors that are harder to detect as they evade traditional data-based
detection methods. However, the drawback of the architectural modification
based backdoor attacks is that the trigger must be visible in order to activate
the backdoor. To further strengthen the invisibility of the backdoor attacks, a
novel backdoor attack method is presented in the paper. To be more specific,
this method embeds the backdoor within the model's architecture and has the
capability to generate inconspicuous and stealthy triggers. The attack is
implemented by modifying pre-trained models, which are then redistributed,
thereby posing a potential threat to unsuspecting users. Comprehensive
experiments conducted on standard computer vision benchmarks validate the
effectiveness of this attack and highlight the stealthiness of its triggers,
which remain undetectable through both manual visual inspection and advanced
detection tools.


http://arxiv.org/abs/2412.17213
Attack by Yourself: Effective and Unnoticeable Multi-Category Graph Backdoor Attacks with Subgraph Triggers Pool. (22%)
Jiangtong Li; Dungy Liu; Dawei Cheng; Changchun Jiang
  \textbf{G}raph \textbf{N}eural \textbf{N}etworks~(GNNs) have achieved
significant success in various real-world applications, including social
networks, finance systems, and traffic management. Recent researches highlight
their vulnerability to backdoor attacks in node classification, where GNNs
trained on a poisoned graph misclassify a test node only when specific triggers
are attached. These studies typically focus on single attack categories and use
adaptive trigger generators to create node-specific triggers. However, adaptive
trigger generators typically have a simple structure, limited parameters, and
lack category-aware graph knowledge, which makes them struggle to handle
backdoor attacks across multiple categories as the number of target categories
increases. We address this gap by proposing a novel approach for
\textbf{E}ffective and \textbf{U}nnoticeable
\textbf{M}ulti-\textbf{C}ategory~(EUMC) graph backdoor attacks, leveraging
subgraph from the attacked graph as category-aware triggers to precisely
control the target category. To ensure the effectiveness of our method, we
construct a \textbf{M}ulti-\textbf{C}ategory \textbf{S}ubgraph
\textbf{T}riggers \textbf{P}ool~(MC-STP) using the subgraphs of the attacked
graph as triggers. We then exploit the attachment probability shifts of each
subgraph trigger as category-aware priors for target category determination.
Moreover, we develop a ``select then attach'' strategy that connects suitable
category-aware trigger to attacked nodes for unnoticeability. Extensive
experiments across different real-world datasets confirm the efficacy of our
method in conducting multi-category graph backdoor attacks on various GNN
models and defense strategies.


http://arxiv.org/abs/2412.17034
Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models. (22%)
Lang Gao; Xiangliang Zhang; Preslav Nakov; Xiuying Chen
  Jailbreaking in Large Language Models (LLMs) is a major security concern as
it can deceive LLMs to generate harmful text. Yet, there is still insufficient
understanding of how jailbreaking works, which makes it hard to develop
effective defense strategies. We aim to shed more light into this issue: we
conduct a detailed large-scale analysis of seven different jailbreak methods
and find that these disagreements stem from insufficient observation samples.
In particular, we introduce \textit{safety boundary}, and we find that
jailbreaks shift harmful activations outside that safety boundary, where LLMs
are less sensitive to harmful information. We also find that the low and the
middle layers are critical in such shifts, while deeper layers have less
impact. Leveraging on these insights, we propose a novel defense called
\textbf{Activation Boundary Defense} (ABD), which adaptively constrains the
activations within the safety boundary. We further use Bayesian optimization to
selectively apply the defense method to the low and the middle layers. Our
experiments on several benchmarks show that ABD achieves an average DSR of over
98\% against various forms of jailbreak attacks, with less than 2\% impact on
the model's general capabilities.


http://arxiv.org/abs/2412.17011
Robustness of Large Language Models Against Adversarial Attacks. (13%)
Yiyi Tao; Yixian Shen; Hang Zhang; Yanxin Shen; Lun Wang; Chuanqi Shi; Shaoshuai Du
  The increasing deployment of Large Language Models (LLMs) in various
applications necessitates a rigorous evaluation of their robustness against
adversarial attacks. In this paper, we present a comprehensive study on the
robustness of GPT LLM family. We employ two distinct evaluation methods to
assess their resilience. The first method introduce character-level text attack
in input prompts, testing the models on three sentiment classification
datasets: StanfordNLP/IMDB, Yelp Reviews, and SST-2. The second method involves
using jailbreak prompts to challenge the safety mechanisms of the LLMs. Our
experiments reveal significant variations in the robustness of these models,
demonstrating their varying degrees of vulnerability to both character-level
and semantic-level adversarial attacks. These findings underscore the necessity
for improved adversarial training and enhanced safety mechanisms to bolster the
robustness of LLMs.


http://arxiv.org/abs/2412.16651
PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentation. (99%)
Yufei Song; Ziqi Zhou; Minghui Li; Xianlong Wang; Menghao Deng; Wei Wan; Shengshan Hu; Leo Yu Zhang
  With the rapid advancement of deep learning, the model robustness has become
a significant research hotspot, \ie, adversarial attacks on deep neural
networks. Existing works primarily focus on image classification tasks, aiming
to alter the model's predicted labels. Due to the output complexity and deeper
network architectures, research on adversarial examples for segmentation models
is still limited, particularly for universal adversarial perturbations. In this
paper, we propose a novel universal adversarial attack method designed for
segmentation models, which includes dual feature separation and low-frequency
scattering modules. The two modules guide the training of adversarial examples
in the pixel and frequency space, respectively. Experiments demonstrate that
our method achieves high attack success rates surpassing the state-of-the-art
methods, and exhibits strong transferability across different models.


http://arxiv.org/abs/2412.16662
Adversarial Attack Against Images Classification based on Generative Adversarial Networks. (98%)
Yahe Yang
  Adversarial attacks on image classification systems have always been an
important problem in the field of machine learning, and generative adversarial
networks (GANs), as popular models in the field of image generation, have been
widely used in various novel scenarios due to their powerful generative
capabilities. However, with the popularity of generative adversarial networks,
the misuse of fake image technology has raised a series of security problems,
such as malicious tampering with other people's photos and videos, and invasion
of personal privacy. Inspired by the generative adversarial networks, this work
proposes a novel adversarial attack method, aiming to gain insight into the
weaknesses of the image classification system and improve its anti-attack
ability. Specifically, the generative adversarial networks are used to generate
adversarial samples with small perturbations but enough to affect the
decision-making of the classifier, and the adversarial samples are generated
through the adversarial learning of the training generator and the classifier.
From extensive experiment analysis, we evaluate the effectiveness of the method
on a classical image classification dataset, and the results show that our
model successfully deceives a variety of advanced classifiers while maintaining
the naturalness of adversarial samples.


http://arxiv.org/abs/2412.16720
OpenAI o1 System Card. (81%)
OpenAI; :; Aaron Jaech; Adam Kalai; Adam Lerer; Adam Richardson; Ahmed El-Kishky; Aiden Low; Alec Helyar; Aleksander Madry; Alex Beutel; Alex Carney; Alex Iftimie; Alex Karpenko; Alex Tachard Passos; Alexander Neitz; Alexander Prokofiev; Alexander Wei; Allison Tam; Ally Bennett; Ananya Kumar; Andre Saraiva; Andrea Vallone; Andrew Duberstein; Andrew Kondrich; Andrey Mishchenko; Andy Applebaum; Angela Jiang; Ashvin Nair; Barret Zoph; Behrooz Ghorbani; Ben Rossen; Benjamin Sokolowsky; Boaz Barak; Bob McGrew; Borys Minaiev; Botao Hao; Bowen Baker; Brandon Houghton; Brandon McKinzie; Brydon Eastman; Camillo Lugaresi; Cary Bassin; Cary Hudson; Chak Ming Li; Bourcy Charles de; Chelsea Voss; Chen Shen; Chong Zhang; Chris Koch; Chris Orsinger; Christopher Hesse; Claudia Fischer; Clive Chan; Dan Roberts; Daniel Kappler; Daniel Levy; Daniel Selsam; David Dohan; David Farhi; David Mely; David Robinson; Dimitris Tsipras; Doug Li; Dragos Oprica; Eben Freeman; Eddie Zhang; Edmund Wong; Elizabeth Proehl; Enoch Cheung; Eric Mitchell; Eric Wallace; Erik Ritter; Evan Mays; Fan Wang; Felipe Petroski Such; Filippo Raso; Florencia Leoni; Foivos Tsimpourlas; Francis Song; Lohmann Fred von; Freddie Sulit; Geoff Salmon; Giambattista Parascandolo; Gildas Chabot; Grace Zhao; Greg Brockman; Guillaume Leclerc; Hadi Salman; Haiming Bao; Hao Sheng; Hart Andrin; Hessam Bagherinezhad; Hongyu Ren; Hunter Lightman; Hyung Won Chung; Ian Kivlichan; Ian O'Connell; Ian Osband; Ignasi Clavera Gilaberte; Ilge Akkaya; Ilya Kostrikov; Ilya Sutskever; Irina Kofman; Jakub Pachocki; James Lennon; Jason Wei; Jean Harb; Jerry Twore; Jiacheng Feng; Jiahui Yu; Jiayi Weng; Jie Tang; Jieqi Yu; Joaquin Quiñonero Candela; Joe Palermo; Joel Parish; Johannes Heidecke; John Hallman; John Rizzo; Jonathan Gordon; Jonathan Uesato; Jonathan Uesato; Jonathan Ward; Joost Huizinga; Julie Wang; Kai Chen; Kai Xiao; Karan Singhal; Karina Nguyen; Karl Cobbe; Katy Shi; Kayla Wood; Kendra Rimbach; Keren Gu-Lemberg; Keren GuLemberg; Kevin Liu; Kevin Lu; Kevin Stone; Kevin Yu; Lama Ahmad; Lauren Yang; Leo Liu; Leon Maksin; Leyton Ho; Liam Fedus; Lilian Weng; Linden Li; Lindsay McCallum; Lindsey Held; Lorenz Kuhn; Lukas Kondraciuk; Lukasz Kaiser; Luke Metz; Madelaine Boyd; Maja Trebacz; Manas Joglekar; Mark Chen; Marko Tintor; Mason Meyer; Matt Jones; Matt Kaufer; Max Schwarzer; Meghan Shah; Mehmet Yatbaz; Melody Guan; Mengyuan Xu; Mengyuan Yan; Mia Glaese; Mianna Chen; Mianna Chen; Michael Lampe; Michael Malek; Michele Wang; Michelle Fradin; Mike McClay; Mikhail Pavlov; Miles Wang; Mingxuan Wang; Mira Murati; Mo Bavarian; Mostafa Rohaninejad; Nat McAleese; Neil Chowdhury; Neil Chowdhury; Nick Ryder; Nikolas Tezak; Noam Brown; Ofir Nachum; Oleg Boiko; Oleg Murk; Olivia Watkins; Patrick Chao; Paul Ashbourne; Pavel Izmailov; Peter Zhokhov; Rachel Dias; Rahul Arora; Randall Lin; Rapha Gontijo Lopes; Raz Gaon; Reah Miyara; Reimar Leike; Renny Hwang; Rhythm Garg; Robin Brown; Roshan James; Rui Shu; Ryan Cheu; Ryan Greene; Saachi Jain; Sam Altman; Sam Toizer; Sam Toyer; Samuel Miserendino; Sandhini Agarwal; Santiago Hernandez; Sasha Baker; Scott McKinney; Scottie Yan; Shengjia Zhao; Shengli Hu; Shibani Santurkar; Shraman Ray Chaudhuri; Shuyuan Zhang; Siyuan Fu; Spencer Papay; Steph Lin; Suchir Balaji; Suvansh Sanjeev; Szymon Sidor; Tal Broda; Aidan Clark; Tao Wang; Taylor Gordon; Ted Sanders; Tejal Patwardhan; Thibault Sottiaux; Thomas Degry; Thomas Dimson; Tianhao Zheng; Timur Garipov; Tom Stasi; Trapit Bansal; Trevor Creech; Troy Peterson; Tyna Eloundou; Valerie Qi; Vineet Kosaraju; Vinnie Monaco; Vitchyr Pong; Vlad Fomenko; Weiyi Zheng; Wenda Zhou; Wes McCabe; Wojciech Zaremba; Yann Dubois; Yinghai Lu; Yining Chen; Young Cha; Yu Bai; Yuchen He; Yuchen Zhang; Yunyun Wang; Zheng Shao; Zhuohan Li
  The o1 model series is trained with large-scale reinforcement learning to
reason using chain of thought. These advanced reasoning capabilities provide
new avenues for improving the safety and robustness of our models. In
particular, our models can reason about our safety policies in context when
responding to potentially unsafe prompts, through deliberative alignment. This
leads to state-of-the-art performance on certain benchmarks for risks such as
generating illicit advice, choosing stereotyped responses, and succumbing to
known jailbreaks. Training models to incorporate a chain of thought before
answering has the potential to unlock substantial benefits, while also
increasing potential risks that stem from heightened intelligence. Our results
underscore the need for building robust alignment methods, extensively
stress-testing their efficacy, and maintaining meticulous risk management
protocols. This report outlines the safety work carried out for the OpenAI o1
and OpenAI o1-mini models, including safety evaluations, external red teaming,
and Preparedness Framework evaluations.


http://arxiv.org/abs/2412.16512
TrojFlow: Flow Models are Natural Targets for Trojan Attacks. (78%)
Zhengyang Qi; Xiaohua Xu
  Flow-based generative models (FMs) have rapidly advanced as a method for
mapping noise to data, its efficient training and sampling process makes it
widely applicable in various fields. FMs can be viewed as a variant of
diffusion models (DMs). At the same time, previous studies have shown that DMs
are vulnerable to Trojan/Backdoor attacks, a type of output manipulation attack
triggered by a maliciously embedded pattern at model input. We found that
Trojan attacks on generative models are essentially equivalent to image
transfer tasks from the backdoor distribution to the target distribution, the
unique ability of FMs to fit any two arbitrary distributions significantly
simplifies the training and sampling setups for attacking FMs, making them
inherently natural targets for backdoor attacks. In this paper, we propose
TrojFlow, exploring the vulnerabilities of FMs through Trojan attacks. In
particular, we consider various attack settings and their combinations and
thoroughly explore whether existing defense methods for DMs can effectively
defend against our proposed attack scenarios. We evaluate TrojFlow on CIFAR-10
and CelebA datasets, our experiments show that our method can compromise FMs
with high utility and specificity, and can easily break through existing
defense mechanisms.


http://arxiv.org/abs/2412.16708
Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks. (76%)
Jinyan Su; Jin Peng Zhou; Zhengxin Zhang; Preslav Nakov; Claire Cardie
  Retrieval-Augmented Generation (RAG) systems have emerged as a promising
solution to mitigate LLM hallucinations and enhance their performance in
knowledge-intensive domains. However, these systems are vulnerable to
adversarial poisoning attacks, where malicious passages injected into retrieval
databases can mislead the model into generating factually incorrect outputs. In
this paper, we investigate both the retrieval and the generation components of
RAG systems to understand how to enhance their robustness against such attacks.
From the retrieval perspective, we analyze why and how the adversarial contexts
are retrieved and assess how the quality of the retrieved passages impacts
downstream generation. From a generation perspective, we evaluate whether LLMs'
advanced critical thinking and internal knowledge capabilities can be leveraged
to mitigate the impact of adversarial contexts, i.e., using skeptical prompting
as a self-defense mechanism. Our experiments and findings provide actionable
insights into designing safer and more resilient retrieval-augmented
frameworks, paving the way for their reliable deployment in real-world
applications.


http://arxiv.org/abs/2412.16633
POEX: Policy Executable Embodied AI Jailbreak Attacks. (62%)
Xuancun Lu; Zhengxian Huang; Xinfeng Li; Xiaoyu ji; Wenyuan Xu
  The integration of large language models (LLMs) into the planning module of
Embodied Artificial Intelligence (Embodied AI) systems has greatly enhanced
their ability to translate complex user instructions into executable policies.
In this paper, we demystified how traditional LLM jailbreak attacks behave in
the Embodied AI context. We conducted a comprehensive safety analysis of the
LLM-based planning module of embodied AI systems against jailbreak attacks.
Using the carefully crafted Harmful-RLbench, we accessed 20 open-source and
proprietary LLMs under traditional jailbreak attacks, and highlighted two key
challenges when adopting the prior jailbreak techniques to embodied AI
contexts: (1) The harmful text output by LLMs does not necessarily induce
harmful policies in Embodied AI context, and (2) even we can generate harmful
policies, we have to guarantee they are executable in practice. To overcome
those challenges, we propose Policy Executable (POEX) jailbreak attacks, where
harmful instructions and optimized suffixes are injected into LLM-based
planning modules, leading embodied AI to perform harmful actions in both
simulated and physical environments. Our approach involves constraining
adversarial suffixes to evade detection and fine-tuning a policy evaluater to
improve the executability of harmful policies. We conducted extensive
experiments on both a robotic arm embodied AI platform and simulators, to
validate the attack and policy success rates on 136 harmful instructions from
Harmful-RLbench. Our findings expose serious safety vulnerabilities in
LLM-based planning modules, including the ability of POEX to be transferred
across models. Finally, we propose mitigation strategies, such as
safety-constrained prompts, pre- and post-planning checks, to address these
vulnerabilities and ensure the safe deployment of embodied AI in real-world
settings.


http://arxiv.org/abs/2412.16780
Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification. (8%)
Changchang Sun; Ren Wang; Yihua Zhang; Jinghan Jia; Jiancheng Liu; Gaowen Liu; Sijia Liu; Yan Yan
  Machine unlearning (MU), which seeks to erase the influence of specific
unwanted data from already-trained models, is becoming increasingly vital in
model editing, particularly to comply with evolving data regulations like the
``right to be forgotten''. Conventional approaches are predominantly
model-based, typically requiring retraining or fine-tuning the model's weights
to meet unlearning requirements. In this work, we approach the MU problem from
a novel input perturbation-based perspective, where the model weights remain
intact throughout the unlearning process. We demonstrate the existence of a
proactive input-based unlearning strategy, referred to forget vector, which can
be generated as an input-agnostic data perturbation and remains as effective as
model-based approximate unlearning approaches. We also explore forget vector
arithmetic, whereby multiple class-specific forget vectors are combined through
simple operations (e.g., linear combinations) to generate new forget vectors
for unseen unlearning tasks, such as forgetting arbitrary subsets across
classes. Extensive experiments validate the effectiveness and adaptability of
the forget vector, showcasing its competitive performance relative to
state-of-the-art model-based methods. Codes are available at
https://github.com/Changchangsun/Forget-Vector.


http://arxiv.org/abs/2412.16682
The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents. (4%)
Feiran Jia; Tong Wu; Xin Qin; Anna Squicciarini
  Large Language Model (LLM) agents are increasingly being deployed as
conversational assistants capable of performing complex real-world tasks
through tool integration. This enhanced ability to interact with external
systems and process various data sources, while powerful, introduces
significant security vulnerabilities. In particular, indirect prompt injection
attacks pose a critical threat, where malicious instructions embedded within
external data sources can manipulate agents to deviate from user intentions.
While existing defenses based on rule constraints, source spotlighting, and
authentication protocols show promise, they struggle to maintain robust
security while preserving task functionality. We propose a novel and orthogonal
perspective that reframes agent security from preventing harmful actions to
ensuring task alignment, requiring every agent action to serve user objectives.
Based on this insight, we develop Task Shield, a test-time defense mechanism
that systematically verifies whether each instruction and tool call contributes
to user-specified goals. Through experiments on the AgentDojo benchmark, we
demonstrate that Task Shield reduces attack success rates (2.07\%) while
maintaining high task utility (69.79\%) on GPT-4o.


http://arxiv.org/abs/2412.19834
RoboSignature: Robust Signature and Watermarking on Network Attacks. (1%)
Aryaman Shaan; Garvit Banga; Raghav Mantri
  Generative models have enabled easy creation and generation of images of all
kinds given a single prompt. However, this has also raised ethical concerns
about what is an actual piece of content created by humans or cameras compared
to model-generated content like images or videos. Watermarking data generated
by modern generative models is a popular method to provide information on the
source of the content. The goal is for all generated images to conceal an
invisible watermark, allowing for future detection or identification. The
Stable Signature finetunes the decoder of Latent Diffusion Models such that a
unique watermark is rooted in any image produced by the decoder. In this paper,
we present a novel adversarial fine-tuning attack that disrupts the model's
ability to embed the intended watermark, exposing a significant vulnerability
in existing watermarking methods. To address this, we further propose a
tamper-resistant fine-tuning algorithm inspired by methods developed for large
language models, tailored to the specific requirements of watermarking in LDMs.
Our findings emphasize the importance of anticipating and defending against
potential vulnerabilities in generative systems.


http://arxiv.org/abs/2412.16254
Adversarial Robustness through Dynamic Ensemble Learning. (99%)
Hetvi Waghela; Jaydip Sen; Sneha Rakshit
  Adversarial attacks pose a significant threat to the reliability of
pre-trained language models (PLMs) such as GPT, BERT, RoBERTa, and T5. This
paper presents Adversarial Robustness through Dynamic Ensemble Learning
(ARDEL), a novel scheme designed to enhance the robustness of PLMs against such
attacks. ARDEL leverages the diversity of multiple PLMs and dynamically adjusts
the ensemble configuration based on input characteristics and detected
adversarial patterns. Key components of ARDEL include a meta-model for dynamic
weighting, an adversarial pattern detection module, and adversarial training
with regularization techniques. Comprehensive evaluations using standardized
datasets and various adversarial attack scenarios demonstrate that ARDEL
significantly improves robustness compared to existing methods. By dynamically
reconfiguring the ensemble to prioritize the most robust models for each input,
ARDEL effectively reduces attack success rates and maintains higher accuracy
under adversarial conditions. This work contributes to the broader goal of
developing more secure and trustworthy AI systems for real-world NLP
applications, offering a practical and scalable solution to enhance adversarial
resilience in PLMs.


http://arxiv.org/abs/2412.16382
EMPRA: Embedding Perturbation Rank Attack against Neural Ranking Models. (98%)
Amin Bigdeli; Negar Arabzadeh; Ebrahim Bagheri; Charles L. A. Clarke
  Recent research has shown that neural information retrieval techniques may be
susceptible to adversarial attacks. Adversarial attacks seek to manipulate the
ranking of documents, with the intention of exposing users to targeted content.
In this paper, we introduce the Embedding Perturbation Rank Attack (EMPRA)
method, a novel approach designed to perform adversarial attacks on black-box
Neural Ranking Models (NRMs). EMPRA manipulates sentence-level embeddings,
guiding them towards pertinent context related to the query while preserving
semantic integrity. This process generates adversarial texts that seamlessly
integrate with the original content and remain imperceptible to humans. Our
extensive evaluation conducted on the widely-used MS MARCO V1 passage
collection demonstrate the effectiveness of EMPRA against a wide range of
state-of-the-art baselines in promoting a specific set of target documents
within a given ranked results. Specifically, EMPRA successfully achieves a
re-ranking of almost 96% of target documents originally ranked between 51-100
to rank within the top 10. Furthermore, EMPRA does not depend on surrogate
models for adversarial text generation, enhancing its robustness against
different NRMs in realistic settings.


http://arxiv.org/abs/2412.15924
Watertox: The Art of Simplicity in Universal Attacks A Cross-Model Framework for Robust Adversarial Generation. (98%)
Zhenghao Gao; Shengjie Xu; Meixi Chen; Fangyao Zhao
  Contemporary adversarial attack methods face significant limitations in
cross-model transferability and practical applicability. We present Watertox,
an elegant adversarial attack framework achieving remarkable effectiveness
through architectural diversity and precision-controlled perturbations. Our
two-stage Fast Gradient Sign Method combines uniform baseline perturbations
($\epsilon_1 = 0.1$) with targeted enhancements ($\epsilon_2 = 0.4$). The
framework leverages an ensemble of complementary architectures, from VGG to
ConvNeXt, synthesizing diverse perspectives through an innovative voting
mechanism. Against state-of-the-art architectures, Watertox reduces model
accuracy from 70.6% to 16.0%, with zero-shot attacks achieving up to 98.8%
accuracy reduction against unseen architectures. These results establish
Watertox as a significant advancement in adversarial methodologies, with
promising applications in visual security systems and CAPTCHA generation.


http://arxiv.org/abs/2412.16359
Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context. (82%)
Nilanjana Das; Edward Raff; Aman Chadha; Manas Gaur
  As the AI systems become deeply embedded in social media platforms, we've
uncovered a concerning security vulnerability that goes beyond traditional
adversarial attacks. It becomes important to assess the risks of LLMs before
the general public use them on social media platforms to avoid any adverse
impacts. Unlike obvious nonsensical text strings that safety systems can easily
catch, our work reveals that human-readable situation-driven adversarial
full-prompts that leverage situational context are effective but much harder to
detect. We found that skilled attackers can exploit the vulnerabilities in
open-source and proprietary LLMs to make a malicious user query safe for LLMs,
resulting in generating a harmful response. This raises an important question
about the vulnerabilities of LLMs. To measure the robustness against
human-readable attacks, which now present a potent threat, our research makes
three major contributions. First, we developed attacks that use movie scripts
as situational contextual frameworks, creating natural-looking full-prompts
that trick LLMs into generating harmful content. Second, we developed a method
to transform gibberish adversarial text into readable, innocuous content that
still exploits vulnerabilities when used within the full-prompts. Finally, we
enhanced the AdvPrompter framework with p-nucleus sampling to generate diverse
human-readable adversarial texts that significantly improve attack
effectiveness against models like GPT-3.5-Turbo-0125 and Gemma-7b. Our findings
show that these systems can be manipulated to operate beyond their intended
ethical boundaries when presented with seemingly normal prompts that contain
hidden adversarial elements. By identifying these vulnerabilities, we aim to
drive the development of more robust safety mechanisms that can withstand
sophisticated attacks in real-world applications.


http://arxiv.org/abs/2412.15623
JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs. (41%)
Hongyi Li; Jiawei Ye; Jie Wu; Tianjie Yan; Chu Wang; Zhixin Li
  Large Language Models (LLMs) aligned with human feedback have recently
garnered significant attention. However, it remains vulnerable to jailbreak
attacks, where adversaries manipulate prompts to induce harmful outputs.
Exploring jailbreak attacks enables us to investigate the vulnerabilities of
LLMs and further guides us in enhancing their security. Unfortunately, existing
techniques mainly rely on handcrafted templates or generated-based
optimization, posing challenges in scalability, efficiency and universality. To
address these issues, we present JailPO, a novel black-box jailbreak framework
to examine LLM alignment. For scalability and universality, JailPO meticulously
trains attack models to automatically generate covert jailbreak prompts.
Furthermore, we introduce a preference optimization-based attack method to
enhance the jailbreak effectiveness, thereby improving efficiency. To analyze
model vulnerabilities, we provide three flexible jailbreak patterns. Extensive
experiments demonstrate that JailPO not only automates the attack process while
maintaining effectiveness but also exhibits superior performance in efficiency,
universality, and robustness against defenses compared to baselines.
Additionally, our analysis of the three JailPO patterns reveals that attacks
based on complex templates exhibit higher attack strength, whereas covert
question transformations elicit riskier responses and are more likely to bypass
defense mechanisms.


http://arxiv.org/abs/2412.15704
PoisonCatcher: Revealing and Identifying LDP Poisoning Attacks in IIoT. (22%)
Lisha Shuai; Shaofeng Tan; Nan Zhang; Jiamin Zhang; Min Zhang; Xiaolong Yang
  Local Differential Privacy (LDP) is widely adopted in the Industrial Internet
of Things (IIoT) for its lightweight, decentralized, and scalable nature.
However, its perturbation-based privacy mechanism makes it difficult to
distinguish between uncontaminated and tainted data, encouraging adversaries to
launch poisoning attacks. While LDP provides some resilience against minor
poisoning, it lacks robustness in IIoT with dynamic networks and substantial
real-time data flows. Effective countermeasures for such attacks are still
underdeveloped. This work narrows the critical gap by revealing and identifying
LDP poisoning attacks in IIoT. We begin by deepening the understanding of such
attacks, revealing novel threats that arise from the interplay between LDP
indistinguishability and IIoT complexity. This exploration uncovers a novel
rule-poisoning attack, and presents a general attack formulation by unifying it
with input-poisoning and output-poisoning. Furthermore, two key attack impacts,
i.e., Statistical Query Result (SQR) accuracy degradation and inter-dataset
correlations disruption, along with two characteristics: attack patterns
unstable and poisoned data stealth are revealed. From this, we propose
PoisonCatcher, a four-stage solution that detects LDP poisoning attacks and
identifies specific contaminated data points. It utilizes temporal similarity,
attribute correlation, and time-series stability analysis to detect datasets
exhibiting SQR accuracy degradation, inter-dataset disruptions, and unstable
patterns. Enhanced feature engineering is used to extract subtle poisoning
signatures, enabling machine learning models to identify specific
contamination. Experimental evaluations show the effectiveness, achieving
state-of-the-art performance with average precision and recall rates of 86.17%
and 97.5%, respectively, across six representative attack scenarios.


http://arxiv.org/abs/2412.15614
Technical Report for ICML 2024 TiFA Workshop MLLM Attack Challenge: Suffix Injection and Projected Gradient Descent Can Easily Fool An MLLM. (13%)
Yangyang Guo; Ziwei Xu; Xilie Xu; YongKang Wong; Liqiang Nie; Mohan Kankanhalli
  This technical report introduces our top-ranked solution that employs two
approaches, \ie suffix injection and projected gradient descent (PGD) , to
address the TiFA workshop MLLM attack challenge. Specifically, we first append
the text from an incorrectly labeled option (pseudo-labeled) to the original
query as a suffix. Using this modified query, our second approach applies the
PGD method to add imperceptible perturbations to the image. Combining these two
techniques enables successful attacks on the LLaVA 1.5 model.


http://arxiv.org/abs/2412.16358
Texture- and Shape-based Adversarial Attacks for Vehicle Detection in Synthetic Overhead Imagery. (9%)
Mikael Yeghiazaryan; Sai Abhishek Siddhartha Namburu; Emily Kim; Stanislav Panev; Melo Celso de; Brent Lance; la Torre Fernando De; Jessica K. Hodgins
  Detecting vehicles in aerial images can be very challenging due to complex
backgrounds, small resolution, shadows, and occlusions. Despite the
effectiveness of SOTA detectors such as YOLO, they remain vulnerable to
adversarial attacks (AAs), compromising their reliability. Traditional AA
strategies often overlook the practical constraints of physical implementation,
focusing solely on attack performance. Our work addresses this issue by
proposing practical implementation constraints for AA in texture and/or shape.
These constraints include pixelation, masking, limiting the color palette of
the textures, and constraining the shape modifications. We evaluated the
proposed constraints through extensive experiments using three widely used
object detector architectures, and compared them to previous works. The results
demonstrate the effectiveness of our solutions and reveal a trade-off between
practicality and performance. Additionally, we introduce a labeled dataset of
overhead images featuring vehicles of various categories. We will make the
code/dataset public upon paper acceptance.


http://arxiv.org/abs/2412.16457
Robust random graph matching in dense graphs via vector approximate message passing. (1%)
Zhangsong Li
  In this paper, we focus on the matching recovery problem between a pair of
correlated Gaussian Wigner matrices with a latent vertex correspondence. We are
particularly interested in a robust version of this problem such that our
observation is a perturbed input $(A+E,B+F)$ where $(A,B)$ is a pair of
correlated Gaussian Wigner matrices and $E,F$ are adversarially chosen matrices
supported on an unknown $\epsilon n * \epsilon n$ principle minor of $A,B$,
respectively. We propose a vector-approximate message passing (vector-AMP)
algorithm that succeeds in polynomial time as long as the correlation $\rho$
between $(A,B)$ is a non-vanishing constant and $\epsilon = o\big(
\tfrac{1}{(\log n)^{20}} \big)$.
  The main methodological inputs for our result are the iterative random graph
matching algorithm proposed in \cite{DL22+, DL23+} and the spectral cleaning
procedure proposed in \cite{IS24+}. To the best of our knowledge, our algorithm
is the first efficient random graph matching type algorithm that is robust
under any adversarial perturbations of $n^{1-o(1)}$ size.


http://arxiv.org/abs/2412.15206
AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving. (5%)
Shuo Xing; Hongyuan Hua; Xiangbo Gao; Shenzhe Zhu; Renjie Li; Kexin Tian; Xiaopeng Li; Heng Huang; Tianbao Yang; Zhangyang Wang; Yang Zhou; Huaxiu Yao; Zhengzhong Tu
  Recent advancements in large vision language models (VLMs) tailored for
autonomous driving (AD) have shown strong scene understanding and reasoning
capabilities, making them undeniable candidates for end-to-end driving systems.
However, limited work exists on studying the trustworthiness of DriveVLMs -- a
critical factor that directly impacts public transportation safety. In this
paper, we introduce AutoTrust, a comprehensive trustworthiness benchmark for
large vision-language models in autonomous driving (DriveVLMs), considering
diverse perspectives -- including trustfulness, safety, robustness, privacy,
and fairness. We constructed the largest visual question-answering dataset for
investigating trustworthiness issues in driving scenarios, comprising over 10k
unique scenes and 18k queries. We evaluated six publicly available VLMs,
spanning from generalist to specialist, from open-source to commercial models.
Our exhaustive evaluations have unveiled previously undiscovered
vulnerabilities of DriveVLMs to trustworthiness threats. Specifically, we found
that the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperform
specialized models fine-tuned for driving in terms of overall trustworthiness.
DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosing
sensitive information. Additionally, both generalist and specialist VLMs remain
susceptible to adversarial attacks and struggle to ensure unbiased
decision-making across diverse environments and populations. Our findings call
for immediate and decisive action to address the trustworthiness of DriveVLMs
-- an issue of critical importance to public safety and the welfare of all
citizens relying on autonomous transportation systems. Our benchmark is
publicly available at \url{https://github.com/taco-group/AutoTrust}, and the
leaderboard is released at \url{https://taco-group.github.io/AutoTrust/}.


http://arxiv.org/abs/2412.14738
Boosting GNN Performance via Training Sample Selection Based on Adversarial Robustness Evaluation. (4%)
Yongyu Wang
  Graph Neural Networks (GNNs) have established themselves as one of the most
powerful neural network architectures, excelling in leveraging graph topology
and node features for various tasks. However, GNNs are inherently vulnerable to
noise in their inputs. Such noise can significantly degrade their performance.
To address this challenge, we propose a novel approach that employs adversarial
robustness evaluation techniques to identify nodes in the graph that are most
susceptible to noise. By selecting and constructing a training set composed of
these particularly noise-prone nodes, we then use them to train a Graph
Convolutional Network (GCN). Our experimental results demonstrate that this
strategy leads to substantial improvements in the GCN's performance.


http://arxiv.org/abs/2412.13880
A Review of the Duality of Adversarial Learning in Network Intrusion: Attacks and Countermeasures. (89%)
Shalini Saini; Anitha Chennamaneni; Babatunde Sawyerr
  Deep learning solutions are instrumental in cybersecurity, harnessing their
ability to analyze vast datasets, identify complex patterns, and detect
anomalies. However, malevolent actors can exploit these capabilities to
orchestrate sophisticated attacks, posing significant challenges to defenders
and traditional security measures. Adversarial attacks, particularly those
targeting vulnerabilities in deep learning models, present a nuanced and
substantial threat to cybersecurity. Our study delves into adversarial learning
threats such as Data Poisoning, Test Time Evasion, and Reverse Engineering,
specifically impacting Network Intrusion Detection Systems. Our research
explores the intricacies and countermeasures of attacks to deepen understanding
of network security challenges amidst adversarial threats. In our study, we
present insights into the dynamic realm of adversarial learning and its
implications for network intrusion. The intersection of adversarial attacks and
defenses within network traffic data, coupled with advances in machine learning
and deep learning techniques, represents a relatively underexplored domain. Our
research lays the groundwork for strengthening defense mechanisms to address
the potential breaches in network security and privacy posed by adversarial
attacks. Through our in-depth analysis, we identify domain-specific research
gaps, such as the scarcity of real-life attack data and the evaluation of
AI-based solutions for network traffic. Our focus on these challenges aims to
stimulate future research efforts toward the development of resilient network
defense strategies.


http://arxiv.org/abs/2412.13879
Crabs: Consuming Resrouce via Auto-generation for LLM-DoS Attack under Black-box Settings. (86%)
Yuanhe Zhang; Zhenhong Zhou; Wei Zhang; Xinyue Wang; Xiaojun Jia; Yang Liu; Sen Su
  Large Language Models (LLMs) have demonstrated remarkable performance across
diverse tasks. LLMs continue to be vulnerable to external threats, particularly
Denial-of-Service (DoS) attacks. Specifically, LLM-DoS attacks aim to exhaust
computational resources and block services. However, prior works tend to focus
on performing white-box attacks, overlooking black-box settings. In this work,
we propose an automated algorithm designed for black-box LLMs, called
Auto-Generation for LLM-DoS Attack (AutoDoS). AutoDoS introduces DoS Attack
Tree and optimizes the prompt node coverage to enhance effectiveness under
black-box conditions. Our method can bypass existing defense with enhanced
stealthiness via semantic improvement of prompt nodes. Furthermore, we reveal
that implanting Length Trojan in Basic DoS Prompt aids in achieving higher
attack efficacy. Experimental results show that AutoDoS amplifies service
response latency by over 250 $\times \uparrow$, leading to severe resource
consumption in terms of GPU utilization and memory usage. Our code is available
at \url{https://github.com/shuita2333/AutoDoS}.


http://arxiv.org/abs/2412.13705
Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation. (83%)
Minkyoung Kim; Yunha Kim; Hyeram Seo; Heejung Choi; Jiye Han; Gaeun Kee; Soyoung Ko; HyoJe Jung; Byeolhee Kim; Young-Hak Kim; Sanghyun Park; Tae Joon Jun
  Large language models (LLMs) have exhibited outstanding performance in
natural language processing tasks. However, these models remain susceptible to
adversarial attacks in which slight input perturbations can lead to harmful or
misleading outputs. A gradient-based defensive suffix generation algorithm is
designed to bolster the robustness of LLMs. By appending carefully optimized
defensive suffixes to input prompts, the algorithm mitigates adversarial
influences while preserving the models' utility. To enhance adversarial
understanding, a novel total loss function ($L_{\text{total}}$) combining
defensive loss ($L_{\text{def}}$) and adversarial loss ($L_{\text{adv}}$)
generates defensive suffixes more effectively. Experimental evaluations
conducted on open-source LLMs such as Gemma-7B, mistral-7B, Llama2-7B, and
Llama2-13B show that the proposed method reduces attack success rates (ASR) by
an average of 11\% compared to models without defensive suffixes. Additionally,
the perplexity score of Gemma-7B decreased from 6.57 to 3.93 when applying the
defensive suffix generated by openELM-270M. Furthermore, TruthfulQA evaluations
demonstrate consistent improvements with Truthfulness scores increasing by up
to 10\% across tested configurations. This approach significantly enhances the
security of LLMs in critical applications without requiring extensive
retraining.


http://arxiv.org/abs/2412.14113
Adversarial Hubness in Multi-Modal Retrieval. (82%)
Tingwei Zhang; Fnu Suya; Rishi Jha; Collin Zhang; Vitaly Shmatikov
  Hubness is a phenomenon in high-dimensional vector spaces where a single
point from the natural distribution is unusually close to many other points.
This is a well-known problem in information retrieval that causes some items to
accidentally (and incorrectly) appear relevant to many queries. In this paper,
we investigate how attackers can exploit hubness to turn any image or audio
input in a multi-modal retrieval system into an adversarial hub. Adversarial
hubs can be used to inject universal adversarial content (e.g., spam) that will
be retrieved in response to thousands of different queries, as well as for
targeted attacks on queries related to specific, attacker-chosen concepts. We
present a method for creating adversarial hubs and evaluate the resulting hubs
on benchmark multi-modal retrieval datasets and an image-to-image retrieval
system based on a tutorial from Pinecone, a popular vector database. For
example, in text-caption-to-image retrieval, a single adversarial hub is
retrieved as the top-1 most relevant image for more than 21,000 out of 25,000
test queries (by contrast, the most common natural hub is the top-1 response to
only 102 queries). We also investigate whether techniques for mitigating
natural hubness are an effective defense against adversarial hubs, and show
that they are not effective against hubs that target queries related to
specific concepts.


http://arxiv.org/abs/2412.13709
Physics-Based Adversarial Attack on Near-Infrared Human Detector for Nighttime Surveillance Camera Systems. (78%)
Muyao Niu; Zhuoxiao Li; Yifan Zhan; Huy H. Nguyen; Isao Echizen; Yinqiang Zheng
  Many surveillance cameras switch between daytime and nighttime modes based on
illuminance levels. During the day, the camera records ordinary RGB images
through an enabled IR-cut filter. At night, the filter is disabled to capture
near-infrared (NIR) light emitted from NIR LEDs typically mounted around the
lens. While RGB-based AI algorithm vulnerabilities have been widely reported,
the vulnerabilities of NIR-based AI have rarely been investigated. In this
paper, we identify fundamental vulnerabilities in NIR-based image understanding
caused by color and texture loss due to the intrinsic characteristics of
clothes' reflectance and cameras' spectral sensitivity in the NIR range. We
further show that the nearly co-located configuration of illuminants and
cameras in existing surveillance systems facilitates concealing and fully
passive attacks in the physical world. Specifically, we demonstrate how
retro-reflective and insulation plastic tapes can manipulate the intensity
distribution of NIR images. We showcase an attack on the YOLO-based human
detector using binary patterns designed in the digital space (via black-box
query and searching) and then physically realized using tapes pasted onto
clothes. Our attack highlights significant reliability concerns for nighttime
surveillance systems, which are intended to enhance security. Codes Available:
https://github.com/MyNiuuu/AdvNIR


http://arxiv.org/abs/2412.13762
Cultivating Archipelago of Forests: Evolving Robust Decision Trees through Island Coevolution. (56%)
Adam Żychowski; Andrew Perrault; Jacek Mańdziuk
  Decision trees are widely used in machine learning due to their simplicity
and interpretability, but they often lack robustness to adversarial attacks and
data perturbations. The paper proposes a novel island-based coevolutionary
algorithm (ICoEvoRDF) for constructing robust decision tree ensembles. The
algorithm operates on multiple islands, each containing populations of decision
trees and adversarial perturbations. The populations on each island evolve
independently, with periodic migration of top-performing decision trees between
islands. This approach fosters diversity and enhances the exploration of the
solution space, leading to more robust and accurate decision tree ensembles.
ICoEvoRDF utilizes a popular game theory concept of mixed Nash equilibrium for
ensemble weighting, which further leads to improvement in results. ICoEvoRDF is
evaluated on 20 benchmark datasets, demonstrating its superior performance
compared to state-of-the-art methods in optimizing both adversarial accuracy
and minimax regret. The flexibility of ICoEvoRDF allows for the integration of
decision trees from various existing methods, providing a unified framework for
combining diverse solutions. Our approach offers a promising direction for
developing robust and interpretable machine learning models


http://arxiv.org/abs/2412.13913
A Black-Box Evaluation Framework for Semantic Robustness in Bird's Eye View Detection. (8%)
Fu Wang; Yanghao Zhang; Xiangyu Yin; Guangliang Cheng; Zeyu Fu; Xiaowei Huang; Wenjie Ruan
  Camera-based Bird's Eye View (BEV) perception models receive increasing
attention for their crucial role in autonomous driving, a domain where concerns
about the robustness and reliability of deep learning have been raised. While
only a few works have investigated the effects of randomly generated semantic
perturbations, aka natural corruptions, on the multi-view BEV detection task,
we develop a black-box robustness evaluation framework that adversarially
optimises three common semantic perturbations: geometric transformation, colour
shifting, and motion blur, to deceive BEV models, serving as the first approach
in this emerging field. To address the challenge posed by optimising the
semantic perturbation, we design a smoothed, distance-based surrogate function
to replace the mAP metric and introduce SimpleDIRECT, a deterministic
optimisation algorithm that utilises observed slopes to guide the optimisation
process. By comparing with randomised perturbation and two optimisation
baselines, we demonstrate the effectiveness of the proposed framework.
Additionally, we provide a benchmark on the semantic robustness of ten recent
BEV models. The results reveal that PolarFormer, which emphasises geometric
information from multi-view images, exhibits the highest robustness, whereas
BEVDet is fully compromised, with its precision reduced to zero.


http://arxiv.org/abs/2412.13917
Speech Watermarking with Discrete Intermediate Representations. (4%)
Shengpeng Ji; Ziyue Jiang; Jialong Zuo; Minghui Fang; Yifu Chen; Tao Jin; Zhou Zhao
  Speech watermarking techniques can proactively mitigate the potential harmful
consequences of instant voice cloning techniques. These techniques involve the
insertion of signals into speech that are imperceptible to humans but can be
detected by algorithms. Previous approaches typically embed watermark messages
into continuous space. However, intuitively, embedding watermark information
into robust discrete latent space can significantly improve the robustness of
watermarking systems. In this paper, we propose DiscreteWM, a novel speech
watermarking framework that injects watermarks into the discrete intermediate
representations of speech. Specifically, we map speech into discrete latent
space with a vector-quantized autoencoder and inject watermarks by changing the
modular arithmetic relation of discrete IDs. To ensure the imperceptibility of
watermarks, we also propose a manipulator model to select the candidate tokens
for watermark embedding. Experimental results demonstrate that our framework
achieves state-of-the-art performance in robustness and imperceptibility,
simultaneously. Moreover, our flexible frame-wise approach can serve as an
efficient solution for both voice cloning detection and information hiding.
Additionally, DiscreteWM can encode 1 to 150 bits of watermark information
within a 1-second speech clip, indicating its encoding capacity. Audio samples
are available at https://DiscreteWM.github.io/discrete_wm.


http://arxiv.org/abs/2412.14080
On the Robustness of Distributed Machine Learning against Transfer Attacks. (2%)
Sébastien Andreina; Pascal Zimmer; Ghassan Karame
  Although distributed machine learning (distributed ML) is gaining
considerable attention in the community, prior works have independently looked
at instances of distributed ML in either the training or the inference phase.
No prior work has examined the combined robustness stemming from distributing
both the learning and the inference process. In this work, we explore, for the
first time, the robustness of distributed ML models that are fully
heterogeneous in training data, architecture, scheduler, optimizer, and other
model parameters. Supported by theory and extensive experimental validation
using CIFAR10 and FashionMNIST, we show that such properly distributed ML
instantiations achieve across-the-board improvements in accuracy-robustness
tradeoffs against state-of-the-art transfer-based attacks that could otherwise
not be realized by current ensemble or federated learning instantiations. For
instance, our experiments on CIFAR10 show that for the Common Weakness attack,
one of the most powerful state-of-the-art transfer-based attacks, our method
improves robust accuracy by up to 40%, with a minimal impact on clean task
accuracy.


http://arxiv.org/abs/2412.13753
Mesoscopic Insights: Orchestrating Multi-scale & Hybrid Architecture for Image Manipulation Localization. (1%)
Xuekang Zhu; Xiaochen Ma; Lei Su; Zhuohang Jiang; Bo Du; Xiwen Wang; Zeyu Lei; Wentao Feng; Chi-Man Pun; Jizhe Zhou
  The mesoscopic level serves as a bridge between the macroscopic and
microscopic worlds, addressing gaps overlooked by both. Image manipulation
localization (IML), a crucial technique to pursue truth from fake images, has
long relied on low-level (microscopic-level) traces. However, in practice, most
tampering aims to deceive the audience by altering image semantics. As a
result, manipulation commonly occurs at the object level (macroscopic level),
which is equally important as microscopic traces. Therefore, integrating these
two levels into the mesoscopic level presents a new perspective for IML
research. Inspired by this, our paper explores how to simultaneously construct
mesoscopic representations of micro and macro information for IML and
introduces the Mesorch architecture to orchestrate both. Specifically, this
architecture i) combines Transformers and CNNs in parallel, with Transformers
extracting macro information and CNNs capturing micro details, and ii) explores
across different scales, assessing micro and macro information seamlessly.
Additionally, based on the Mesorch architecture, the paper introduces two
baseline models aimed at solving IML tasks through mesoscopic representation.
Extensive experiments across four datasets have demonstrated that our models
surpass the current state-of-the-art in terms of performance, computational
complexity, and robustness.


http://arxiv.org/abs/2412.13525
Hybrid Data-Free Knowledge Distillation. (1%)
Jialiang Tang; Shuo Chen; Chen Gong
  Data-free knowledge distillation aims to learn a compact student network from
a pre-trained large teacher network without using the original training data of
the teacher network. Existing collection-based and generation-based methods
train student networks by collecting massive real examples and generating
synthetic examples, respectively. However, they inevitably become weak in
practical scenarios due to the difficulties in gathering or emulating
sufficient real-world data. To solve this problem, we propose a novel method
called \textbf{H}ybr\textbf{i}d \textbf{D}ata-\textbf{F}ree
\textbf{D}istillation (HiDFD), which leverages only a small amount of collected
data as well as generates sufficient examples for training student networks.
Our HiDFD comprises two primary modules, \textit{i.e.}, the teacher-guided
generation and student distillation. The teacher-guided generation module
guides a Generative Adversarial Network (GAN) by the teacher network to produce
high-quality synthetic examples from very few real-world collected examples.
Specifically, we design a feature integration mechanism to prevent the GAN from
overfitting and facilitate the reliable representation learning from the
teacher network. Meanwhile, we drive a category frequency smoothing technique
via the teacher network to balance the generative training of each category. In
the student distillation module, we explore a data inflation strategy to
properly utilize a blend of real and synthetic data to train the student
network via a classifier-sharing-based feature alignment technique. Intensive
experiments across multiple benchmarks demonstrate that our HiDFD can achieve
state-of-the-art performance using 120 times less collected data than existing
methods. Code is available at https://github.com/tangjialiang97/HiDFD.


http://arxiv.org/abs/2412.13866
SHAP scores fail pervasively even when Lipschitz succeeds. (1%)
Olivier Letoffe; Xuanxiang Huang; Joao Marques-Silva
  The ubiquitous use of Shapley values in eXplainable AI (XAI) has been
triggered by the tool SHAP, and as a result are commonly referred to as SHAP
scores. Recent work devised examples of machine learning (ML) classifiers for
which the computed SHAP scores are thoroughly unsatisfactory, by allowing human
decision-makers to be misled. Nevertheless, such examples could be perceived as
somewhat artificial, since the selected classes must be interpreted as numeric.
Furthermore, it was unclear how general were the issues identified with SHAP
scores. This paper answers these criticisms. First, the paper shows that for
Boolean classifiers there are arbitrarily many examples for which the SHAP
scores must be deemed unsatisfactory. Second, the paper shows that the issues
with SHAP scores are also observed in the case of regression models. In
addition, the paper studies the class of regression models that respect
Lipschitz continuity, a measure of a function's rate of change that finds
important recent uses in ML, including model robustness. Concretely, the paper
shows that the issues with SHAP scores occur even for regression models that
respect Lipschitz continuity. Finally, the paper shows that the same issues are
guaranteed to exist for arbitrarily differentiable regression models.


http://arxiv.org/abs/2412.13507
Novel AI Camera Camouflage: Face Cloaking Without Full Disguise. (1%)
David Noever; Forrest McKee
  This study demonstrates a novel approach to facial camouflage that combines
targeted cosmetic perturbations and alpha transparency layer manipulation to
evade modern facial recognition systems. Unlike previous methods -- such as CV
dazzle, adversarial patches, and theatrical disguises -- this work achieves
effective obfuscation through subtle modifications to key-point regions,
particularly the brow, nose bridge, and jawline. Empirical testing with Haar
cascade classifiers and commercial systems like BetaFaceAPI and Microsoft Bing
Visual Search reveals that vertical perturbations near dense facial key points
significantly disrupt detection without relying on overt disguises.
Additionally, leveraging alpha transparency attacks in PNG images creates a
dual-layer effect: faces remain visible to human observers but disappear in
machine-readable RGB layers, rendering them unidentifiable during reverse image
searches. The results highlight the potential for creating scalable,
low-visibility facial obfuscation strategies that balance effectiveness and
subtlety, opening pathways for defeating surveillance while maintaining
plausible anonymity.


http://arxiv.org/abs/2412.12626
Improving the Transferability of 3D Point Cloud Attack via Spectral-aware Admix and Optimization Designs. (99%)
Shiyu Hu; Daizong Liu; Wei Hu
  Deep learning models for point clouds have shown to be vulnerable to
adversarial attacks, which have received increasing attention in various
safety-critical applications such as autonomous driving, robotics, and
surveillance. Existing 3D attackers generally design various attack strategies
in the white-box setting, requiring the prior knowledge of 3D model details.
However, real-world 3D applications are in the black-box setting, where we can
only acquire the outputs of the target classifier. Although few recent works
try to explore the black-box attack, they still achieve limited attack success
rates (ASR). To alleviate this issue, this paper focuses on attacking the 3D
models in a transfer-based black-box setting, where we first carefully design
adversarial examples in a white-box surrogate model and then transfer them to
attack other black-box victim models. Specifically, we propose a novel
Spectral-aware Admix with Augmented Optimization method (SAAO) to improve the
adversarial transferability. In particular, since traditional Admix strategy
are deployed in the 2D domain that adds pixel-wise images for perturbing, we
can not directly follow it to merge point clouds in coordinate domain as it
will destroy the geometric shapes. Therefore, we design spectral-aware fusion
that performs Graph Fourier Transform (GFT) to get spectral features of the
point clouds and add them in the spectral domain. Afterward, we run a few steps
with spectral-aware weighted Admix to select better optimization paths as well
as to adjust corresponding learning weights. At last, we run more steps to
generate adversarial spectral feature along the optimization path and perform
Inverse-GFT on the adversarial spectral feature to obtain the adversarial
example in the data domain. Experiments show that our SAAO achieves better
transferability compared to existing 3D attack methods.


http://arxiv.org/abs/2412.13376
Targeted View-Invariant Adversarial Perturbations for 3D Object Recognition. (99%)
Christian Green; Mehmet Ergezer; Abdurrahman Zeybey
  Adversarial attacks pose significant challenges in 3D object recognition,
especially in scenarios involving multi-view analysis where objects can be
observed from varying angles. This paper introduces View-Invariant Adversarial
Perturbations (VIAP), a novel method for crafting robust adversarial examples
that remain effective across multiple viewpoints. Unlike traditional methods,
VIAP enables targeted attacks capable of manipulating recognition systems to
classify objects as specific, pre-determined labels, all while using a single
universal perturbation. Leveraging a dataset of 1,210 images across 121 diverse
rendered 3D objects, we demonstrate the effectiveness of VIAP in both targeted
and untargeted settings. Our untargeted perturbations successfully generate a
singular adversarial noise robust to 3D transformations, while targeted attacks
achieve exceptional results, with top-1 accuracies exceeding 95% across various
epsilon values. These findings highlight VIAPs potential for real-world
applications, such as testing the robustness of 3D recognition systems. The
proposed method sets a new benchmark for view-invariant adversarial robustness,
advancing the field of adversarial machine learning for 3D object recognition.


http://arxiv.org/abs/2412.16213
AdvIRL: Reinforcement Learning-Based Adversarial Attacks on 3D NeRF Models. (98%)
Tommy Nguyen; Mehmet Ergezer; Christian Green
  The increasing deployment of AI models in critical applications has exposed
them to significant risks from adversarial attacks. While adversarial
vulnerabilities in 2D vision models have been extensively studied, the threat
landscape for 3D generative models, such as Neural Radiance Fields (NeRF),
remains underexplored. This work introduces \textit{AdvIRL}, a novel framework
for crafting adversarial NeRF models using Instant Neural Graphics Primitives
(Instant-NGP) and Reinforcement Learning. Unlike prior methods, \textit{AdvIRL}
generates adversarial noise that remains robust under diverse 3D
transformations, including rotations and scaling, enabling effective black-box
attacks in real-world scenarios. Our approach is validated across a wide range
of scenes, from small objects (e.g., bananas) to large environments (e.g.,
lighthouses). Notably, targeted attacks achieved high-confidence
misclassifications, such as labeling a banana as a slug and a truck as a
cannon, demonstrating the practical risks posed by adversarial NeRFs. Beyond
attacking, \textit{AdvIRL}-generated adversarial models can serve as
adversarial training data to enhance the robustness of vision systems. The
implementation of \textit{AdvIRL} is publicly available at
\url{https://github.com/Tommy-Nguyen-cpu/AdvIRL/tree/MultiView-Clean}, ensuring
reproducibility and facilitating future research.


http://arxiv.org/abs/2412.15276
Exploring Query Efficient Data Generation towards Data-free Model Stealing in Hard Label Setting. (92%)
Gaozheng Pei; Shaojie lyu; Ke Ma; Pinci Yang; Qianqian Xu; Yingfei Sun
  Data-free model stealing involves replicating the functionality of a target
model into a substitute model without accessing the target model's structure,
parameters, or training data. The adversary can only access the target model's
predictions for generated samples. Once the substitute model closely
approximates the behavior of the target model, attackers can exploit its
white-box characteristics for subsequent malicious activities, such as
adversarial attacks. Existing methods within cooperative game frameworks often
produce samples with high confidence for the prediction of the substitute
model, which makes it difficult for the substitute model to replicate the
behavior of the target model. This paper presents a new data-free model
stealing approach called Query Efficient Data Generation (\textbf{QEDG}). We
introduce two distinct loss functions to ensure the generation of sufficient
samples that closely and uniformly align with the target model's decision
boundary across multiple classes. Building on the limitation of current
methods, which typically yield only one piece of supervised information per
query, we propose the query-free sample augmentation that enables the
acquisition of additional supervised information without increasing the number
of queries. Motivated by theoretical analysis, we adopt the consistency rate
metric, which more accurately evaluates the similarity between the substitute
and target models. We conducted extensive experiments to verify the
effectiveness of our proposed method, which achieved better performance with
fewer queries compared to the state-of-the-art methods on the real
\textbf{MLaaS} scenario and five datasets.


http://arxiv.org/abs/2412.15275
Fooling LLM graders into giving better grades through neural activity guided adversarial prompting. (75%)
Atsushi Yamamura; Surya Ganguli
  The deployment of artificial intelligence (AI) in critical decision-making
and evaluation processes raises concerns about inherent biases that malicious
actors could exploit to distort decision outcomes. We propose a systematic
method to reveal such biases in AI evaluation systems and apply it to automated
essay grading as an example. Our approach first identifies hidden neural
activity patterns that predict distorted decision outcomes and then optimizes
an adversarial input suffix to amplify such patterns. We demonstrate that this
combination can effectively fool large language model (LLM) graders into
assigning much higher grades than humans would. We further show that this
white-box attack transfers to black-box attacks on other models, including
commercial closed-source models like Gemini. They further reveal the existence
of a "magic word" that plays a pivotal role in the efficacy of the attack. We
trace the origin of this magic word bias to the structure of commonly-used chat
templates for supervised fine-tuning of LLMs and show that a minor change in
the template can drastically reduce the bias. This work not only uncovers
vulnerabilities in current LLMs but also proposes a systematic method to
identify and remove hidden biases, contributing to the goal of ensuring AI
safety and security.


http://arxiv.org/abs/2412.13134
Practicable Black-box Evasion Attacks on Link Prediction in Dynamic Graphs -- A Graph Sequential Embedding Method. (70%)
Jiate Li; Meng Pang; Binghui Wang
  Link prediction in dynamic graphs (LPDG) has been widely applied to
real-world applications such as website recommendation, traffic flow
prediction, organizational studies, etc. These models are usually kept local
and secure, with only the interactive interface restrictively available to the
public. Thus, the problem of the black-box evasion attack on the LPDG model,
where model interactions and data perturbations are restricted, seems to be
essential and meaningful in practice. In this paper, we propose the first
practicable black-box evasion attack method that achieves effective attacks
against the target LPDG model, within a limited amount of interactions and
perturbations. To perform effective attacks under limited perturbations, we
develop a graph sequential embedding model to find the desired state embedding
of the dynamic graph sequences, under a deep reinforcement learning framework.
To overcome the scarcity of interactions, we design a multi-environment
training pipeline and train our agent for multiple instances, by sharing an
aggregate interaction buffer. Finally, we evaluate our attack against three
advanced LPDG models on three real-world graph datasets of different scales and
compare its performance with related methods under the interaction and
perturbation constraints. Experimental results show that our attack is both
effective and practicable.


http://arxiv.org/abs/2412.12621
Jailbreaking? One Step Is Enough! (64%)
Weixiong Zheng; Peijian Zeng; Yiwei Li; Hongyan Wu; Nankai Lin; Junhao Chen; Aimin Yang; Yongmei Zhou
  Large language models (LLMs) excel in various tasks but remain vulnerable to
jailbreak attacks, where adversaries manipulate prompts to generate harmful
outputs. Examining jailbreak prompts helps uncover the shortcomings of LLMs.
However, current jailbreak methods and the target model's defenses are engaged
in an independent and adversarial process, resulting in the need for frequent
attack iterations and redesigning attacks for different models. To address
these gaps, we propose a Reverse Embedded Defense Attack (REDA) mechanism that
disguises the attack intention as the "defense". intention against harmful
content. Specifically, REDA starts from the target response, guiding the model
to embed harmful content within its defensive measures, thereby relegating
harmful content to a secondary role and making the model believe it is
performing a defensive task. The attacking model considers that it is guiding
the target model to deal with harmful content, while the target model thinks it
is performing a defensive task, creating an illusion of cooperation between the
two. Additionally, to enhance the model's confidence and guidance in
"defensive" intentions, we adopt in-context learning (ICL) with a small number
of attack examples and construct a corresponding dataset of attack examples.
Extensive evaluations demonstrate that the REDA method enables cross-model
attacks without the need to redesign attack strategies for different models,
enables successful jailbreak in one iteration, and outperforms existing methods
on both open-source and closed-source models.


http://arxiv.org/abs/2412.15267
Toxicity Detection towards Adaptability to Changing Perturbations. (11%)
Hankun Kang; Jianhao Chen; Yongqi Li; Xin Miao; Mayi Xu; Ming Zhong; Yuanyuan Zhu; Tieyun Qian
  Toxicity detection is crucial for maintaining the peace of the society. While
existing methods perform well on normal toxic contents or those generated by
specific perturbation methods, they are vulnerable to evolving perturbation
patterns. However, in real-world scenarios, malicious users tend to create new
perturbation patterns for fooling the detectors. For example, some users may
circumvent the detector of large language models (LLMs) by adding `I am a
scientist' at the beginning of the prompt. In this paper, we introduce a novel
problem, i.e., continual learning jailbreak perturbation patterns, into the
toxicity detection field. To tackle this problem, we first construct a new
dataset generated by 9 types of perturbation patterns, 7 of them are summarized
from prior work and 2 of them are developed by us. We then systematically
validate the vulnerability of current methods on this new perturbation
pattern-aware dataset via both the zero-shot and fine tuned cross-pattern
detection. Upon this, we present the domain incremental learning paradigm and
the corresponding benchmark to ensure the detector's robustness to dynamically
emerging types of perturbed toxic text. Our code and dataset are provided in
the appendix and will be publicly available at GitHub, by which we wish to
offer new research opportunities for the security-relevant communities.


http://arxiv.org/abs/2412.13017
A New Adversarial Perspective for LiDAR-based 3D Object Detection. (9%)
Shijun Zheng; Weiquan Liu; Yu Guo; Yu Zang; Siqi Shen; Cheng Wang
  Autonomous vehicles (AVs) rely on LiDAR sensors for environmental perception
and decision-making in driving scenarios. However, ensuring the safety and
reliability of AVs in complex environments remains a pressing challenge. To
address this issue, we introduce a real-world dataset (ROLiD) comprising
LiDAR-scanned point clouds of two random objects: water mist and smoke. In this
paper, we introduce a novel adversarial perspective by proposing an attack
framework that utilizes water mist and smoke to simulate environmental
interference. Specifically, we propose a point cloud sequence generation method
using a motion and content decomposition generative adversarial network named
PCS-GAN to simulate the distribution of random objects. Furthermore, leveraging
the simulated LiDAR scanning characteristics implemented with Range Image, we
examine the effects of introducing random object perturbations at various
positions on the target vehicle. Extensive experiments demonstrate that
adversarial perturbations based on random objects effectively deceive vehicle
detection and reduce the recognition rate of 3D object detection models.


http://arxiv.org/abs/2412.13099
Accuracy Limits as a Barrier to Biometric System Security. (2%)
Axel Durbet; Paul-Marie Grollemund; Pascal Lafourcade; Kevin Thiry-Atighehchi
  Biometric systems are widely used for identity verification and
identification, including authentication (i.e., one-to-one matching to verify a
claimed identity) and identification (i.e., one-to-many matching to find a
subject in a database). The matching process relies on measuring similarities
or dissimilarities between a fresh biometric template and enrolled templates.
The False Match Rate FMR is a key metric for assessing the accuracy and
reliability of such systems. This paper analyzes biometric systems based on
their FMR, with two main contributions. First, we explore untargeted attacks,
where an adversary aims to impersonate any user within a database. We determine
the number of trials required for an attacker to successfully impersonate a
user and derive the critical population size (i.e., the maximum number of users
in the database) required to maintain a given level of security. Furthermore,
we compute the critical FMR value needed to ensure resistance against
untargeted attacks as the database size increases. Second, we revisit the
biometric birthday problem to evaluate the approximate and exact probabilities
that two users in a database collide (i.e., can impersonate each other). Based
on this analysis, we derive both the approximate critical population size and
the critical FMR value needed to bound the likelihood of such collisions
occurring with a given probability. These thresholds offer insights for
designing systems that mitigate the risk of impersonation and collisions,
particularly in large-scale biometric databases. Our findings indicate that
current biometric systems fail to deliver sufficient accuracy to achieve an
adequate security level against untargeted attacks, even in small-scale
databases. Moreover, state-of-the-art systems face significant challenges in
addressing the biometric birthday problem, especially as database sizes grow.


http://arxiv.org/abs/2412.12996
Neural Control and Certificate Repair via Runtime Monitoring. (1%)
Emily Yu; Đorđe Žikelić; Thomas A. Henzinger
  Learning-based methods provide a promising approach to solving highly
non-linear control tasks that are often challenging for classical control
methods. To ensure the satisfaction of a safety property, learning-based
methods jointly learn a control policy together with a certificate function for
the property. Popular examples include barrier functions for safety and
Lyapunov functions for asymptotic stability. While there has been significant
progress on learning-based control with certificate functions in the white-box
setting, where the correctness of the certificate function can be formally
verified, there has been little work on ensuring their reliability in the
black-box setting where the system dynamics are unknown. In this work, we
consider the problems of certifying and repairing neural network control
policies and certificate functions in the black-box setting. We propose a novel
framework that utilizes runtime monitoring to detect system behaviors that
violate the property of interest under some initially trained neural network
policy and certificate. These violating behaviors are used to extract new
training data, that is used to re-train the neural network policy and the
certificate function and to ultimately repair them. We demonstrate the
effectiveness of our approach empirically by using it to repair and to boost
the safety rate of neural network policies learned by a state-of-the-art method
for learning-based control on two autonomous system control tasks.


http://arxiv.org/abs/2412.13229
Training Verification-Friendly Neural Networks via Neuron Behavior Consistency. (1%)
Zongxin Liu; Zhe Zhao; Fu Song; Jun Sun; Pengfei Yang; Xiaowei Huang; Lijun Zhang
  Formal verification provides critical security assurances for neural
networks, yet its practical application suffers from the long verification
time. This work introduces a novel method for training verification-friendly
neural networks, which are robust, easy to verify, and relatively accurate. Our
method integrates neuron behavior consistency into the training process, making
neuron activation states consistent across different inputs in a local
neighborhood, reducing the number of unstable neurons and tightening the bounds
of neurons thereby enhancing neural network verifiability. We evaluated our
method using the MNIST, Fashion-MNIST, and CIFAR-10 datasets across various
network architectures. The results of the experiment demonstrate that networks
trained using our method are verification-friendly across different radii and
different model architectures, whereas other tools fail to maintain
verifiability as the radius increases. We also show that our method can be
combined with existing methods to further improve the verifiability of
networks.


http://arxiv.org/abs/2412.13394
Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation. (1%)
Burak Ekim; Girmaw Abebe Tadesse; Caleb Robinson; Gilles Hacheme; Michael Schmitt; Rahul Dodhia; Juan M. Lavista Ferres
  Training robust deep learning models is critical in Earth Observation, where
globally deployed models often face distribution shifts that degrade
performance, especially in low-data regions. Out-of-distribution (OOD)
detection addresses this challenge by identifying inputs that differ from
in-distribution (ID) data. However, existing methods either assume access to
OOD data or compromise primary task performance, making them unsuitable for
real-world deployment. We propose TARDIS, a post-hoc OOD detection method for
scalable geospatial deployments. The core novelty lies in generating surrogate
labels by integrating information from ID data and unknown distributions,
enabling OOD detection at scale. Our method takes a pre-trained model, ID data,
and WILD samples, disentangling the latter into surrogate ID and surrogate OOD
labels based on internal activations, and fits a binary classifier as an OOD
detector. We validate TARDIS on EuroSAT and xBD datasets, across 17
experimental setups covering covariate and semantic shifts, showing that it
performs close to the theoretical upper bound in assigning surrogate ID and OOD
samples in 13 cases. To demonstrate scalability, we deploy TARDIS on the Fields
of the World dataset, offering actionable insights into pre-trained model
behavior for large-scale deployments. The code is publicly available at
https://github.com/microsoft/geospatial-ood-detection.


http://arxiv.org/abs/2412.12449
Adversarially robust generalization theory via Jacobian regularization for deep neural networks. (99%)
Dongya Wu; Xin Li
  Powerful deep neural networks are vulnerable to adversarial attacks. To
obtain adversarially robust models, researchers have separately developed
adversarial training and Jacobian regularization techniques. There are abundant
theoretical and empirical studies for adversarial training, but theoretical
foundations for Jacobian regularization are still lacking. In this study, we
show that Jacobian regularization is closely related to adversarial training in
that $\ell_{2}$ or $\ell_{1}$ Jacobian regularized loss serves as an
approximate upper bound on the adversarially robust loss under $\ell_{2}$ or
$\ell_{\infty}$ adversarial attack respectively. Further, we establish the
robust generalization gap for Jacobian regularized risk minimizer via bounding
the Rademacher complexity of both the standard loss function class and Jacobian
regularization function class. Our theoretical results indicate that the norms
of Jacobian are related to both standard and robust generalization. We also
perform experiments on MNIST data classification to demonstrate that Jacobian
regularized risk minimization indeed serves as a surrogate for adversarially
robust risk minimization, and that reducing the norms of Jacobian can improve
both standard and robust generalization. This study promotes both theoretical
and empirical understandings to adversarially robust generalization via
Jacobian regularization.


http://arxiv.org/abs/2412.12478
Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script. (99%)
Xi Cao; Yuan Sun; Jiajun Li; Quzong Gesang; Nuo Qun; Tashi Nyima
  DNN-based language models perform excellently on various tasks, but even SOTA
LLMs are susceptible to textual adversarial attacks. Adversarial texts play
crucial roles in multiple subfields of NLP. However, current research has the
following issues. (1) Most textual adversarial attack methods target
rich-resourced languages. How do we generate adversarial texts for less-studied
languages? (2) Most textual adversarial attack methods are prone to generating
invalid or ambiguous adversarial texts. How do we construct high-quality
adversarial robustness benchmarks? (3) New language models may be immune to
part of previously generated adversarial texts. How do we update adversarial
robustness benchmarks? To address the above issues, we introduce HITL-GAT, a
system based on a general approach to human-in-the-loop generation of
adversarial texts. HITL-GAT contains four stages in one pipeline: victim model
construction, adversarial example generation, high-quality benchmark
construction, and adversarial robustness evaluation. Additionally, we utilize
HITL-GAT to make a case study on Tibetan script which can be a reference for
the adversarial research of other less-studied languages.


http://arxiv.org/abs/2412.11735
Transferable Adversarial Face Attack with Text Controlled Attribute. (98%)
Wenyun Li; Zheng Zhang; Xiangyuan Lan; Dongmei Jiang
  Traditional adversarial attacks typically produce adversarial examples under
norm-constrained conditions, whereas unrestricted adversarial examples are
free-form with semantically meaningful perturbations. Current unrestricted
adversarial impersonation attacks exhibit limited control over adversarial face
attributes and often suffer from low transferability. In this paper, we propose
a novel Text Controlled Attribute Attack (TCA$^2$) to generate photorealistic
adversarial impersonation faces guided by natural language. Specifically, the
category-level personal softmax vector is employed to precisely guide the
impersonation attacks. Additionally, we propose both data and model
augmentation strategies to achieve transferable attacks on unknown target
models. Finally, a generative model, \textit{i.e}, Style-GAN, is utilized to
synthesize impersonated faces with desired attributes. Extensive experiments on
two high-resolution face recognition datasets validate that our TCA$^2$ method
can generate natural text-guided adversarial impersonation faces with high
transferability. We also evaluate our method on real-world face recognition
systems, \textit{i.e}, Face++ and Aliyun, further demonstrating the practical
potential of our approach.


http://arxiv.org/abs/2412.11608
Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation. (86%)
Svetlana Pavlitska; Enrico Eisen; J. Marius Zöllner
  Vulnerability to adversarial attacks is a well-known deficiency of deep
neural networks. Larger networks are generally more robust, and ensembling is
one method to increase adversarial robustness: each model's weaknesses are
compensated by the strengths of others. While an ensemble uses a deterministic
rule to combine model outputs, a mixture of experts (MoE) includes an
additional learnable gating component that predicts weights for the outputs of
the expert models, thus determining their contributions to the final
prediction. MoEs have been shown to outperform ensembles on specific tasks, yet
their susceptibility to adversarial attacks has not been studied yet. In this
work, we evaluate the adversarial vulnerability of MoEs for semantic
segmentation of urban and highway traffic scenes. We show that MoEs are, in
most cases, more robust to per-instance and universal white-box adversarial
attacks and can better withstand transfer attacks. Our code is available at
\url{https://github.com/KASTEL-MobilityLab/mixtures-of-experts/}.


http://arxiv.org/abs/2412.11487
WFCAT: Augmenting Website Fingerprinting with Channel-wise Attention on Timing Features. (33%)
Jiajun Gong; Wei Cai; Siyuan Liang; Zhong Guan; Tao Wang; Ee-Chien Chang
  Website Fingerprinting (WF) aims to deanonymize users on the Tor network by
analyzing encrypted network traffic. Recent deep-learning-based attacks show
high accuracy on undefended traces. However, they struggle against modern
defenses that use tactics like injecting dummy packets and delaying real
packets, which significantly degrade classification performance. Our analysis
reveals that current attacks inadequately leverage the timing information
inherent in traffic traces, which persists as a source of leakage even under
robust defenses. Addressing this shortfall, we introduce a novel feature
representation named the Inter-Arrival Time (IAT) histogram, which quantifies
the frequencies of packet inter-arrival times across predetermined time slots.
Complementing this feature, we propose a new CNN-based attack, WFCAT, enhanced
with two innovative architectural blocks designed to optimally extract and
utilize timing information. Our approach uses kernels of varying sizes to
capture multi-scale features, which are then integrated using a weighted sum
across all feature channels to enhance the model's efficacy in identifying
temporal patterns. Our experiments validate that WFCAT substantially
outperforms existing methods on defended traces in both closed- and open-world
scenarios. Notably, WFCAT achieves over 59% accuracy against Surakav, a
recently developed robust defense, marking an improvement of over 28% and 48%
against the state-of-the-art attacks RF and Tik-Tok, respectively, in the
closed-world scenario.


http://arxiv.org/abs/2412.11471
Red Pill and Blue Pill: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learning. (22%)
Siyuan Liang; Jiajun Gong; Tianmeng Fang; Aishan Liu; Tao Wang; Xianglong Liu; Xiaochun Cao; Dacheng Tao; Chang Ee-Chien
  Website fingerprint (WF) attacks, which covertly monitor user communications
to identify the web pages they visit, pose a serious threat to user privacy.
Existing WF defenses attempt to reduce the attacker's accuracy by disrupting
unique traffic patterns; however, they often suffer from the trade-off between
overhead and effectiveness, resulting in less usefulness in practice. To
overcome this limitation, we introduce Controllable Website Fingerprint Defense
(CWFD), a novel defense perspective based on backdoor learning. CWFD exploits
backdoor vulnerabilities in neural networks to directly control the attacker's
model by designing trigger patterns based on network traffic. Specifically,
CWFD injects only incoming packets on the server side into the target web
page's traffic, keeping overhead low while effectively poisoning the attacker's
model during training. During inference, the defender can influence the
attacker's model through a 'red pill, blue pill' choice: traces with the
trigger (red pill) lead to misclassification as the target web page, while
normal traces (blue pill) are classified correctly, achieving directed control
over the defense outcome. We use the Fast Levenshtein-like distance as the
optimization objective to compute trigger patterns that can be effectively
associated with our target page. Experiments show that CWFD significantly
reduces RF's accuracy from 99% to 6% with 74% data overhead. In comparison,
FRONT reduces accuracy to only 97% at similar overhead, while Palette achieves
32% accuracy with 48% more overhead. We further validate the practicality of
our method in a real Tor network environment.


http://arxiv.org/abs/2412.11840
Sonar-based Deep Learning in Underwater Robotics: Overview, Robustness and Challenges. (1%)
Martin Aubard; Ana Madureira; Luís Teixeira; José Pinto
  With the growing interest in underwater exploration and monitoring,
Autonomous Underwater Vehicles (AUVs) have become essential. The recent
interest in onboard Deep Learning (DL) has advanced real-time environmental
interaction capabilities relying on efficient and accurate vision-based DL
models. However, the predominant use of sonar in underwater environments,
characterized by limited training data and inherent noise, poses challenges to
model robustness. This autonomy improvement raises safety concerns for
deploying such models during underwater operations, potentially leading to
hazardous situations. This paper aims to provide the first comprehensive
overview of sonar-based DL under the scope of robustness. It studies
sonar-based DL perception task models, such as classification, object
detection, segmentation, and SLAM. Furthermore, the paper systematizes
sonar-based state-of-the-art datasets, simulators, and robustness methods such
as neural network verification, out-of-distribution, and adversarial attacks.
This paper highlights the lack of robustness in sonar-based DL research and
suggests future research pathways, notably establishing a baseline sonar-based
dataset and bridging the simulation-to-reality gap.


http://arxiv.org/abs/2412.12000
CP-Guard: Malicious Agent Detection and Defense in Collaborative Bird's Eye View Perception. (1%)
Senkang Hu; Yihang Tao; Guowen Xu; Yiqin Deng; Xianhao Chen; Yuguang Fang; Sam Kwong
  Collaborative Perception (CP) has shown a promising technique for autonomous
driving, where multiple connected and autonomous vehicles (CAVs) share their
perception information to enhance the overall perception performance and expand
the perception range. However, in CP, ego CAV needs to receive messages from
its collaborators, which makes it easy to be attacked by malicious agents. For
example, a malicious agent can send harmful information to the ego CAV to
mislead it. To address this critical issue, we propose a novel method,
CP-Guard, a tailored defense mechanism for CP that can be deployed by each
agent to accurately detect and eliminate malicious agents in its collaboration
network. Our key idea is to enable CP to reach a consensus rather than a
conflict against the ego CAV's perception results. Based on this idea, we first
develop a probability-agnostic sample consensus (PASAC) method to effectively
sample a subset of the collaborators and verify the consensus without prior
probabilities of malicious agents. Furthermore, we define a collaborative
consistency loss (CCLoss) to capture the discrepancy between the ego CAV and
its collaborators, which is used as a verification criterion for consensus.
Finally, we conduct extensive experiments in collaborative bird's eye view
(BEV) tasks and our results demonstrate the effectiveness of our CP-Guard. Code
is available at https://github.com/CP-Security/CP-Guard


http://arxiv.org/abs/2412.11934
Stepwise Reasoning Error Disruption Attack of LLMs. (1%)
Jingyu Peng; Maolin Wang; Xiangyu Zhao; Kai Zhang; Wanyu Wang; Pengyue Jia; Qidong Liu; Ruocheng Guo; Qi Liu
  Large language models (LLMs) have made remarkable strides in complex
reasoning tasks, but their safety and robustness in reasoning processes remain
underexplored. Existing attacks on LLM reasoning are constrained by specific
settings or lack of imperceptibility, limiting their feasibility and
generalizability. To address these challenges, we propose the Stepwise
rEasoning Error Disruption (SEED) attack, which subtly injects errors into
prior reasoning steps to mislead the model into producing incorrect subsequent
reasoning and final answers. Unlike previous methods, SEED is compatible with
zero-shot and few-shot settings, maintains the natural reasoning flow, and
ensures covert execution without modifying the instruction. Extensive
experiments on four datasets across four different models demonstrate SEED's
effectiveness, revealing the vulnerabilities of LLMs to disruptions in
reasoning processes. These findings underscore the need for greater attention
to the robustness of LLM reasoning to ensure safety in practical applications.
Our code is available at:
https://github.com/Applied-Machine-Learning-Lab/SEED-Attack.


http://arxiv.org/abs/2412.12217
Comprehensive Survey on Adversarial Examples in Cybersecurity: Impacts, Challenges, and Mitigation Strategies. (99%)
Li Li
  Deep learning (DL) has significantly transformed cybersecurity, enabling
advancements in malware detection, botnet identification, intrusion detection,
user authentication, and encrypted traffic analysis. However, the rise of
adversarial examples (AE) poses a critical challenge to the robustness and
reliability of DL-based systems. These subtle, crafted perturbations can
deceive models, leading to severe consequences like misclassification and
system vulnerabilities. This paper provides a comprehensive review of the
impact of AE attacks on key cybersecurity applications, highlighting both their
theoretical and practical implications. We systematically examine the methods
used to generate adversarial examples, their specific effects across various
domains, and the inherent trade-offs attackers face between efficacy and
resource efficiency. Additionally, we explore recent advancements in defense
mechanisms, including gradient masking, adversarial training, and detection
techniques, evaluating their potential to enhance model resilience. By
summarizing cutting-edge research, this study aims to bridge the gap between
adversarial research and practical security applications, offering insights to
fortify the adoption of DL solutions in cybersecurity.


http://arxiv.org/abs/2412.11172
Unpacking the Resilience of SNLI Contradiction Examples to Attacks. (99%)
Chetan Verma; Archit Agarwal
  Pre-trained models excel on NLI benchmarks like SNLI and MultiNLI, but their
true language understanding remains uncertain. Models trained only on
hypotheses and labels achieve high accuracy, indicating reliance on dataset
biases and spurious correlations. To explore this issue, we applied the
Universal Adversarial Attack to examine the model's vulnerabilities. Our
analysis revealed substantial drops in accuracy for the entailment and neutral
classes, whereas the contradiction class exhibited a smaller decline.
Fine-tuning the model on an augmented dataset with adversarial examples
restored its performance to near-baseline levels for both the standard and
challenge sets. Our findings highlight the value of adversarial triggers in
identifying spurious correlations and improving robustness while providing
insights into the resilience of the contradiction class to adversarial attacks.


http://arxiv.org/abs/2412.11119
Impact of Adversarial Attacks on Deep Learning Model Explainability. (99%)
Gazi Nazia Nur; Mohammad Ahnaf Sadat
  In this paper, we investigate the impact of adversarial attacks on the
explainability of deep learning models, which are commonly criticized for their
black-box nature despite their capacity for autonomous feature extraction. This
black-box nature can affect the perceived trustworthiness of these models. To
address this, explainability techniques such as GradCAM, SmoothGrad, and LIME
have been developed to clarify model decision-making processes. Our research
focuses on the robustness of these explanations when models are subjected to
adversarial attacks, specifically those involving subtle image perturbations
that are imperceptible to humans but can significantly mislead models. For
this, we utilize attack methods like the Fast Gradient Sign Method (FGSM) and
the Basic Iterative Method (BIM) and observe their effects on model accuracy
and explanations. The results reveal a substantial decline in model accuracy,
with accuracies dropping from 89.94% to 58.73% and 45.50% under FGSM and BIM
attacks, respectively. Despite these declines in accuracy, the explanation of
the models measured by metrics such as Intersection over Union (IoU) and Root
Mean Square Error (RMSE) shows negligible changes, suggesting that these
metrics may not be sensitive enough to detect the presence of adversarial
perturbations.


http://arxiv.org/abs/2412.11441
UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models. (99%)
Yuning Han; Bingyin Zhao; Rui Chu; Feng Luo; Biplab Sikdar; Yingjie Lao
  Recent studies show that diffusion models (DMs) are vulnerable to backdoor
attacks. Existing backdoor attacks impose unconcealed triggers (e.g., a gray
box and eyeglasses) that contain evident patterns, rendering remarkable attack
effects yet easy detection upon human inspection and defensive algorithms.
While it is possible to improve stealthiness by reducing the strength of the
backdoor, doing so can significantly compromise its generality and
effectiveness. In this paper, we propose UIBDiffusion, the universal
imperceptible backdoor attack for diffusion models, which allows us to achieve
superior attack and generation performance while evading state-of-the-art
defenses. We propose a novel trigger generation approach based on universal
adversarial perturbations (UAPs) and reveal that such perturbations, which are
initially devised for fooling pre-trained discriminative models, can be adapted
as potent imperceptible backdoor triggers for DMs. We evaluate UIBDiffusion on
multiple types of DMs with different kinds of samplers across various datasets
and targets. Experimental results demonstrate that UIBDiffusion brings three
advantages: 1) Universality, the imperceptible trigger is universal (i.e.,
image and model agnostic) where a single trigger is effective to any images and
all diffusion models with different samplers; 2) Utility, it achieves
comparable generation quality (e.g., FID) and even better attack success rate
(i.e., ASR) at low poison rates compared to the prior works; and 3)
Undetectability, UIBDiffusion is plausible to human perception and can bypass
Elijah and TERD, the SOTA defenses against backdoors for DMs. We will release
our backdoor triggers and code.


http://arxiv.org/abs/2412.11168
PGD-Imp: Rethinking and Unleashing Potential of Classic PGD with Dual Strategies for Imperceptible Adversarial Attacks. (98%)
Jin Li; Zitong Yu; Ziqiang He; Z. Jane Wang; Xiangui Kang
  Imperceptible adversarial attacks have recently attracted increasing research
interests. Existing methods typically incorporate external modules or loss
terms other than a simple $l_p$-norm into the attack process to achieve
imperceptibility, while we argue that such additional designs may not be
necessary. In this paper, we rethink the essence of imperceptible attacks and
propose two simple yet effective strategies to unleash the potential of PGD,
the common and classical attack, for imperceptibility from an optimization
perspective. Specifically, the Dynamic Step Size is introduced to find the
optimal solution with minimal attack cost towards the decision boundary of the
attacked model, and the Adaptive Early Stop strategy is adopted to reduce the
redundant strength of adversarial perturbations to the minimum level. The
proposed PGD-Imperceptible (PGD-Imp) attack achieves state-of-the-art results
in imperceptible adversarial attacks for both untargeted and targeted
scenarios. When performing untargeted attacks against ResNet-50, PGD-Imp
attains 100$\%$ (+0.3$\%$) ASR, 0.89 (-1.76) $l_2$ distance, and 52.93 (+9.2)
PSNR with 57s (-371s) running time, significantly outperforming existing
methods.


http://arxiv.org/abs/2412.11066
Learning Robust and Privacy-Preserving Representations via Information Theory. (92%)
Binghui Zhang; Sayedeh Leila Noorbakhsh; Yun Dong; Yuan Hong; Binghui Wang
  Machine learning models are vulnerable to both security attacks (e.g.,
adversarial examples) and privacy attacks (e.g., private attribute inference).
We take the first step to mitigate both the security and privacy attacks, and
maintain task utility as well. Particularly, we propose an
information-theoretic framework to achieve the goals through the lens of
representation learning, i.e., learning representations that are robust to both
adversarial examples and attribute inference adversaries. We also derive novel
theoretical results under our framework, e.g., the inherent trade-off between
adversarial robustness/utility and attribute privacy, and guaranteed attribute
privacy leakage against attribute inference adversaries.


http://arxiv.org/abs/2412.11384
A Comprehensive Review of Adversarial Attacks on Machine Learning. (75%)
Syed Quiser Ahmed; Bharathi Vokkaliga Ganesh; Sathyanarayana Sampath Kumar; Prakhar Mishra; Ravi Anand; Bhanuteja Akurathi
  This research provides a comprehensive overview of adversarial attacks on AI
and ML models, exploring various attack types, techniques, and their potential
harms. We also delve into the business implications, mitigation strategies, and
future research directions. To gain practical insights, we employ the
Adversarial Robustness Toolbox (ART) [1] library to simulate these attacks on
real-world use cases, such as self-driving cars. Our goal is to inform
practitioners and researchers about the challenges and opportunities in
defending AI systems against adversarial threats. By providing a comprehensive
comparison of different attack methods, we aim to contribute to the development
of more robust and secure AI systems.


http://arxiv.org/abs/2412.12212
Finding a Wolf in Sheep's Clothing: Combating Adversarial Text-To-Image Prompts with Text Summarization. (2%)
Portia Cooper; Harshita Narnoli; Mihai Surdeanu
  Text-to-image models are vulnerable to the stepwise "Divide-and-Conquer
Attack" (DACA) that utilize a large language model to obfuscate inappropriate
content in prompts by wrapping sensitive text in a benign narrative. To
mitigate stepwise DACA attacks, we propose a two-layer method involving text
summarization followed by binary classification. We assembled the Adversarial
Text-to-Image Prompt (ATTIP) dataset ($N=940$), which contained DACA-obfuscated
and non-obfuscated prompts. From the ATTIP dataset, we created two summarized
versions: one generated by a small encoder model and the other by a large
language model. Then, we used an encoder classifier and a GPT-4o classifier to
perform content moderation on the summarized and unsummarized prompts. When
compared with a classifier that operated over the unsummarized data, our method
improved F1 score performance by 31%. Further, the highest recorded F1 score
achieved (98%) was produced by the encoder classifier on a summarized ATTIP
variant. This study indicates that pre-classification text summarization can
inoculate content detection models against stepwise DACA obfuscations.


http://arxiv.org/abs/2412.11057
Set-Valued Sensitivity Analysis of Deep Neural Networks. (2%)
Xin Jeff Wang; Feiling Jeff wang; Xuegang Jeff Ban
  This paper proposes a sensitivity analysis framework based on set valued
mapping for deep neural networks (DNN) to understand and compute how the
solutions (model weights) of DNN respond to perturbations in the training data.
As a DNN may not exhibit a unique solution (minima) and the algorithm of
solving a DNN may lead to different solutions with minor perturbations to input
data, we focus on the sensitivity of the solution set of DNN, instead of
studying a single solution. In particular, we are interested in the expansion
and contraction of the set in response to data perturbations. If the change of
solution set can be bounded by the extent of the data perturbation, the model
is said to exhibit the Lipschitz like property. This "set-to-set" analysis
approach provides a deeper understanding of the robustness and reliability of
DNNs during training. Our framework incorporates both isolated and non-isolated
minima, and critically, does not require the assumption that the Hessian of
loss function is non-singular. By developing set-level metrics such as distance
between sets, convergence of sets, derivatives of set-valued mapping, and
stability across the solution set, we prove that the solution set of the Fully
Connected Neural Network holds Lipschitz-like properties. For general neural
networks (e.g., Resnet), we introduce a graphical-derivative-based method to
estimate the new solution set following data perturbation without retraining.


http://arxiv.org/abs/2412.11109
SpearBot: Leveraging Large Language Models in a Generative-Critique Framework for Spear-Phishing Email Generation. (1%)
Qinglin Qi; Yun Luo; Yijia Xu; Wenbo Guo; Yong Fang
  Large Language Models (LLMs) are increasingly capable, aiding in tasks such
as content generation, yet they also pose risks, particularly in generating
harmful spear-phishing emails. These emails, crafted to entice clicks on
malicious URLs, threaten personal information security. This paper proposes an
adversarial framework, SpearBot, which utilizes LLMs to generate spear-phishing
emails with various phishing strategies. Through specifically crafted jailbreak
prompts, SpearBot circumvents security policies and introduces other LLM
instances as critics. When a phishing email is identified by the critic,
SpearBot refines the generated email based on the critique feedback until it
can no longer be recognized as phishing, thereby enhancing its deceptive
quality. To evaluate the effectiveness of SpearBot, we implement various
machine-based defenders and assess how well the phishing emails generated could
deceive them. Results show these emails often evade detection to a large
extent, underscoring their deceptive quality. Additionally, human evaluations
of the emails' readability and deception are conducted through questionnaires,
confirming their convincing nature and the significant potential harm of the
generated phishing emails.


http://arxiv.org/abs/2412.11390
A3E: Aligned and Augmented Adversarial Ensemble for Accurate, Robust and Privacy-Preserving EEG Decoding. (1%)
Xiaoqing Chen; Tianwang Jia; Dongrui Wu
  An electroencephalogram (EEG) based brain-computer interface (BCI) enables
direct communication between the brain and external devices. However, EEG-based
BCIs face at least three major challenges in real-world applications: data
scarcity and individual differences, adversarial vulnerability, and data
privacy. While previous studies have addressed one or two of these issues,
simultaneous accommodation of all three challenges remains challenging and
unexplored. This paper fills this gap, by proposing an Aligned and Augmented
Adversarial Ensemble (A3E) algorithm and integrating it into three privacy
protection scenarios (centralized source-free transfer, federated source-free
transfer, and source data perturbation), achieving simultaneously accurate
decoding, adversarial robustness, and privacy protection of EEG-based BCIs.
Experiments on three public EEG datasets demonstrated that our proposed
approach outperformed over 10 classic and state-of-the-art approaches in both
accuracy and robustness in all three privacy-preserving scenarios, even
outperforming state-of-the-art transfer learning approaches that do not
consider privacy protection at all. This is the first time that three major
challenges in EEG-based BCIs can be addressed simultaneously, significantly
improving the practicalness of EEG decoding in real-world BCIs.


http://arxiv.org/abs/2412.10713
RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors. (98%)
Fengshuo Bai; Runze Liu; Yali Du; Ying Wen; Yaodong Yang
  Evaluating deep reinforcement learning (DRL) agents against targeted behavior
attacks is critical for assessing their robustness. These attacks aim to
manipulate the victim into specific behaviors that align with the attacker's
objectives, often bypassing traditional reward-based defenses. Prior methods
have primarily focused on reducing cumulative rewards; however, rewards are
typically too generic to capture complex safety requirements effectively. As a
result, focusing solely on reward reduction can lead to suboptimal attack
strategies, particularly in safety-critical scenarios where more precise
behavior manipulation is needed. To address these challenges, we propose RAT, a
method designed for universal, targeted behavior attacks. RAT trains an
intention policy that is explicitly aligned with human preferences, serving as
a precise behavioral target for the adversary. Concurrently, an adversary
manipulates the victim's policy to follow this target behavior. To enhance the
effectiveness of these attacks, RAT dynamically adjusts the state occupancy
measure within the replay buffer, allowing for more controlled and effective
behavior manipulation. Our empirical results on robotic simulation tasks
demonstrate that RAT outperforms existing adversarial attack algorithms in
inducing specific behaviors. Additionally, RAT shows promise in improving agent
robustness, leading to more resilient policies. We further validate RAT by
guiding Decision Transformer agents to adopt behaviors aligned with human
preferences in various MuJoCo tasks, demonstrating its effectiveness across
diverse tasks.


http://arxiv.org/abs/2412.10681
One Pixel is All I Need. (80%)
Deng Siqin; Zhou Xiaoyi
  Vision Transformers (ViTs) have achieved record-breaking performance in
various visual tasks. However, concerns about their robustness against backdoor
attacks have grown. Backdoor attacks involve associating a specific trigger
with a target label, causing the model to predict the attacker-specified label
when the trigger is present, while correctly identifying clean images.We found
that ViTs exhibit higher attack success rates for quasi-triggers(patterns
different from but similar to the original training triggers)compared to CNNs.
Moreover, some backdoor features in clean samples can suppress the original
trigger, making quasi-triggers more effective.To better understand and exploit
these vulnerabilities, we developed a tool called the Perturbation Sensitivity
Distribution Map (PSDM). PSDM computes and sums gradients over many inputs to
show how sensitive the model is to small changes in the input. In ViTs, PSDM
reveals a patch-like pattern where central pixels are more sensitive than
edges. We use PSDM to guide the creation of quasi-triggers.Based on these
findings, we designed "WorstVIT," a simple yet effective data poisoning
backdoor for ViT models. This attack requires an extremely low poisoning rate,
trains for just one epoch, and modifies a single pixel to successfully attack
all validation images.


http://arxiv.org/abs/2412.10807
Towards Action Hijacking of Large Language Model-based Agent. (26%)
Yuyang Zhang; Kangjie Chen; Jiaxin Gao; Ronghao Cui; Run Wang; Lina Wang; Tianwei Zhang
  Recently, applications powered by Large Language Models (LLMs) have made
significant strides in tackling complex tasks. By harnessing the advanced
reasoning capabilities and extensive knowledge embedded in LLMs, these
applications can generate detailed action plans that are subsequently executed
by external tools. Furthermore, the integration of retrieval-augmented
generation (RAG) enhances performance by incorporating up-to-date,
domain-specific knowledge into the planning and execution processes. This
approach has seen widespread adoption across various sectors, including
healthcare, finance, and software development. Meanwhile, there are also
growing concerns regarding the security of LLM-based applications. Researchers
have disclosed various attacks, represented by jailbreak and prompt injection,
to hijack the output actions of these applications. Existing attacks mainly
focus on crafting semantically harmful prompts, and their validity could
diminish when security filters are employed. In this paper, we introduce
AI$\mathbf{^2}$, a novel attack to manipulate the action plans of LLM-based
applications. Different from existing solutions, the innovation of
AI$\mathbf{^2}$ lies in leveraging the knowledge from the application's
database to facilitate the construction of malicious but semantically-harmless
prompts. To this end, it first collects action-aware knowledge from the victim
application. Based on such knowledge, the attacker can generate misleading
input, which can mislead the LLM to generate harmful action plans, while
bypassing possible detection mechanisms easily. Our evaluations on three
real-world applications demonstrate the effectiveness of AI$\mathbf{^2}$: it
achieves an average attack success rate of 84.30\% with the best of 99.70\%.
Besides, it gets an average bypass rate of 92.7\% against common safety filters
and 59.45\% against dedicated defense.


http://arxiv.org/abs/2412.10805
Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languages. (1%)
Poulami Ghosh; Raj Dabre; Pushpak Bhattacharyya
  Pre-trained language models (PLMs) are known to be susceptible to
perturbations to the input text, but existing works do not explicitly focus on
linguistically grounded attacks, which are subtle and more prevalent in nature.
In this paper, we study whether PLMs are agnostic to linguistically grounded
attacks or not. To this end, we offer the first study addressing this,
investigating different Indic languages and various downstream tasks. Our
findings reveal that although PLMs are susceptible to linguistic perturbations,
when compared to non-linguistic attacks, PLMs exhibit a slightly lower
susceptibility to linguistic attacks. This highlights that even constrained
attacks are effective. Moreover, we investigate the implications of these
outcomes across a range of languages, encompassing diverse language families
and different scripts.


http://arxiv.org/abs/2412.10831
Unbiased General Annotated Dataset Generation. (1%)
Dengyang Jiang; Haoyu Wang; Lei Zhang; Wei Wei; Guang Dai; Mengmeng Wang; Jingdong Wang; Yanning Zhang
  Pre-training backbone networks on a general annotated dataset (e.g.,
ImageNet) that comprises numerous manually collected images with category
annotations has proven to be indispensable for enhancing the generalization
capacity of downstream visual tasks. However, those manually collected images
often exhibit bias, which is non-transferable across either categories or
domains, thus causing the model's generalization capacity degeneration. To
mitigate this problem, we present an unbiased general annotated dataset
generation framework (ubGen). Instead of expensive manual collection, we aim at
directly generating unbiased images with category annotations. To achieve this
goal, we propose to leverage the advantage of a multimodal foundation model
(e.g., CLIP), in terms of aligning images in an unbiased semantic space defined
by language. Specifically, we develop a bi-level semantic alignment loss, which
not only forces all generated images to be consistent with the semantic
distribution of all categories belonging to the target dataset in an
adversarial learning manner, but also requires each generated image to match
the semantic description of its category name. In addition, we further cast an
existing image quality scoring model into a quality assurance loss to preserve
the quality of the generated image. By leveraging these two loss functions, we
can obtain an unbiased image generation model by simply fine-tuning a
pre-trained diffusion model using only all category names in the target dataset
as input. Experimental results confirm that, compared with the manually labeled
dataset or other synthetic datasets, the utilization of our generated unbiased
datasets leads to stable generalization capacity enhancement of different
backbone networks across various tasks, especially in tasks where the manually
labeled samples are scarce.


http://arxiv.org/abs/2412.09910
Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Images. (99%)
Yasamin Medghalchi; Moein Heidari; Clayton Allard; Leonid Sigal; Ilker Hacihaliloglu
  Deep neural networks (DNNs) offer significant promise for improving breast
cancer diagnosis in medical imaging. However, these models are highly
susceptible to adversarial attacks--small, imperceptible changes that can
mislead classifiers--raising critical concerns about their reliability and
security. Traditional attacks rely on fixed-norm perturbations, misaligning
with human perception. In contrast, diffusion-based attacks require pre-trained
models, demanding substantial data when these models are unavailable, limiting
practical use in data-scarce scenarios. In medical imaging, however, this is
often unfeasible due to the limited availability of datasets. Building on
recent advancements in learnable prompts, we propose Prompt2Perturb (P2P), a
novel language-guided attack method capable of generating meaningful attack
examples driven by text instructions. During the prompt learning phase, our
approach leverages learnable prompts within the text encoder to create subtle,
yet impactful, perturbations that remain imperceptible while guiding the model
towards targeted outcomes. In contrast to current prompt learning-based
approaches, our P2P stands out by directly updating text embeddings, avoiding
the need for retraining diffusion models. Further, we leverage the finding that
optimizing only the early reverse diffusion steps boosts efficiency while
ensuring that the generated adversarial examples incorporate subtle noise, thus
preserving ultrasound image quality without introducing noticeable artifacts.
We show that our method outperforms state-of-the-art attack techniques across
three breast ultrasound datasets in FID and LPIPS. Moreover, the generated
images are both more natural in appearance and more effective compared to
existing adversarial attacks. Our code will be publicly available
https://github.com/yasamin-med/P2P.


http://arxiv.org/abs/2412.10353
Robust image classification with multi-modal large language models. (99%)
Francesco Villani; Igor Maljkovic; Dario Lazzaro; Angelo Sotgiu; Antonio Emanuele Cinà; Fabio Roli
  Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully
crafted input samples that can cause models to make incorrect predictions with
high confidence. To mitigate these vulnerabilities, adversarial training and
detection-based defenses have been proposed to strengthen models in advance.
However, most of these approaches focus on a single data modality, overlooking
the relationships between visual patterns and textual descriptions of the
input. In this paper, we propose a novel defense, MultiShield, designed to
combine and complement these defenses with multi-modal information to further
enhance their robustness. MultiShield leverages multi-modal large language
models to detect adversarial examples and abstain from uncertain
classifications when there is no alignment between textual and visual
representations of the input. Extensive evaluations on CIFAR-10 and ImageNet
datasets, using robust and non-robust image classification models, demonstrate
that MultiShield can be easily integrated to detect and reject adversarial
examples, outperforming the original defenses.


http://arxiv.org/abs/2412.09954
A2RNet: Adversarial Attack Resilient Network for Robust Infrared and Visible Image Fusion. (92%)
Jiawei Li; Hongwei Yu; Jiansheng Chen; Xinlong Ding; Jinlong Wang; Jinyuan Liu; Bochao Zou; Huimin Ma
  Infrared and visible image fusion (IVIF) is a crucial technique for enhancing
visual performance by integrating unique information from different modalities
into one fused image. Exiting methods pay more attention to conducting fusion
with undisturbed data, while overlooking the impact of deliberate interference
on the effectiveness of fusion results. To investigate the robustness of fusion
models, in this paper, we propose a novel adversarial attack resilient network,
called $\textrm{A}^{\textrm{2}}$RNet. Specifically, we develop an adversarial
paradigm with an anti-attack loss function to implement adversarial attacks and
training. It is constructed based on the intrinsic nature of IVIF and provide a
robust foundation for future research advancements. We adopt a Unet as the
pipeline with a transformer-based defensive refinement module (DRM) under this
paradigm, which guarantees fused image quality in a robust coarse-to-fine
manner. Compared to previous works, our method mitigates the adverse effects of
adversarial perturbations, consistently maintaining high-fidelity fusion
results. Furthermore, the performance of downstream tasks can also be well
maintained under adversarial attacks. Code is available at
https://github.com/lok-18/A2RNet.


http://arxiv.org/abs/2412.10597
Err on the Side of Texture: Texture Bias on Real Data. (82%)
Blaine Hoak; Ryan Sheatsley; Patrick McDaniel
  Bias significantly undermines both the accuracy and trustworthiness of
machine learning models. To date, one of the strongest biases observed in image
classification models is texture bias-where models overly rely on texture
information rather than shape information. Yet, existing approaches for
measuring and mitigating texture bias have not been able to capture how
textures impact model robustness in real-world settings. In this work, we
introduce the Texture Association Value (TAV), a novel metric that quantifies
how strongly models rely on the presence of specific textures when classifying
objects. Leveraging TAV, we demonstrate that model accuracy and robustness are
heavily influenced by texture. Our results show that texture bias explains the
existence of natural adversarial examples, where over 90% of these samples
contain textures that are misaligned with the learned texture of their true
label, resulting in confident mispredictions.


http://arxiv.org/abs/2412.10617
BinarySelect to Improve Accessibility of Black-Box Attack Research. (80%)
Shatarupa Ghosh; Jonathan Rusert
  Adversarial text attack research is useful for testing the robustness of NLP
models, however, the rise of transformers has greatly increased the time
required to test attacks. Especially when researchers do not have access to
adequate resources (e.g. GPUs). This can hinder attack research, as modifying
one example for an attack can require hundreds of queries to a model,
especially for black-box attacks. Often these attacks remove one token at a
time to find the ideal one to change, requiring $n$ queries (the length of the
text) right away. We propose a more efficient selection method called
BinarySelect which combines binary search and attack selection methods to
greatly reduce the number of queries needed to find a token. We find that
BinarySelect only needs $\text{log}_2(n) * 2$ queries to find the first token
compared to $n$ queries. We also test BinarySelect in an attack setting against
5 classifiers across 3 datasets and find a viable tradeoff between number of
queries saved and attack effectiveness. For example, on the Yelp dataset, the
number of queries is reduced by 32% (72 less) with a drop in attack
effectiveness of only 5 points. We believe that BinarySelect can help future
researchers study adversarial attacks and black-box problems more efficiently
and opens the door for researchers with access to less resources.


http://arxiv.org/abs/2412.09921
FaceShield: Defending Facial Image against Deepfake Threats. (80%)
Jaehwan Jeong; Sumin In; Sieun Kim; Hannie Shin; Jongheon Jeong; Sang Ho Yoon; Jaewook Chung; Sangpil Kim
  The rising use of deepfakes in criminal activities presents a significant
issue, inciting widespread controversy. While numerous studies have tackled
this problem, most primarily focus on deepfake detection. These reactive
solutions are insufficient as a fundamental approach for crimes where
authenticity is disregarded. Existing proactive defenses also have limitations,
as they are effective only for deepfake models based on specific Generative
Adversarial Networks (GANs), making them less applicable in light of recent
advancements in diffusion-based models. In this paper, we propose a proactive
defense method named FaceShield, which introduces novel defense strategies
targeting deepfakes generated by Diffusion Models (DMs) and facilitates
defenses on various existing GAN-based deepfake models through facial feature
extractor manipulations. Our approach consists of three main components: (i)
manipulating the attention mechanism of DMs to exclude protected facial
features during the denoising process, (ii) targeting prominent facial feature
extraction models to enhance the robustness of our adversarial perturbation,
and (iii) employing Gaussian blur and low-pass filtering techniques to improve
imperceptibility while enhancing robustness against JPEG compression.
Experimental results on the CelebA-HQ and VGGFace2-HQ datasets demonstrate that
our method achieves state-of-the-art performance against the latest deepfake
models based on DMs, while also exhibiting transferability to GANs and
showcasing greater imperceptibility of noise along with enhanced robustness.


http://arxiv.org/abs/2412.10535
On Adversarial Robustness and Out-of-Distribution Robustness of Large Language Models. (78%)
April Yang; Jordan Tab; Parth Shah; Paul Kotchavong
  The increasing reliance on large language models (LLMs) for diverse
applications necessitates a thorough understanding of their robustness to
adversarial perturbations and out-of-distribution (OOD) inputs. In this study,
we investigate the correlation between adversarial robustness and OOD
robustness in LLMs, addressing a critical gap in robustness evaluation. By
applying methods originally designed to improve one robustness type across both
contexts, we analyze their performance on adversarial and out-of-distribution
benchmark datasets. The input of the model consists of text samples, with the
output prediction evaluated in terms of accuracy, precision, recall, and F1
scores in various natural language inference tasks.
  Our findings highlight nuanced interactions between adversarial robustness
and OOD robustness, with results indicating limited transferability between the
two robustness types. Through targeted ablations, we evaluate how these
correlations evolve with different model sizes and architectures, uncovering
model-specific trends: smaller models like LLaMA2-7b exhibit neutral
correlations, larger models like LLaMA2-13b show negative correlations, and
Mixtral demonstrates positive correlations, potentially due to domain-specific
alignment. These results underscore the importance of hybrid robustness
frameworks that integrate adversarial and OOD strategies tailored to specific
models and domains. Further research is needed to evaluate these interactions
across larger models and varied architectures, offering a pathway to more
reliable and generalizable LLMs.


http://arxiv.org/abs/2412.10049
SuperMark: Robust and Training-free Image Watermarking via Diffusion-based Super-Resolution. (67%)
Runyi Hu; Jie Zhang; Yiming Li; Jiwei Li; Qing Guo; Han Qiu; Tianwei Zhang
  In today's digital landscape, the blending of AI-generated and authentic
content has underscored the need for copyright protection and content
authentication. Watermarking has become a vital tool to address these
challenges, safeguarding both generated and real content. Effective
watermarking methods must withstand various distortions and attacks. Current
deep watermarking techniques often use an encoder-noise layer-decoder
architecture and include distortions to enhance robustness. However, they
struggle to balance robustness and fidelity and remain vulnerable to adaptive
attacks, despite extensive training. To overcome these limitations, we propose
SuperMark, a robust, training-free watermarking framework. Inspired by the
parallels between watermark embedding/extraction in watermarking and the
denoising/noising processes in diffusion models, SuperMark embeds the watermark
into initial Gaussian noise using existing techniques. It then applies
pre-trained Super-Resolution (SR) models to denoise the watermarked noise,
producing the final watermarked image. For extraction, the process is reversed:
the watermarked image is inverted back to the initial watermarked noise via
DDIM Inversion, from which the embedded watermark is extracted. This flexible
framework supports various noise injection methods and diffusion-based SR
models, enabling enhanced customization. The robustness of the DDIM Inversion
process against perturbations allows SuperMark to achieve strong resilience to
distortions while maintaining high fidelity. Experiments demonstrate that
SuperMark achieves fidelity comparable to existing methods while significantly
improving robustness. Under standard distortions, it achieves an average
watermark extraction accuracy of 99.46%, and 89.29% under adaptive attacks.
Moreover, SuperMark shows strong transferability across datasets, SR models,
embedding methods, and resolutions.


http://arxiv.org/abs/2412.10605
Client-Side Patching against Backdoor Attacks in Federated Learning. (61%)
Borja Molina-Coronado
  Federated learning is a versatile framework for training models in
decentralized environments. However, the trust placed in clients makes
federated learning vulnerable to backdoor attacks launched by malicious
participants. While many defenses have been proposed, they often fail short
when facing heterogeneous data distributions among participating clients. In
this paper, we propose a novel defense mechanism for federated learning systems
designed to mitigate backdoor attacks on the clients-side. Our approach
leverages adversarial learning techniques and model patching to neutralize the
impact of backdoor attacks. Through extensive experiments on the MNIST and
Fashion-MNIST datasets, we demonstrate that our defense effectively reduces
backdoor accuracy, outperforming existing state-of-the-art defenses, such as
LFighter, FLAME, and RoseAgg, in i.i.d. and non-i.i.d. scenarios, while
maintaining competitive or superior accuracy on clean data.


http://arxiv.org/abs/2412.12192
No Free Lunch for Defending Against Prefilling Attack by In-Context Learning. (22%)
Zhiyu Xue; Guangliang Liu; Bocheng Chen; Kristen Marie Johnson; Ramtin Pedarsani
  The security of Large Language Models (LLMs) has become an important research
topic since the emergence of ChatGPT. Though there have been various effective
methods to defend against jailbreak attacks, prefilling attacks remain an
unsolved and popular threat against open-sourced LLMs. In-Context Learning
(ICL) offers a computationally efficient defense against various jailbreak
attacks, yet no effective ICL methods have been developed to counter prefilling
attacks. In this paper, we: (1) show that ICL can effectively defend against
prefilling jailbreak attacks by employing adversative sentence structures
within demonstrations; (2) characterize the effectiveness of this defense
through the lens of model size, number of demonstrations, over-defense,
integration with other jailbreak attacks, and the presence of safety alignment.
Given the experimental results and our analysis, we conclude that there is no
free lunch for defending against prefilling jailbreak attacks with ICL. On the
one hand, current safety alignment methods fail to mitigate prefilling
jailbreak attacks, but adversative structures within ICL demonstrations provide
robust defense across various model sizes and complex jailbreak attacks. On the
other hand, LLMs exhibit similar over-defensiveness when utilizing ICL
demonstrations with adversative structures, and this behavior appears to be
independent of model size.


http://arxiv.org/abs/2412.09933
Active Poisoning: Efficient Backdoor Attacks on Transfer Learning-Based Brain-Computer Interfaces. (13%)
X. Jiang; L. Meng; S. Li; D. Wu
  Transfer learning (TL) has been widely used in electroencephalogram
(EEG)-based brain-computer interfaces (BCIs) for reducing calibration efforts.
However, backdoor attacks could be introduced through TL. In such attacks, an
attacker embeds a backdoor with a specific pattern into the machine learning
model. As a result, the model will misclassify a test sample with the backdoor
trigger into a prespecified class while still maintaining good performance on
benign samples. Accordingly, this study explores backdoor attacks in the TL of
EEG-based BCIs, where source-domain data are poisoned by a backdoor trigger and
then used in TL. We propose several active poisoning approaches to select
source-domain samples, which are most effective in embedding the backdoor
pattern, to improve the attack success rate and efficiency. Experiments on four
EEG datasets and three deep learning models demonstrate the effectiveness of
the approaches. To our knowledge, this is the first study about backdoor
attacks on TL models in EEG-based BCIs. It exposes a serious security risk in
BCIs, which should be immediately addressed.


http://arxiv.org/abs/2412.10265
Adversarial Robustness of Bottleneck Injected Deep Neural Networks for Task-Oriented Communication. (10%)
Alireza Furutanpey; Pantelis A. Frangoudis; Patrik Szabo; Schahram Dustdar
  This paper investigates the adversarial robustness of Deep Neural Networks
(DNNs) using Information Bottleneck (IB) objectives for task-oriented
communication systems. We empirically demonstrate that while IB-based
approaches provide baseline resilience against attacks targeting downstream
tasks, the reliance on generative models for task-oriented communication
introduces new vulnerabilities. Through extensive experiments on several
datasets, we analyze how bottleneck depth and task complexity influence
adversarial robustness. Our key findings show that Shallow Variational
Bottleneck Injection (SVBI) provides less adversarial robustness compared to
Deep Variational Information Bottleneck (DVIB) approaches, with the gap
widening for more complex tasks. Additionally, we reveal that IB-based
objectives exhibit stronger robustness against attacks focusing on salient
pixels with high intensity compared to those perturbing many pixels with lower
intensity. Lastly, we demonstrate that task-oriented communication systems that
rely on generative models to extract and recover salient information have an
increased attack surface. The results highlight important security
considerations for next-generation communication systems that leverage neural
networks for goal-oriented compression.


http://arxiv.org/abs/2412.10186
BiCert: A Bilinear Mixed Integer Programming Formulation for Precise Certified Bounds Against Data Poisoning Attacks. (1%)
Tobias Lorenz; Marta Kwiatkowska; Mario Fritz
  Data poisoning attacks pose one of the biggest threats to modern AI systems,
necessitating robust defenses. While extensive efforts have been made to
develop empirical defenses, attackers continue to evolve, creating
sophisticated methods to circumvent these measures. To address this, we must
move beyond empirical defenses and establish provable certification methods
that guarantee robustness. This paper introduces a novel certification
approach, BiCert, using Bilinear Mixed Integer Programming (BMIP) to compute
sound deterministic bounds that provide such provable robustness. Using BMIP,
we compute the reachable set of parameters that could result from training with
potentially manipulated data. A key element to make this computation feasible
is to relax the reachable parameter set to a convex set between training
iterations. At test time, this parameter set allows us to predict all possible
outcomes, guaranteeing robustness. BiCert is more precise than previous
methods, which rely solely on interval and polyhedral bounds. Crucially, our
approach overcomes the fundamental limitation of prior approaches where
parameter bounds could only grow, often uncontrollably. We show that BiCert's
tighter bounds eliminate a key source of divergence issues, resulting in more
stable training and higher certified accuracy.


http://arxiv.org/abs/2412.10198
From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection. (1%)
Haowei Wang; Rupeng Zhang; Junjie Wang; Mingyang Li; Yuekai Huang; Dandan Wang; Qing Wang
  Tool-calling has changed Large Language Model (LLM) applications by
integrating external tools, significantly enhancing their functionality across
diverse tasks. However, this integration also introduces new security
vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which
have not been extensively studied. To fill this gap, we present ToolCommander,
a novel framework designed to exploit vulnerabilities in LLM tool-calling
systems through adversarial tool injection. Our framework employs a
well-designed two-stage attack strategy. Firstly, it injects malicious tools to
collect user queries, then dynamically updates the injected tools based on the
stolen information to enhance subsequent attacks. These stages enable
ToolCommander to execute privacy theft, launch denial-of-service attacks, and
even manipulate business competition by triggering unscheduled tool-calling.
Notably, the ASR reaches 91.67% for privacy theft and hits 100% for
denial-of-service and unscheduled tool calling in certain cases. Our work
demonstrates that these vulnerabilities can lead to severe consequences beyond
simple misuse of tool-calling systems, underscoring the urgent need for robust
defensive strategies to secure LLM Tool-calling systems.


http://arxiv.org/abs/2412.09692
Three-in-One: Robust Enhanced Universal Transferable Anti-Facial Retrieval in Online Social Networks. (99%)
Yunna Lv; Long Tang; Dengpan Ye; Caiyun Xie; Jiacheng Deng; Yiheng He
  Deep hash-based retrieval techniques are widely used in facial retrieval
systems to improve the efficiency of facial matching. However, it also carries
the danger of exposing private information. Deep hash models are easily
influenced by adversarial examples, which can be leveraged to protect private
images from malicious retrieval. The existing adversarial example methods
against deep hash models focus on universality and transferability, lacking the
research on its robustness in online social networks (OSNs), which leads to
their failure in anti-retrieval after post-processing. Therefore, we provide
the first in-depth discussion on robustness adversarial perturbation in
universal transferable anti-facial retrieval and propose Three-in-One
Adversarial Perturbation (TOAP). Specifically, we construct a local and global
Compression Generator (CG) to simulate complex post-processing scenarios, which
can be used to mitigate perturbation. Then, we propose robust optimization
objectives based on the discovery of the variation patterns of model's
distribution after post-processing, and generate adversarial examples using
these objectives and meta-learning. Finally, we iteratively optimize
perturbation by alternately generating adversarial examples and fine-tuning the
CG, balancing the performance of perturbation while enhancing CG's ability to
mitigate them. Numerous experiments demonstrate that, in addition to its
advantages in universality and transferability, TOAP significantly outperforms
current state-of-the-art methods in multiple robustness metrics. It further
improves universality and transferability by 5% to 28%, and achieves up to
about 33% significant improvement in several simulated post-processing
scenarios as well as mainstream OSNs, demonstrating that TOAP can effectively
protect private images from malicious retrieval in real-world scenarios.


http://arxiv.org/abs/2412.09195
On the Generation and Removal of Speaker Adversarial Perturbation for Voice-Privacy Protection. (95%)
Chenyang Guo; Liping Chen; Zhuhai Li; Kong Aik Lee; Zhen-Hua Ling; Wu Guo
  Neural networks are commonly known to be vulnerable to adversarial attacks
mounted through subtle perturbation on the input data. Recent development in
voice-privacy protection has shown the positive use cases of the same technique
to conceal speaker's voice attribute with additive perturbation signal
generated by an adversarial network. This paper examines the reversibility
property where an entity generating the adversarial perturbations is authorized
to remove them and restore original speech (e.g., the speaker him/herself). A
similar technique could also be used by an investigator to deanonymize a
voice-protected speech to restore criminals' identities in security and
forensic analysis. In this setting, the perturbation generative module is
assumed to be known in the removal process. To this end, a joint training of
perturbation generation and removal modules is proposed. Experimental results
on the LibriSpeech dataset demonstrated that the subtle perturbations added to
the original speech can be predicted from the anonymized speech while achieving
the goal of privacy protection. By removing these perturbations from the
anonymized sample, the original speech can be restored. Audio samples can be
found in \url{https://voiceprivacy.github.io/Perturbation-Generation-Removal/}.


http://arxiv.org/abs/2412.09844
Real-time Identity Defenses against Malicious Personalization of Diffusion Models. (95%)
Hanzhong Guo; Shen Nie; Chao Du; Tianyu Pang; Hao Sun; Chongxuan Li
  Personalized diffusion models, capable of synthesizing highly realistic
images based on a few reference portraits, pose substantial social, ethical,
and legal risks by enabling identity replication. Existing defense mechanisms
rely on computationally intensive adversarial perturbations tailored to
individual images, rendering them impractical for real-world deployment. This
study introduces Real-time Identity Defender (RID), a neural network designed
to generate adversarial perturbations through a single forward pass, bypassing
the need for image-specific optimization. RID achieves unprecedented
efficiency, with defense times as low as 0.12 seconds on a single GPU (4,400
times faster than leading methods) and 1.1 seconds per image on a standard
Intel i9 CPU, making it suitable for edge devices such as smartphones. Despite
its efficiency, RID matches state-of-the-art performance across visual and
quantitative benchmarks, effectively mitigating identity replication risks. Our
analysis reveals that RID's perturbations mimic the efficacy of traditional
defenses while exhibiting properties distinct from natural noise, such as
Gaussian perturbations. To enhance robustness, we extend RID into an ensemble
framework that integrates multiple pre-trained text-to-image diffusion models,
ensuring resilience against black-box attacks and post-processing techniques,
including JPEG compression and diffusion-based purification.


http://arxiv.org/abs/2412.08969
Deep Learning Model Security: Threats and Defenses. (92%)
Tianyang Wang; Ziqian Bi; Yichao Zhang; Ming Liu; Weiche Hsieh; Pohsun Feng; Lawrence K. Q. Yan; Yizhu Wen; Benji Peng; Junyu Liu; Keyu Chen; Sen Zhang; Ming Li; Chuanqi Jiang; Xinyuan Song; Junjie Yang; Bowen Jing; Jintao Ren; Junhao Song; Hong-Ming Tseng; Silin Chen; Yunze Wang; Chia Xin Liang; Jiawei Xu; Xuanhe Pan; Jinlang Wang; Qian Niu
  Deep learning has transformed AI applications but faces critical security
challenges, including adversarial attacks, data poisoning, model theft, and
privacy leakage. This survey examines these vulnerabilities, detailing their
mechanisms and impact on model integrity and confidentiality. Practical
implementations, including adversarial examples, label flipping, and backdoor
attacks, are explored alongside defenses such as adversarial training,
differential privacy, and federated learning, highlighting their strengths and
limitations.
  Advanced methods like contrastive and self-supervised learning are presented
for enhancing robustness. The survey concludes with future directions,
emphasizing automated defenses, zero-trust architectures, and the security
challenges of large AI models. A balanced approach to performance and security
is essential for developing reliable deep learning systems.


http://arxiv.org/abs/2412.09450
A Semi Black-Box Adversarial Bit-Flip Attack with Limited DNN Model Information. (69%)
Behnam Ghavami; Mani Sadati; Mohammad Shahidzadeh; Lesley Shannon; Steve Wilton
  Despite the rising prevalence of deep neural networks (DNNs) in
cyber-physical systems, their vulnerability to adversarial bit-flip attacks
(BFAs) is a noteworthy concern. This paper proposes B3FA, a semi-black-box
BFA-based parameter attack on DNNs, assuming the adversary has limited
knowledge about the model. We consider practical scenarios often feature a more
restricted threat model for real-world systems, contrasting with the typical
BFA models that presuppose the adversary's full access to a network's inputs
and parameters. The introduced bit-flip approach utilizes a magnitude-based
ranking method and a statistical re-construction technique to identify the
vulnerable bits. We demonstrate the effectiveness of B3FA on several DNN models
in a semi-black-box setting. For example, B3FA could drop the accuracy of a
MobileNetV2 from 69.84% to 9% with only 20 bit-flips in a real-world setting.


http://arxiv.org/abs/2412.09150
Evaluating Adversarial Attacks on Traffic Sign Classifiers beyond Standard Baselines. (45%)
Svetlana Pavlitska; Leopold Müller; J. Marius Zöllner
  Adversarial attacks on traffic sign classification models were among the
first successfully tried in the real world. Since then, the research in this
area has been mainly restricted to repeating baseline models, such as LISA-CNN
or GTSRB-CNN, and similar experiment settings, including white and black
patches on traffic signs. In this work, we decouple model architectures from
the datasets and evaluate on further generic models to make a fair comparison.
Furthermore, we compare two attack settings, inconspicuous and visible, which
are usually regarded without direct comparison. Our results show that standard
baselines like LISA-CNN or GTSRB-CNN are significantly more susceptible than
the generic ones. We, therefore, suggest evaluating new attacks on a broader
spectrum of baselines in the future. Our code is available at
\url{https://github.com/KASTEL-MobilityLab/attacks-on-traffic-sign-recognition/}.


http://arxiv.org/abs/2412.09073
SVasP: Self-Versatility Adversarial Style Perturbation for Cross-Domain Few-Shot Learning. (3%)
Wenqian Li; Pengfei Fang; Hui Xue
  Cross-Domain Few-Shot Learning (CD-FSL) aims to transfer knowledge from seen
source domains to unseen target domains, which is crucial for evaluating the
generalization and robustness of models. Recent studies focus on utilizing
visual styles to bridge the domain gap between different domains. However, the
serious dilemma of gradient instability and local optimization problem occurs
in those style-based CD-FSL methods. This paper addresses these issues and
proposes a novel crop-global style perturbation method, called
\underline{\textbf{S}}elf-\underline{\textbf{V}}ersatility
\underline{\textbf{A}}dversarial \underline{\textbf{S}}tyle
\underline{\textbf{P}}erturbation (\textbf{SVasP}), which enhances the gradient
stability and escapes from poor sharp minima jointly. Specifically, SVasP
simulates more diverse potential target domain adversarial styles via
diversifying input patterns and aggregating localized crop style gradients, to
serve as global style perturbation stabilizers within one image, a concept we
refer to as self-versatility. Then a novel objective function is proposed to
maximize visual discrepancy while maintaining semantic consistency between
global, crop, and adversarial features. Having the stabilized global style
perturbation in the training phase, one can obtain a flattened minima in the
loss landscape, boosting the transferability of the model to the target
domains. Extensive experiments on multiple benchmark datasets demonstrate that
our method significantly outperforms existing state-of-the-art methods. Our
codes are available at https://github.com/liwenqianSEU/SVasP.


http://arxiv.org/abs/2412.09565
Obfuscated Activations Bypass LLM Latent-Space Defenses. (2%)
Luke Bailey; Alex Serrano; Abhay Sheshadri; Mikhail Seleznyov; Jordan Taylor; Erik Jenner; Jacob Hilton; Stephen Casper; Carlos Guestrin; Scott Emmons
  Recent latent-space monitoring techniques have shown promise as defenses
against LLM attacks. These defenses act as scanners that seek to detect harmful
activations before they lead to undesirable actions. This prompts the question:
Can models execute harmful behavior via inconspicuous latent states? Here, we
study such obfuscated activations. We show that state-of-the-art latent-space
defenses -- including sparse autoencoders, representation probing, and latent
OOD detection -- are all vulnerable to obfuscated activations. For example,
against probes trained to classify harmfulness, our attacks can often reduce
recall from 100% to 0% while retaining a 90% jailbreaking rate. However,
obfuscation has limits: we find that on a complex task (writing SQL code),
obfuscation reduces model performance. Together, our results demonstrate that
neural activations are highly malleable: we can reshape activation patterns in
a variety of ways, often while preserving a network's behavior. This poses a
fundamental challenge to latent-space defenses.


http://arxiv.org/abs/2412.09269
Towards Understanding the Robustness of LLM-based Evaluations under Perturbations. (1%)
Manav Chaudhary; Harshit Gupta; Savita Bhat; Vasudeva Varma
  Traditional evaluation metrics like BLEU and ROUGE fall short when capturing
the nuanced qualities of generated text, particularly when there is no single
ground truth. In this paper, we explore the potential of Large Language Models
(LLMs), specifically Google Gemini 1, to serve as automatic evaluators for
non-standardized metrics in summarization and dialog-based tasks. We conduct
experiments across multiple prompting strategies to examine how LLMs fare as
quality evaluators when compared with human judgments on the SummEval and USR
datasets, asking the model to generate both a score as well as a justification
for the score. Furthermore, we explore the robustness of the LLM evaluator by
using perturbed inputs. Our findings suggest that while LLMs show promise,
their alignment with human evaluators is limited, they are not robust against
perturbations and significant improvements are required for their standalone
use as reliable evaluators for subjective metrics.


http://arxiv.org/abs/2412.09765
L-WISE: Boosting Human Image Category Learning Through Model-Based Image Selection And Enhancement. (1%)
Morgan B. Talbot; Gabriel Kreiman; James J. DiCarlo; Guy Gaziv
  The currently leading artificial neural network (ANN) models of the visual
ventral stream -- which are derived from a combination of performance
optimization and robustification methods -- have demonstrated a remarkable
degree of behavioral alignment with humans on visual categorization tasks.
Extending upon previous work, we show that not only can these models guide
image perturbations that change the induced human category percepts, but they
also can enhance human ability to accurately report the original ground truth.
Furthermore, we find that the same models can also be used out-of-the-box to
predict the proportion of correct human responses to individual images,
providing a simple, human-aligned estimator of the relative difficulty of each
image. Motivated by these observations, we propose to augment visual learning
in humans in a way that improves human categorization accuracy at test time.
Our learning augmentation approach consists of (i) selecting images based on
their model-estimated recognition difficulty, and (ii) using image
perturbations that aid recognition for novice learners. We find that combining
these model-based strategies gives rise to test-time categorization accuracy
gains of 33-72% relative to control subjects without these interventions,
despite using the same number of training feedback trials. Surprisingly, beyond
the accuracy gain, the training time for the augmented learning group was also
shorter by 20-23%. We demonstrate the efficacy of our approach in a
fine-grained categorization task with natural images, as well as tasks in two
clinically relevant image domains -- histology and dermoscopy -- where visual
learning is notoriously challenging. To the best of our knowledge, this is the
first application of ANNs to increase visual learning performance in humans by
enhancing category-specific features.


http://arxiv.org/abs/2412.08394
Adversarial Purification by Consistency-aware Latent Space Optimization on Data Manifolds. (99%)
Shuhai Zhang; Jiahao Yang; Hui Luo; Jie Chen; Li Wang; Feng Liu; Bo Han; Mingkui Tan
  Deep neural networks (DNNs) are vulnerable to adversarial samples crafted by
adding imperceptible perturbations to clean data, potentially leading to
incorrect and dangerous predictions. Adversarial purification has been an
effective means to improve DNNs robustness by removing these perturbations
before feeding the data into the model. However, it faces significant
challenges in preserving key structural and semantic information of data, as
the imperceptible nature of adversarial perturbations makes it hard to avoid
over-correcting, which can destroy important information and degrade model
performance. In this paper, we break away from traditional adversarial
purification methods by focusing on the clean data manifold. To this end, we
reveal that samples generated by a well-trained generative model are close to
clean ones but far from adversarial ones. Leveraging this insight, we propose
Consistency Model-based Adversarial Purification (CMAP), which optimizes
vectors within the latent space of a pre-trained consistency model to generate
samples for restoring clean data. Specifically, 1) we propose a
\textit{Perceptual consistency restoration} mechanism by minimizing the
discrepancy between generated samples and input samples in both pixel and
perceptual spaces. 2) To maintain the optimized latent vectors within the valid
data manifold, we introduce a \textit{Latent distribution consistency
constraint} strategy to align generated samples with the clean data
distribution. 3) We also apply a \textit{Latent vector consistency prediction}
scheme via an ensemble approach to enhance prediction reliability. CMAP
fundamentally addresses adversarial perturbations at their source, providing a
robust purification. Extensive experiments on CIFAR-10 and ImageNet-100 show
that our CMAP significantly enhances robustness against strong adversarial
attacks while preserving high natural accuracy.


http://arxiv.org/abs/2412.08108
Doubly-Universal Adversarial Perturbations: Deceiving Vision-Language Models Across Both Images and Text with a Single Perturbation. (98%)
Hee-Seon Kim; Minbeom Kim; Changick Kim
  Large Vision-Language Models (VLMs) have demonstrated remarkable performance
across multimodal tasks by integrating vision encoders with large language
models (LLMs). However, these models remain vulnerable to adversarial attacks.
Among such attacks, Universal Adversarial Perturbations (UAPs) are especially
powerful, as a single optimized perturbation can mislead the model across
various input images. In this work, we introduce a novel UAP specifically
designed for VLMs: the Doubly-Universal Adversarial Perturbation (Doubly-UAP),
capable of universally deceiving VLMs across both image and text inputs. To
successfully disrupt the vision encoder's fundamental process, we analyze the
core components of the attention mechanism. After identifying value vectors in
the middle-to-late layers as the most vulnerable, we optimize Doubly-UAP in a
label-free manner with a frozen model. Despite being developed as a black-box
to the LLM, Doubly-UAP achieves high attack success rates on VLMs, consistently
outperforming baseline methods across vision-language tasks. Extensive ablation
studies and analyses further demonstrate the robustness of Doubly-UAP and
provide insights into how it influences internal attention mechanisms.


http://arxiv.org/abs/2412.08555
Grimm: A Plug-and-Play Perturbation Rectifier for Graph Neural Networks Defending against Poisoning Attacks. (93%)
Ao Liu; Wenshan Li; Beibei Li; Wengang Ma; Tao Li; Pan Zhou
  Recent studies have revealed the vulnerability of graph neural networks
(GNNs) to adversarial poisoning attacks on node classification tasks. Current
defensive methods require substituting the original GNNs with defense models,
regardless of the original's type. This approach, while targeting adversarial
robustness, compromises the enhancements developed in prior research to boost
GNNs' practical performance. Here we introduce Grimm, the first plug-and-play
defense model. With just a minimal interface requirement for extracting
features from any layer of the protected GNNs, Grimm is thus enabled to
seamlessly rectify perturbations. Specifically, we utilize the feature
trajectories (FTs) generated by GNNs, as they evolve through epochs, to reflect
the training status of the networks. We then theoretically prove that the FTs
of victim nodes will inevitably exhibit discriminable anomalies. Consequently,
inspired by the natural parallelism between the biological nervous and immune
systems, we construct Grimm, a comprehensive artificial immune system for GNNs.
Grimm not only detects abnormal FTs and rectifies adversarial edges during
training but also operates efficiently in parallel, thereby mirroring the
concurrent functionalities of its biological counterparts. We experimentally
confirm that Grimm offers four empirically validated advantages: 1)
Harmlessness, as it does not actively interfere with GNN training; 2)
Parallelism, ensuring monitoring, detection, and rectification functions
operate independently of the GNN training process; 3) Generalizability,
demonstrating compatibility with mainstream GNNs such as GCN, GAT, and
GraphSAGE; and 4) Transferability, as the detectors for abnormal FTs can be
efficiently transferred across different systems for one-step rectification.


http://arxiv.org/abs/2412.08615
Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models. (83%)
Jiahui Li; Yongchang Hao; Haoyu Xu; Xing Wang; Yu Hong
  Despite the advancements in training Large Language Models (LLMs) with
alignment techniques to enhance the safety of generated content, these models
remain susceptible to jailbreak, an adversarial attack method that exposes
security vulnerabilities in LLMs. Notably, the Greedy Coordinate Gradient (GCG)
method has demonstrated the ability to automatically generate adversarial
suffixes that jailbreak state-of-the-art LLMs. However, the optimization
process involved in GCG is highly time-consuming, rendering the jailbreaking
pipeline inefficient. In this paper, we investigate the process of GCG and
identify an issue of Indirect Effect, the key bottleneck of the GCG
optimization. To this end, we propose the Model Attack Gradient Index GCG
(MAGIC), that addresses the Indirect Effect by exploiting the gradient
information of the suffix tokens, thereby accelerating the procedure by having
less computation and fewer iterations. Our experiments on AdvBench show that
MAGIC achieves up to a 1.5x speedup, while maintaining Attack Success Rates
(ASR) on par or even higher than other baselines. Our MAGIC achieved an ASR of
74% on the Llama-2 and an ASR of 54% when conducting transfer attacks on
GPT-3.5. Code is available at https://github.com/jiah-li/magic.


http://arxiv.org/abs/2412.08608
AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models. (82%)
Mintong Kang; Chejian Xu; Bo Li
  Recent advancements in large audio-language models (LALMs) have enabled
speech-based user interactions, significantly enhancing user experience and
accelerating the deployment of LALMs in real-world applications. However,
ensuring the safety of LALMs is crucial to prevent risky outputs that may raise
societal concerns or violate AI regulations. Despite the importance of this
issue, research on jailbreaking LALMs remains limited due to their recent
emergence and the additional technical challenges they present compared to
attacks on DNN-based audio models. Specifically, the audio encoders in LALMs,
which involve discretization operations, often lead to gradient shattering,
hindering the effectiveness of attacks relying on gradient-based optimizations.
The behavioral variability of LALMs further complicates the identification of
effective (adversarial) optimization targets. Moreover, enforcing stealthiness
constraints on adversarial audio waveforms introduces a reduced, non-convex
feasible solution space, further intensifying the challenges of the
optimization process. To overcome these challenges, we develop AdvWave, the
first jailbreak framework against LALMs. We propose a dual-phase optimization
method that addresses gradient shattering, enabling effective end-to-end
gradient-based optimization. Additionally, we develop an adaptive adversarial
target search algorithm that dynamically adjusts the adversarial optimization
target based on the response patterns of LALMs for specific queries. To ensure
that adversarial audio remains perceptually natural to human listeners, we
design a classifier-guided optimization approach that generates adversarial
noise resembling common urban sounds. Extensive evaluations on multiple
advanced LALMs demonstrate that AdvWave outperforms baseline methods, achieving
a 40% higher average jailbreak attack success rate.


http://arxiv.org/abs/2412.08755
Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images. (45%)
Kyle Stein; Andrew Arash Mahyari; Guillermo Francia; Eman El-Sheikh
  Backdoor attacks pose a critical threat by embedding hidden triggers into
inputs, causing models to misclassify them into target labels. While extensive
research has focused on mitigating these attacks in object recognition models
through weight fine-tuning, much less attention has been given to detecting
backdoored samples directly. Given the vast datasets used in training, manual
inspection for backdoor triggers is impractical, and even state-of-the-art
defense mechanisms fail to fully neutralize their impact. To address this gap,
we introduce a groundbreaking method to detect unseen backdoored images during
both training and inference. Leveraging the transformative success of prompt
tuning in Vision Language Models (VLMs), our approach trains learnable text
prompts to differentiate clean images from those with hidden backdoor triggers.
Experiments demonstrate the exceptional efficacy of this method, achieving an
impressive average accuracy of 86% across two renowned datasets for detecting
unseen backdoor triggers, establishing a new standard in backdoor defense.


http://arxiv.org/abs/2412.08366
Backdoor attacks on DNN and GBDT -- A Case Study from the insurance domain. (16%)
Robin Kühlem; Daniel Otten; Daniel Ludwig; Anselm Hudde; Alexander Rosenbaum; Andreas Mauthe
Machine learning (ML) will likely play a large role in many processes in the future, also for insurance companies. However, ML models are at risk of being attacked and manipulated. In this work, the robustness of Gradient Boosted Decision Tree (GBDT) models and Deep Neural Networks (DNN) within an insurance context will be evaluated. Therefore, two GBDT models and two DNNs are trained on two different tabular datasets from an insurance context. Past research in this domain mainly used homogenous data and there are comparably few insights regarding heterogenous tabular data. The ML tasks performed on the datasets are claim prediction (regression) and fraud detection (binary classification). For the backdoor attacks different samples containing a specific pattern were crafted and added to the training data. It is shown, that this type of attack can be highly successful, even with a few added samples. The backdoor attacks worked well on the models trained on one dataset but poorly on the models trained on the other. In real-world scenarios the attacker will have to face several obstacles but as attacks can work with very few added samples this risk should be evaluated.

http://arxiv.org/abs/2412.08156
Antelope: Potent and Concealed Jailbreak Attack Strategy. (10%)
Xin Zhao; Xiaojun Chen; Haoyu Gao
  Due to the remarkable generative potential of diffusion-based models,
numerous researches have investigated jailbreak attacks targeting these
frameworks. A particularly concerning threat within image models is the
generation of Not-Safe-for-Work (NSFW) content. Despite the implementation of
security filters, numerous efforts continue to explore ways to circumvent these
safeguards. Current attack methodologies primarily encompass adversarial prompt
engineering or concept obfuscation, yet they frequently suffer from slow search
efficiency, conspicuous attack characteristics and poor alignment with targets.
To overcome these challenges, we propose Antelope, a more robust and covert
jailbreak attack strategy designed to expose security vulnerabilities inherent
in generative models. Specifically, Antelope leverages the confusion of
sensitive concepts with similar ones, facilitates searches in the semantically
adjacent space of these related concepts and aligns them with the target
imagery, thereby generating sensitive images that are consistent with the
target and capable of evading detection. Besides, we successfully exploit the
transferability of model-based attacks to penetrate online black-box services.
Experimental evaluations demonstrate that Antelope outperforms existing
baselines across multiple defensive mechanisms, underscoring its efficacy and
versatility.


http://arxiv.org/abs/2412.08201
Model-Editing-Based Jailbreak against Safety-aligned Large Language Models. (1%)
Yuxi Li; Zhibo Zhang; Kailong Wang; Ling Shi; Haoyu Wang
  Large Language Models (LLMs) have transformed numerous fields by enabling
advanced natural language interactions but remain susceptible to critical
vulnerabilities, particularly jailbreak attacks. Current jailbreak techniques,
while effective, often depend on input modifications, making them detectable
and limiting their stealth and scalability. This paper presents Targeted Model
Editing (TME), a novel white-box approach that bypasses safety filters by
minimally altering internal model structures while preserving the model's
intended functionalities. TME identifies and removes safety-critical
transformations (SCTs) embedded in model matrices, enabling malicious queries
to bypass restrictions without input modifications. By analyzing distinct
activation patterns between safe and unsafe queries, TME isolates and
approximates SCTs through an optimization process. Implemented in the D-LLM
framework, our method achieves an average Attack Success Rate (ASR) of 84.86%
on four mainstream open-source LLMs, maintaining high performance. Unlike
existing methods, D-LLM eliminates the need for specific triggers or harmful
response collections, offering a stealthier and more effective jailbreak
strategy. This work reveals a covert and robust threat vector in LLM security
and emphasizes the need for stronger safeguards in model safety alignment.


http://arxiv.org/abs/2412.07326
Addressing Key Challenges of Adversarial Attacks and Defenses in the Tabular Domain: A Methodological Framework for Coherence and Consistency. (99%)
Yael Itzhakev; Amit Giloni; Yuval Elovici; Asaf Shabtai
  Machine learning models trained on tabular data are vulnerable to adversarial
attacks, even in realistic scenarios where attackers have access only to the
model's outputs. Researchers evaluate such attacks by considering metrics like
success rate, perturbation magnitude, and query count. However, unlike other
data domains, the tabular domain contains complex interdependencies among
features, presenting a unique aspect that should be evaluated: the need for the
attack to generate coherent samples and ensure feature consistency for
indistinguishability. Currently, there is no established methodology for
evaluating adversarial samples based on these criteria. In this paper, we
address this gap by proposing new evaluation criteria tailored for tabular
attacks' quality; we defined anomaly-based framework to assess the
distinguishability of adversarial samples and utilize the SHAP explainability
technique to identify inconsistencies in the model's decision-making process
caused by adversarial samples. These criteria could form the basis for
potential detection methods and be integrated into established evaluation
metrics for assessing attack's quality Additionally, we introduce a novel
technique for perturbing dependent features while maintaining coherence and
feature consistency within the sample. We compare different attacks'
strategies, examining black-box query-based attacks and transferability-based
gradient attacks across four target models. Our experiments, conducted on
benchmark tabular datasets, reveal significant differences between the examined
attacks' strategies in terms of the attacker's risk and effort and the attacks'
quality. The findings provide valuable insights on the strengths, limitations,
and trade-offs of various adversarial attacks in the tabular domain, laying a
foundation for future research on attacks and defense development.


http://arxiv.org/abs/2412.07274
A Generative Victim Model for Segmentation. (99%)
Aixuan Li; Jing Zhang; Jiawei Shi; Yiran Zhong; Yuchao Dai
  We find that the well-trained victim models (VMs), against which the attacks
are generated, serve as fundamental prerequisites for adversarial attacks, i.e.
a segmentation VM is needed to generate attacks for segmentation. In this
context, the victim model is assumed to be robust to achieve effective
adversarial perturbation generation. Instead of focusing on improving the
robustness of the task-specific victim models, we shift our attention to image
generation. From an image generation perspective, we derive a novel VM for
segmentation, aiming to generate adversarial perturbations for segmentation
tasks without requiring models explicitly designed for image segmentation. Our
approach to adversarial attack generation diverges from conventional white-box
or black-box attacks, offering a fresh outlook on adversarial attack
strategies. Experiments show that our attack method is able to generate
effective adversarial attacks with good transferability.


http://arxiv.org/abs/2412.07277
Backdoor Attacks against No-Reference Image Quality Assessment Models via a Scalable Trigger. (99%)
Yi Yu; Song Xia; Xun Lin; Wenhan Yang; Shijian Lu; Yap-peng Tan; Alex Kot
  No-Reference Image Quality Assessment (NR-IQA), responsible for assessing the
quality of a single input image without using any reference, plays a critical
role in evaluating and optimizing computer vision systems, e.g., low-light
enhancement. Recent research indicates that NR-IQA models are susceptible to
adversarial attacks, which can significantly alter predicted scores with
visually imperceptible perturbations. Despite revealing vulnerabilities, these
attack methods have limitations, including high computational demands,
untargeted manipulation, limited practical utility in white-box scenarios, and
reduced effectiveness in black-box scenarios. To address these challenges, we
shift our focus to another significant threat and present a novel
poisoning-based backdoor attack against NR-IQA (BAIQA), allowing the attacker
to manipulate the IQA model's output to any desired target value by simply
adjusting a scaling coefficient $\alpha$ for the trigger. We propose to inject
the trigger in the discrete cosine transform (DCT) domain to improve the local
invariance of the trigger for countering trigger diminishment in NR-IQA models
due to widely adopted data augmentations. Furthermore, the universal
adversarial perturbations (UAP) in the DCT space are designed as the trigger,
to increase IQA model susceptibility to manipulation and improve attack
effectiveness. In addition to the heuristic method for poison-label BAIQA
(P-BAIQA), we explore the design of clean-label BAIQA (C-BAIQA), focusing on
$\alpha$ sampling and image data refinement, driven by theoretical insights we
reveal. Extensive experiments on diverse datasets and various NR-IQA models
demonstrate the effectiveness of our attacks. Code can be found at
https://github.com/yuyi-sd/BAIQA.


http://arxiv.org/abs/2412.07468
AHSG: Adversarial Attack on High-level Semantics in Graph Neural Networks. (99%)
Kai Yuan; Jiahao Zhang; Yidi Wang; Xiaobing Pei
  Adversarial attacks on Graph Neural Networks aim to perturb the performance
of the learner by carefully modifying the graph topology and node attributes.
Existing methods achieve attack stealthiness by constraining the modification
budget and differences in graph properties. However, these methods typically
disrupt task-relevant primary semantics directly, which results in low
defensibility and detectability of the attack. In this paper, we propose an
Adversarial Attack on High-level Semantics for Graph Neural Networks (AHSG),
which is a graph structure attack model that ensures the retention of primary
semantics. By combining latent representations with shared primary semantics,
our model retains detectable attributes and relational patterns of the original
graph while leveraging more subtle changes to carry out the attack. Then we use
the Projected Gradient Descent algorithm to map the latent representations with
attack effects to the adversarial graph. Through experiments on robust graph
deep learning models equipped with defense strategies, we demonstrate that AHSG
outperforms other state-of-the-art methods in attack effectiveness.
Additionally, using Contextual Stochastic Block Models to detect the attacked
graph further validates that our method preserves the primary semantics of the
graph.


http://arxiv.org/abs/2412.07575
Defending Against Neural Network Model Inversion Attacks via Data Poisoning. (98%)
Shuai Zhou; Dayong Ye; Tianqing Zhu; Wanlei Zhou
  Model inversion attacks pose a significant privacy threat to machine learning
models by reconstructing sensitive data from their outputs. While various
defenses have been proposed to counteract these attacks, they often come at the
cost of the classifier's utility, thus creating a challenging trade-off between
privacy protection and model utility. Moreover, most existing defenses require
retraining the classifier for enhanced robustness, which is impractical for
large-scale, well-established models. This paper introduces a novel defense
mechanism to better balance privacy and utility, particularly against
adversaries who employ a machine learning model (i.e., inversion model) to
reconstruct private data. Drawing inspiration from data poisoning attacks,
which can compromise the performance of machine learning models, we propose a
strategy that leverages data poisoning to contaminate the training data of
inversion models, thereby preventing model inversion attacks.
  Two defense methods are presented. The first, termed label-preserving
poisoning attacks for all output vectors (LPA), involves subtle perturbations
to all output vectors while preserving their labels. Our findings demonstrate
that these minor perturbations, introduced through a data poisoning approach,
significantly increase the difficulty of data reconstruction without
compromising the utility of the classifier. Subsequently, we introduce a second
method, label-flipping poisoning for partial output vectors (LFP), which
selectively perturbs a small subset of output vectors and alters their labels
during the process. Empirical results indicate that LPA is notably effective,
outperforming the current state-of-the-art defenses. Our data poisoning-based
defense provides a new retraining-free defense paradigm that preserves the
victim classifier's utility.


http://arxiv.org/abs/2412.08099
Adversarial Vulnerabilities in Large Language Models for Time Series Forecasting. (98%)
Fuqiang Liu; Sicong Jiang; Luis Miranda-Moreno; Seongjin Choi; Lijun Sun
  Large Language Models (LLMs) have recently demonstrated significant potential
in the field of time series forecasting, offering impressive capabilities in
handling complex temporal data. However, their robustness and reliability in
real-world applications remain under-explored, particularly concerning their
susceptibility to adversarial attacks. In this paper, we introduce a targeted
adversarial attack framework for LLM-based time series forecasting. By
employing both gradient-free and black-box optimization methods, we generate
minimal yet highly effective perturbations that significantly degrade the
forecasting accuracy across multiple datasets and LLM architectures. Our
experiments, which include models like TimeGPT and LLM-Time with GPT-3.5,
GPT-4, LLaMa, and Mistral, show that adversarial attacks lead to much more
severe performance degradation than random noise, and demonstrate the broad
effectiveness of our attacks across different LLMs. The results underscore the
critical vulnerabilities of LLMs in time series forecasting, highlighting the
need for robust defense mechanisms to ensure their reliable deployment in
practical applications.


http://arxiv.org/abs/2412.08098
What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models. (92%)
Bangshuo Zhu; Jiawen Wen; Huaming Chen
  Recent studies have demonstrated outstanding capabilities of large language
models (LLMs) in software engineering domain, covering numerous tasks such as
code generation and comprehension. While the benefit of LLMs for coding task is
well noted, it is perceived that LLMs are vulnerable to adversarial attacks. In
this paper, we study the specific LLM vulnerability to imperceptible character
attacks, a type of prompt-injection attack that uses special characters to
befuddle an LLM whilst keeping the attack hidden to human eyes. We devise four
categories of attacks and investigate their effects on the performance outcomes
of tasks relating to code analysis and code comprehension. Two generations of
ChatGPT are included to evaluate the impact of advancements made to
contemporary models. Our experimental design consisted of comparing perturbed
and unperturbed code snippets and evaluating two performance outcomes, which
are model confidence using log probabilities of response, and correctness of
response. We conclude that earlier version of ChatGPT exhibits a strong
negative linear correlation between the amount of perturbation and the
performance outcomes, while the recent ChatGPT presents a strong negative
correlation between the presence of perturbation and performance outcomes, but
no valid correlational relationship between perturbation budget and performance
outcomes. We anticipate this work contributes to an in-depth understanding of
leveraging LLMs for coding tasks. It is suggested future research should delve
into how to create LLMs that can return a correct response even if the prompt
exhibits perturbations.


http://arxiv.org/abs/2412.08053
DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time. (92%)
Jin Hu; Xianglong Liu; Jiakai Wang; Junkai Zhang; Xianqi Yang; Haotong Qin; Yuqing Ma; Ke Xu
  Physical adversarial examples (PAEs) are regarded as "whistle-blowers" of
real-world risks in deep-learning applications. However, current PAE generation
studies show limited adaptive attacking ability to diverse and varying scenes.
The key challenges in generating dynamic PAEs are exploring their patterns
under noisy gradient feedback and adapting the attack to agnostic scenario
natures. To address the problems, we present DynamicPAE, the first generative
framework that enables scene-aware real-time physical attacks beyond static
attacks. Specifically, to train the dynamic PAE generator under noisy gradient
feedback, we introduce the residual-driven sample trajectory guidance
technique, which redefines the training task to break the limited feedback
information restriction that leads to the degeneracy problem. Intuitively, it
allows the gradient feedback to be passed to the generator through a low-noise
auxiliary task, thereby guiding the optimization away from degenerate solutions
and facilitating a more comprehensive and stable exploration of feasible PAEs.
To adapt the generator to agnostic scenario natures, we introduce the
context-aligned scene expectation simulation process, consisting of the
conditional-uncertainty-aligned data module and the skewness-aligned objective
re-weighting module. The former enhances robustness in the context of
incomplete observation by employing a conditional probabilistic model for
domain randomization, while the latter facilitates consistent stealth control
across different attack targets by automatically reweighting losses based on
the skewness indicator. Extensive digital and physical evaluations demonstrate
the superior attack performance of DynamicPAE, attaining a 1.95 $\times$ boost
(65.55% average AP drop under attack) on representative object detectors (e.g.,
Yolo-v8) over state-of-the-art static PAE generating methods.


http://arxiv.org/abs/2412.07559
Adaptive Epsilon Adversarial Training for Robust Gravitational Wave Parameter Estimation Using Normalizing Flows. (86%)
Yiqian Yang; Xihua Zhu; Fan Zhang
  Adversarial training with Normalizing Flow (NF) models is an emerging
research area aimed at improving model robustness through adversarial samples.
In this study, we focus on applying adversarial training to NF models for
gravitational wave parameter estimation. We propose an adaptive epsilon method
for Fast Gradient Sign Method (FGSM) adversarial training, which dynamically
adjusts perturbation strengths based on gradient magnitudes using logarithmic
scaling. Our hybrid architecture, combining ResNet and Inverse Autoregressive
Flow, reduces the Negative Log Likelihood (NLL) loss by 47\% under FGSM attacks
compared to the baseline model, while maintaining an NLL of 4.2 on clean data
(only 5\% higher than the baseline). For perturbation strengths between 0.01
and 0.1, our model achieves an average NLL of 5.8, outperforming both
fixed-epsilon (NLL: 6.7) and progressive-epsilon (NLL: 7.2) methods. Under
stronger Projected Gradient Descent attacks with perturbation strength of 0.05,
our model maintains an NLL of 6.4, demonstrating superior robustness while
avoiding catastrophic overfitting.


http://arxiv.org/abs/2412.08014
MAGIC: Mastering Physical Adversarial Generation in Context through Collaborative LLM Agents. (82%)
Yun Xing; Nhat Chung; Jie Zhang; Yue Cao; Ivor Tsang; Yang Liu; Lei Ma; Qing Guo
  Physical adversarial attacks in driving scenarios can expose critical
vulnerabilities in visual perception models. However, developing such attacks
remains challenging due to diverse real-world environments and the requirement
for maintaining visual naturality. Building upon this challenge, we reformulate
physical adversarial attacks as a one-shot patch generation problem. Our
approach generates adversarial patches through a deep generative model that
considers the specific scene context, enabling direct physical deployment in
matching environments. The primary challenge lies in simultaneously achieving
two objectives: generating adversarial patches that effectively mislead object
detection systems while determining contextually appropriate deployment within
the scene. We propose MAGIC (Mastering Physical Adversarial Generation In
Context), a novel framework powered by multi-modal LLM agents to address these
challenges. MAGIC automatically understands scene context and generates
adversarial patch through the synergistic interaction of language and vision
capabilities. In particular, MAGIC orchestrates three specialized LLM agents:
The adv-patch generation agent (GAgent) masters the creation of deceptive
patches through strategic prompt engineering for text-to-image models. The
adv-patch deployment agent (DAgent) ensures contextual coherence by determining
optimal deployment strategies based on scene understanding. The
self-examination agent (EAgent) completes this trilogy by providing critical
oversight and iterative refinement of both processes. We validate our method on
both digital and physical levels, i.e., nuImage and manually captured
real-world scenes, where both statistical and visual results prove that our
MAGIC is powerful and effective for attacking widely applied object detection
systems, i.e., YOLO and DETR series.


http://arxiv.org/abs/2412.07672
FlexLLM: Exploring LLM Customization for Moving Target Defense on Black-Box LLMs Against Jailbreak Attacks. (81%)
Bocheng Chen; Hanqing Guo; Qiben Yan
  Defense in large language models (LLMs) is crucial to counter the numerous
attackers exploiting these systems to generate harmful content through
manipulated prompts, known as jailbreak attacks. Although many defense
strategies have been proposed, they often require access to the model's
internal structure or need additional training, which is impractical for
service providers using LLM APIs, such as OpenAI APIs or Claude APIs. In this
paper, we propose a moving target defense approach that alters decoding
hyperparameters to enhance model robustness against various jailbreak attacks.
Our approach does not require access to the model's internal structure and
incurs no additional training costs. The proposed defense includes two key
components: (1) optimizing the decoding strategy by identifying and adjusting
decoding hyperparameters that influence token generation probabilities, and (2)
transforming the decoding hyperparameters and model system prompts into dynamic
targets, which are continuously altered during each runtime. By continuously
modifying decoding strategies and prompts, the defense effectively mitigates
the existing attacks. Our results demonstrate that our defense is the most
effective against jailbreak attacks in three of the models tested when using
LLMs as black-box APIs. Moreover, our defense offers lower inference costs and
maintains comparable response quality, making it a potential layer of
protection when used alongside other defense methods.


http://arxiv.org/abs/2412.07511
Stealthy and Robust Backdoor Attack against 3D Point Clouds through Additional Point Features. (76%)
Xiaoyang Ning; Qing Xie; Jinyu Xu; Wenbo Jiang; Jiachen Li; Yanchun Ma
  Recently, 3D backdoor attacks have posed a substantial threat to 3D Deep
Neural Networks (3D DNNs) designed for 3D point clouds, which are extensively
deployed in various security-critical applications. Although the existing 3D
backdoor attacks achieved high attack performance, they remain vulnerable to
preprocessing-based defenses (e.g., outlier removal and rotation augmentation)
and are prone to detection by human inspection. In pursuit of a more
challenging-to-defend and stealthy 3D backdoor attack, this paper introduces
the Stealthy and Robust Backdoor Attack (SRBA), which ensures robustness and
stealthiness through intentional design considerations. The key insight of our
attack involves applying a uniform shift to the additional point features of
point clouds (e.g., reflection intensity) widely utilized as part of inputs for
3D DNNs as the trigger. Without altering the geometric information of the point
clouds, our attack ensures visual consistency between poisoned and benign
samples, and demonstrate robustness against preprocessing-based defenses. In
addition, to automate our attack, we employ Bayesian Optimization (BO) to
identify the suitable trigger. Extensive experiments suggest that SRBA achieves
an attack success rate (ASR) exceeding 94% in all cases, and significantly
outperforms previous SOTA methods when multiple preprocessing operations are
applied during training.


http://arxiv.org/abs/2412.07231
Adversarial Filtering Based Evasion and Backdoor Attacks to EEG-Based Brain-Computer Interfaces. (68%)
Lubin Meng; Xue Jiang; Xiaoqing Chen; Wenzhong Liu; Hanbin Luo; Dongrui Wu
  A brain-computer interface (BCI) enables direct communication between the
brain and an external device. Electroencephalogram (EEG) is a common input
signal for BCIs, due to its convenience and low cost. Most research on
EEG-based BCIs focuses on the accurate decoding of EEG signals, while ignoring
their security. Recent studies have shown that machine learning models in BCIs
are vulnerable to adversarial attacks. This paper proposes adversarial
filtering based evasion and backdoor attacks to EEG-based BCIs, which are very
easy to implement. Experiments on three datasets from different BCI paradigms
demonstrated the effectiveness of our proposed attack approaches. To our
knowledge, this is the first study on adversarial filtering for EEG-based BCIs,
raising a new security concern and calling for more attention on the security
of BCIs.


http://arxiv.org/abs/2412.07199
A Parametric Approach to Adversarial Augmentation for Cross-Domain Iris Presentation Attack Detection. (61%)
Debasmita Pal; Redwan Sony; Arun Ross
  Iris-based biometric systems are vulnerable to presentation attacks (PAs),
where adversaries present physical artifacts (e.g., printed iris images,
textured contact lenses) to defeat the system. This has led to the development
of various presentation attack detection (PAD) algorithms, which typically
perform well in intra-domain settings. However, they often struggle to
generalize effectively in cross-domain scenarios, where training and testing
employ different sensors, PA instruments, and datasets. In this work, we use
adversarial training samples of both bonafide irides and PAs to improve the
cross-domain performance of a PAD classifier. The novelty of our approach lies
in leveraging transformation parameters from classical data augmentation
schemes (e.g., translation, rotation) to generate adversarial samples. We
achieve this through a convolutional autoencoder, ADV-GEN, that inputs original
training samples along with a set of geometric and photometric transformations.
The transformation parameters act as regularization variables, guiding ADV-GEN
to generate adversarial samples in a constrained search space. Experiments
conducted on the LivDet-Iris 2017 database, comprising four datasets, and the
LivDet-Iris 2020 dataset, demonstrate the efficacy of our proposed method. The
code is available at https://github.com/iPRoBe-lab/ADV-GEN-IrisPAD.


http://arxiv.org/abs/2412.12145
Na'vi or Knave: Jailbreaking Language Models via Metaphorical Avatars. (50%)
Yu Yan; Sheng Sun; Junqi Tong; Min Liu; Qi Li
  Metaphor serves as an implicit approach to convey information, while enabling
the generalized comprehension of complex subjects. However, metaphor can
potentially be exploited to bypass the safety alignment mechanisms of Large
Language Models (LLMs), leading to the theft of harmful knowledge. In our
study, we introduce a novel attack framework that exploits the imaginative
capacity of LLMs to achieve jailbreaking, the J\underline{\textbf{A}}ilbreak
\underline{\textbf{V}}ia \underline{\textbf{A}}dversarial
Me\underline{\textbf{TA}} -pho\underline{\textbf{R}} (\textit{AVATAR}).
Specifically, to elicit the harmful response, AVATAR extracts harmful entities
from a given harmful target and maps them to innocuous adversarial entities
based on LLM's imagination. Then, according to these metaphors, the harmful
target is nested within human-like interaction for jailbreaking adaptively.
Experimental results demonstrate that AVATAR can effectively and transferablly
jailbreak LLMs and achieve a state-of-the-art attack success rate across
multiple advanced LLMs. Our study exposes a security risk in LLMs from their
endogenous imaginative capabilities. Furthermore, the analytical study reveals
the vulnerability of LLM to adversarial metaphors and the necessity of
developing defense methods against jailbreaking caused by the adversarial
metaphor. \textcolor{orange}{ \textbf{Warning: This paper contains potentially
harmful content from LLMs.}}


http://arxiv.org/abs/2412.07253
CapGen:An Environment-Adaptive Generator of Adversarial Patches. (13%)
Chaoqun Li; Zhuodong Liu; Huanqian Yan; Hang Su
  Adversarial patches, often used to provide physical stealth protection for
critical assets and assess perception algorithm robustness, usually neglect the
need for visual harmony with the background environment, making them easily
noticeable. Moreover, existing methods primarily concentrate on improving
attack performance, disregarding the intricate dynamics of adversarial patch
elements. In this work, we introduce the Camouflaged Adversarial Pattern
Generator (CAPGen), a novel approach that leverages specific base colors from
the surrounding environment to produce patches that seamlessly blend with their
background for superior visual stealthiness while maintaining robust
adversarial performance. We delve into the influence of both patterns (i.e.,
color-agnostic texture information) and colors on the effectiveness of attacks
facilitated by patches, discovering that patterns exert a more pronounced
effect on performance than colors. Based on these findings, we propose a rapid
generation strategy for adversarial patches. This involves updating the colors
of high-performance adversarial patches to align with those of the new
environment, ensuring visual stealthiness without compromising adversarial
impact. This paper is the first to comprehensively examine the roles played by
patterns and colors in the context of adversarial patches.


http://arxiv.org/abs/2412.07192
PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips. (10%)
Zachary Coalson; Jeonghyun Woo; Yu Sun; Shiyang Chen; Lishan Yang; Prashant Nair; Bo Fang; Sanghyun Hong
  We introduce a new class of attacks on commercial-scale (human-aligned)
language models that induce jailbreaking through targeted bitwise corruptions
in model parameters. Our adversary can jailbreak billion-parameter language
models with fewer than 25 bit-flips in all cases$-$and as few as 5 in
some$-$using up to 40$\times$ less bit-flips than existing attacks on computer
vision models at least 100$\times$ smaller. Unlike prompt-based jailbreaks, our
attack renders these models in memory 'uncensored' at runtime, allowing them to
generate harmful responses without any input modifications. Our attack
algorithm efficiently identifies target bits to flip, offering up to 20$\times$
more computational efficiency than previous methods. This makes it practical
for language models with billions of parameters. We show an end-to-end
exploitation of our attack using software-induced fault injection, Rowhammer
(RH). Our work examines 56 DRAM RH profiles from DDR4 and LPDDR4X devices with
different RH vulnerabilities. We show that our attack can reliably induce
jailbreaking in systems similar to those affected by prior bit-flip attacks.
Moreover, our approach remains effective even against highly RH-secure systems
(e.g., 46$\times$ more secure than previously tested systems). Our analyses
further reveal that: (1) models with less post-training alignment require fewer
bit flips to jailbreak; (2) certain model components, such as value projection
layers, are substantially more vulnerable than others; and (3) our method is
mechanistically different than existing jailbreaks. Our findings highlight a
pressing, practical threat to the language model ecosystem and underscore the
need for research to protect these models from bit-flip attacks.


http://arxiv.org/abs/2412.07249
Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generation. (2%)
Xin Zhao; Xiaojun Chen; Yuexin Xuan; Zhendong Zhao; Xiaojun Jia; Xinfeng Li; Xiaofeng Wang
  The rise of deep learning models in the digital era has raised substantial
concerns regarding the generation of Not-Safe-for-Work (NSFW) content. Existing
defense methods primarily involve model fine-tuning and post-hoc content
moderation. Nevertheless, these approaches largely lack scalability in
eliminating harmful content, degrade the quality of benign image generation, or
incur high inference costs. To address these challenges, we propose an
innovative framework named \textit{Buster}, which injects backdoors into the
text encoder to prevent NSFW content generation. Buster leverages deep semantic
information rather than explicit prompts as triggers, redirecting NSFW prompts
towards targeted benign prompts. Additionally, Buster employs energy-based
training data generation through Langevin dynamics for adversarial knowledge
augmentation, thereby ensuring robustness in harmful concept definition. This
approach demonstrates exceptional resilience and scalability in mitigating NSFW
content. Particularly, Buster fine-tunes the text encoder of Text-to-Image
models within merely five minutes, showcasing its efficiency. Our extensive
experiments denote that Buster outperforms nine state-of-the-art baselines,
achieving a superior NSFW content removal rate of at least 91.2\% while
preserving the quality of harmless images.


http://arxiv.org/abs/2412.06727
Take Fake as Real: Realistic-like Robust Black-box Adversarial Attack to Evade AIGC Detection. (99%)
Caiyun Xie; Dengpan Ye; Yunming Zhang; Long Tang; Yunna Lv; Jiacheng Deng; Jiawei Song
  The security of AI-generated content (AIGC) detection is crucial for ensuring
multimedia content credibility. To enhance detector security, research on
adversarial attacks has become essential. However, most existing adversarial
attacks focus only on GAN-generated facial images detection, struggle to be
effective on multi-class natural images and diffusion-based detectors, and
exhibit poor invisibility. To fill this gap, we first conduct an in-depth
analysis of the vulnerability of AIGC detectors and discover the feature that
detectors vary in vulnerability to different post-processing. Then, considering
that the detector is agnostic in real-world scenarios and given this discovery,
we propose a Realistic-like Robust Black-box Adversarial attack (R$^2$BA) with
post-processing fusion optimization. Unlike typical perturbations, R$^2$BA uses
real-world post-processing, i.e., Gaussian blur, JPEG compression, Gaussian
noise and light spot to generate adversarial examples. Specifically, we use a
stochastic particle swarm algorithm with inertia decay to optimize
post-processing fusion intensity and explore the detector's decision boundary.
Guided by the detector's fake probability, R$^2$BA enhances/weakens the
detector-vulnerable/detector-robust post-processing intensity to strike a
balance between adversariality and invisibility. Extensive experiments on
popular/commercial AIGC detectors and datasets demonstrate that R$^2$BA
exhibits impressive anti-detection performance, excellent invisibility, and
strong robustness in GAN-based and diffusion-based cases. Compared to
state-of-the-art white-box and black-box attacks, R$^2$BA shows significant
improvements of 15\%--72\% and 21\%--47\% in anti-detection performance under
the original and robust scenario respectively, offering valuable insights for
the security of AIGC detection in real-world applications.


http://arxiv.org/abs/2412.07078
Defensive Dual Masking for Robust Adversarial Defense. (99%)
Wangli Yang; Jie Yang; Yi Guo; Johan Barthelemy
  The field of textual adversarial defenses has gained considerable attention
in recent years due to the increasing vulnerability of natural language
processing (NLP) models to adversarial attacks, which exploit subtle
perturbations in input text to deceive models. This paper introduces the
Defensive Dual Masking (DDM) algorithm, a novel approach designed to enhance
model robustness against such attacks. DDM utilizes a unique adversarial
training strategy where [MASK] tokens are strategically inserted into training
samples to prepare the model to handle adversarial perturbations more
effectively. During inference, potentially adversarial tokens are dynamically
replaced with [MASK] tokens to neutralize potential threats while preserving
the core semantics of the input. The theoretical foundation of our approach is
explored, demonstrating how the selective masking mechanism strengthens the
model's ability to identify and mitigate adversarial manipulations. Our
empirical evaluation across a diverse set of benchmark datasets and attack
mechanisms consistently shows that DDM outperforms state-of-the-art defense
techniques, improving model accuracy and robustness. Moreover, when applied to
Large Language Models (LLMs), DDM also enhances their resilience to adversarial
attacks, providing a scalable defense mechanism for large-scale NLP
applications.


http://arxiv.org/abs/2412.06215
A Real-Time Defense Against Object Vanishing Adversarial Patch Attacks for Object Detection in Autonomous Vehicles. (97%)
Jaden Mu
  Autonomous vehicles (AVs) increasingly use DNN-based object detection models
in vision-based perception. Correct detection and classification of obstacles
is critical to ensure safe, trustworthy driving decisions. Adversarial patches
aim to fool a DNN with intentionally generated patterns concentrated in a
localized region of an image. In particular, object vanishing patch attacks can
cause object detection models to fail to detect most or all objects in a scene,
posing a significant practical threat to AVs.
  This work proposes ADAV (Adversarial Defense for Autonomous Vehicles), a
novel defense methodology against object vanishing patch attacks specifically
designed for autonomous vehicles. Unlike existing defense methods which have
high latency or are designed for static images, ADAV runs in real-time and
leverages contextual information from prior frames in an AV's video feed. ADAV
checks if the object detector's output for the target frame is temporally
consistent with the output from a previous reference frame to detect the
presence of a patch. If the presence of a patch is detected, ADAV uses
gradient-based attribution to localize adversarial pixels that break temporal
consistency. This two stage procedure allows ADAV to efficiently process clean
inputs, and both stages are optimized to be low latency. ADAV is evaluated
using real-world driving data from the Berkeley Deep Drive BDD100K dataset, and
demonstrates high adversarial and clean performance.


http://arxiv.org/abs/2412.06219
Data Free Backdoor Attacks. (64%)
Bochuan Cao; Jinyuan Jia; Chuxuan Hu; Wenbo Guo; Zhen Xiang; Jinghui Chen; Bo Li; Dawn Song
  Backdoor attacks aim to inject a backdoor into a classifier such that it
predicts any input with an attacker-chosen backdoor trigger as an
attacker-chosen target class. Existing backdoor attacks require either
retraining the classifier with some clean data or modifying the model's
architecture. As a result, they are 1) not applicable when clean data is
unavailable, 2) less efficient when the model is large, and 3) less stealthy
due to architecture changes. In this work, we propose DFBA, a novel
retraining-free and data-free backdoor attack without changing the model
architecture. Technically, our proposed method modifies a few parameters of a
classifier to inject a backdoor. Through theoretical analysis, we verify that
our injected backdoor is provably undetectable and unremovable by various
state-of-the-art defenses under mild assumptions. Our evaluation on multiple
datasets further demonstrates that our injected backdoor: 1) incurs negligible
classification loss, 2) achieves 100% attack success rates, and 3) bypasses six
existing state-of-the-art defenses. Moreover, our comparison with a
state-of-the-art non-data-free backdoor attack shows our attack is more
stealthy and effective against various defenses while achieving less
classification accuracy loss.


http://arxiv.org/abs/2412.07097
On Evaluating the Durability of Safeguards for Open-Weight LLMs. (38%)
Xiangyu Qi; Boyi Wei; Nicholas Carlini; Yangsibo Huang; Tinghao Xie; Luxi He; Matthew Jagielski; Milad Nasr; Prateek Mittal; Peter Henderson
  Stakeholders -- from model developers to policymakers -- seek to minimize the
dual-use risks of large language models (LLMs). An open challenge to this goal
is whether technical safeguards can impede the misuse of LLMs, even when models
are customizable via fine-tuning or when model weights are fully open. In
response, several recent studies have proposed methods to produce durable LLM
safeguards for open-weight LLMs that can withstand adversarial modifications of
the model's weights via fine-tuning. This holds the promise of raising
adversaries' costs even under strong threat models where adversaries can
directly fine-tune model weights. However, in this paper, we urge for more
careful characterization of the limits of these approaches. Through several
case studies, we demonstrate that even evaluating these defenses is exceedingly
difficult and can easily mislead audiences into thinking that safeguards are
more durable than they really are. We draw lessons from the evaluation pitfalls
that we identify and suggest future research carefully cabin claims to more
constrained, well-defined, and rigorously examined threat models, which can
provide more useful and candid assessments to stakeholders.


http://arxiv.org/abs/2412.06966
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice. (3%)
A. Feder Cooper; Christopher A. Choquette-Choo; Miranda Bogen; Matthew Jagielski; Katja Filippova; Ken Ziyu Liu; Alexandra Chouldechova; Jamie Hayes; Yangsibo Huang; Niloofar Mireshghallah; Ilia Shumailov; Eleni Triantafillou; Peter Kairouz; Nicole Mitchell; Percy Liang; Daniel E. Ho; Yejin Choi; Sanmi Koyejo; Fernando Delgado; James Grimmelmann; Vitaly Shmatikov; Sa Christopher De; Solon Barocas; Amy Cyphert; Mark Lemley; danah boyd; Jennifer Wortman Vaughan; Miles Brundage; David Bau; Seth Neel; Abigail Z. Jacobs; Andreas Terzis; Hanna Wallach; Nicolas Papernot; Katherine Lee
  We articulate fundamental mismatches between technical methods for machine
unlearning in Generative AI, and documented aspirations for broader impact that
these methods could have for law and policy. These aspirations are both
numerous and varied, motivated by issues that pertain to privacy, copyright,
safety, and more. For example, unlearning is often invoked as a solution for
removing the effects of targeted information from a generative-AI model's
parameters, e.g., a particular individual's personal data or in-copyright
expression of Spiderman that was included in the model's training data.
Unlearning is also proposed as a way to prevent a model from generating
targeted types of information in its outputs, e.g., generations that closely
resemble a particular individual's data or reflect the concept of "Spiderman."
Both of these goals--the targeted removal of information from a model and the
targeted suppression of information from a model's outputs--present various
technical and substantive challenges. We provide a framework for thinking
rigorously about these challenges, which enables us to be clear about why
unlearning is not a general-purpose solution for circumscribing generative-AI
model behavior in service of broader positive impact. We aim for conceptual
clarity and to encourage more thoughtful communication among machine learning
(ML), law, and policy experts who seek to develop and apply technical methods
for compliance with policy objectives.


http://arxiv.org/abs/2412.07129
StyleMark: A Robust Watermarking Method for Art Style Images Against Black-Box Arbitrary Style Transfer. (2%)
Yunming Zhang; Dengpan Ye; Sipeng Shen; Jun Wang
  Arbitrary Style Transfer (AST) achieves the rendering of real natural images
into the painting styles of arbitrary art style images, promoting art
communication. However, misuse of unauthorized art style images for AST may
infringe on artists' copyrights. One countermeasure is robust watermarking,
which tracks image propagation by embedding copyright watermarks into carriers.
Unfortunately, AST-generated images lose the structural and semantic
information of the original style image, hindering end-to-end robust tracking
by watermarks. To fill this gap, we propose StyleMark, the first robust
watermarking method for black-box AST, which can be seamlessly applied to art
style images achieving precise attribution of artistic styles after AST.
Specifically, we propose a new style watermark network that adjusts the mean
activations of style features through multi-scale watermark embedding, thereby
planting watermark traces into the shared style feature space of style images.
Furthermore, we design a distribution squeeze loss, which constrain content
statistical feature distortion, forcing the reconstruction network to focus on
integrating style features with watermarks, thus optimizing the intrinsic
watermark distribution. Finally, based on solid end-to-end training, StyleMark
mitigates the optimization conflict between robustness and watermark
invisibility through decoder fine-tuning under random noise. Experimental
results demonstrate that StyleMark exhibits significant robustness against
black-box AST and common pixel-level distortions, while also securely defending
against malicious adaptive attacks.


http://arxiv.org/abs/2412.07003
Understanding Gradient Descent through the Training Jacobian. (1%)
Nora Belrose; Adam Scherlis
  We examine the geometry of neural network training using the Jacobian of
trained network parameters with respect to their initial values. Our analysis
reveals low-dimensional structure in the training process which is dependent on
the input data but largely independent of the labels. We find that the singular
value spectrum of the Jacobian matrix consists of three distinctive regions: a
"chaotic" region of values orders of magnitude greater than one, a large "bulk"
region of values extremely close to one, and a "stable" region of values less
than one. Along each bulk direction, the left and right singular vectors are
nearly identical, indicating that perturbations to the initialization are
carried through training almost unchanged. These perturbations have virtually
no effect on the network's output in-distribution, yet do have an effect far
out-of-distribution. While the Jacobian applies only locally around a single
initialization, we find substantial overlap in bulk subspaces for different
random seeds.


http://arxiv.org/abs/2412.06556
Vulnerability, Where Art Thou? An Investigation of Vulnerability Management in Android Smartphone Chipsets. (1%)
Daniel Klischies; Philipp Mackensen; Veelasha Moonsamy
  Vulnerabilities in Android smartphone chipsets have severe consequences, as
recent real-world attacks have demonstrated that adversaries can leverage
vulnerabilities to execute arbitrary code or exfiltrate confidential
information. Despite the far-reaching impact of such attacks, the lifecycle of
chipset vulnerabilities has yet to be investigated, with existing papers
primarily investigating vulnerabilities in the Android operating system. This
paper provides a comprehensive and empirical study of the current state of
smartphone chipset vulnerability management within the Android ecosystem. For
the first time, we create a unified knowledge base of 3,676 chipset
vulnerabilities affecting 437 chipset models from all four major chipset
manufacturers, combined with 6,866 smartphone models. Our analysis revealed
that the same vulnerabilities are often included in multiple generations of
chipsets, providing novel empirical evidence that vulnerabilities are inherited
through multiple chipset generations. Furthermore, we demonstrate that the
commonly accepted 90-day responsible vulnerability disclosure period is seldom
adhered to. We find that a single vulnerability often affects hundreds to
thousands of different smartphone models, for which update availability is, as
we show, often unclear or heavily delayed. Leveraging the new insights gained
from our empirical analysis, we recommend several changes that chipset
manufacturers can implement to improve the security posture of their products.
At the same time, our knowledge base enables academic researchers to conduct
more representative evaluations of smartphone chipsets, accurately assess the
impact of vulnerabilities they discover, and identify avenues for future
research.


http://arxiv.org/abs/2412.05943
Adversarial Transferability in Deep Denoising Models: Theoretical Insights and Robustness Enhancement via Out-of-Distribution Typical Set Sampling. (99%)
Jie Ning; Jiebao Sun; Shengzhu Shi; Zhichang Guo; Yao Li; Hongwei Li; Boying Wu
  Deep learning-based image denoising models demonstrate remarkable
performance, but their lack of robustness analysis remains a significant
concern. A major issue is that these models are susceptible to adversarial
attacks, where small, carefully crafted perturbations to input data can cause
them to fail. Surprisingly, perturbations specifically crafted for one model
can easily transfer across various models, including CNNs, Transformers,
unfolding models, and plug-and-play models, leading to failures in those models
as well. Such high adversarial transferability is not observed in
classification models. We analyze the possible underlying reasons behind the
high adversarial transferability through a series of hypotheses and validation
experiments. By characterizing the manifolds of Gaussian noise and adversarial
perturbations using the concept of typical set and the asymptotic equipartition
property, we prove that adversarial samples deviate slightly from the typical
set of the original input distribution, causing the models to fail. Based on
these insights, we propose a novel adversarial defense method: the
Out-of-Distribution Typical Set Sampling Training strategy (TS). TS not only
significantly enhances the model's robustness but also marginally improves
denoising performance compared to the original model.


http://arxiv.org/abs/2412.06149
An Effective and Resilient Backdoor Attack Framework against Deep Neural Networks and Vision Transformers. (89%)
Xueluan Gong; Bowei Tian; Meng Xue; Yuan Wu; Yanjiao Chen; Qian Wang
  Recent studies have revealed the vulnerability of Deep Neural Network (DNN)
models to backdoor attacks. However, existing backdoor attacks arbitrarily set
the trigger mask or use a randomly selected trigger, which restricts the
effectiveness and robustness of the generated backdoor triggers. In this paper,
we propose a novel attention-based mask generation methodology that searches
for the optimal trigger shape and location. We also introduce a
Quality-of-Experience (QoE) term into the loss function and carefully adjust
the transparency value of the trigger in order to make the backdoored samples
to be more natural. To further improve the prediction accuracy of the victim
model, we propose an alternating retraining algorithm in the backdoor injection
process. The victim model is retrained with mixed poisoned datasets in even
iterations and with only benign samples in odd iterations. Besides, we launch
the backdoor attack under a co-optimized attack framework that alternately
optimizes the backdoor trigger and backdoored model to further improve the
attack performance. Apart from DNN models, we also extend our proposed attack
method against vision transformers. We evaluate our proposed method with
extensive experiments on VGG-Flower, CIFAR-10, GTSRB, CIFAR-100, and ImageNette
datasets. It is shown that we can increase the attack success rate by as much
as 82\% over baselines when the poison ratio is low and achieve a high QoE of
the backdoored samples. Our proposed backdoor attack framework also showcases
robustness against state-of-the-art backdoor defenses.


http://arxiv.org/abs/2412.05980
Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation. (22%)
Yiren Song; Shengtao Lou; Xiaokang Liu; Hai Ci; Pei Yang; Jiaming Liu; Mike Zheng Shou
  Diffusion models have revolutionized generative modeling with their
exceptional ability to produce high-fidelity images. However, misuse of such
potent tools can lead to the creation of fake news or disturbing content
targeting individuals, resulting in significant social harm. In this paper, we
introduce Anti-Reference, a novel method that protects images from the threats
posed by reference-based generation techniques by adding imperceptible
adversarial noise to the images. We propose a unified loss function that
enables joint attacks on fine-tuning-based customization methods,
non-fine-tuning customization methods, and human-centric driving methods. Based
on this loss, we train a Adversarial Noise Encoder to predict the noise or
directly optimize the noise using the PGD method. Our method shows certain
transfer attack capabilities, effectively challenging both gray-box models and
some commercial APIs. Extensive experiments validate the performance of
Anti-Reference, establishing a new benchmark in image security.


http://arxiv.org/abs/2412.05892
PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximization. (15%)
Ruoxi Cheng; Yizhong Ding; Shuirong Cao; Ranjie Duan; Xiaoshuang Jia; Shaowei Yuan; Zhiqiang Wang; Xiaojun Jia
  Understanding the vulnerabilities of Large Vision Language Models (LVLMs) to
jailbreak attacks is essential for their responsible real-world deployment.
Most previous work requires access to model gradients, or is based on human
knowledge (prompt engineering) to complete jailbreak, and they hardly consider
the interaction of images and text, resulting in inability to jailbreak in
black box scenarios or poor performance. To overcome these limitations, we
propose a Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for
toxicity maximization, referred to as PBI-Attack. Our method begins by
extracting malicious features from a harmful corpus using an alternative LVLM
and embedding these features into a benign image as prior information.
Subsequently, we enhance these features through bidirectional cross-modal
interaction optimization, which iteratively optimizes the bimodal perturbations
in an alternating manner through greedy search, aiming to maximize the toxicity
of the generated response. The toxicity level is quantified using a
well-trained evaluation model. Experiments demonstrate that PBI-Attack
outperforms previous state-of-the-art jailbreak methods, achieving an average
attack success rate of 92.5% across three open-source LVLMs and around 67.3% on
three closed-source LVLMs. Disclaimer: This paper contains potentially
disturbing and offensive content.


http://arxiv.org/abs/2412.05829
SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation. (4%)
Naizhu Jin; Zhong Li; Yinggang Guo; Chao Su; Tian Zhang; Qingkai Zeng
  Recent studies have proposed integrating Chain-of-Thought (CoT) reasoning to
further enhance the reliability of Code Language Models (CLMs) in generating
code, a step-by-step approach that breaks down complex programming tasks into
manageable sub-problems. Advances in this area have introduced CoT models,
specifically designed to integrate CoT reasoning effectively into language
models, achieving notable improvements in code generation. Despite these
advancements, the security of CoT models has not been systematically studied.
In this study, we aim to fill this gap by investigating the vulnerability of
CoT models to backdoor injection in code generation tasks. To address this, we
propose a model-agnostic backdoor attack method SABER
(\textbf{S}elf-\textbf{A}ttention-\textbf{B}as\textbf{E}d backdoo\textbf{R})
based on the self-attention mechanism. SABER begins by selecting a malicious
output as the backdoor using code mutation operations. It then identifies
tokens most relevant to poisoned content by analyzing self-attention scores in
the CodeBERT model. Finally, it applies semantic-preserving perturbations to
generate adaptive and natural triggers. Our experiments on HumanEval-CoT and
OpenEval-CoT test sets demonstrate that CoT models are susceptible to backdoor
attacks via data poisoning. Taking the OpenEval-CoT dataset as an example,
SABER achieves an ASR of 76.19%, representing an improvement of 14.29% over
RIPPLe and a substantial 23.08% enhancement compared to BadPre. Further
evaluations using ONION for automated detection and human studies reveal that
SABER is stealthier and harder to detect, bypassing 77.27% of automated
detection, with a human detection rate of just 3.17%. Our findings reveal that
backdoors can be injected into CoT models to manipulate downstream code
generation tasks.


http://arxiv.org/abs/2412.06181
Enhancing Adversarial Resistance in LLMs with Recursion. (1%)
Bryan Li; Sounak Bagchi; Zizhan Wang
  The increasing integration of Large Language Models (LLMs) into society
necessitates robust defenses against vulnerabilities from jailbreaking and
adversarial prompts. This project proposes a recursive framework for enhancing
the resistance of LLMs to manipulation through the use of prompt simplification
techniques. By increasing the transparency of complex and confusing adversarial
prompts, the proposed method enables more reliable detection and prevention of
malicious inputs. Our findings attempt to address a critical problem in AI
safety and security, providing a foundation for the development of systems able
to distinguish harmless inputs from prompts containing malicious intent. As
LLMs continue to be used in diverse applications, the importance of such
safeguards will only grow.


http://arxiv.org/abs/2412.06157
Membership Inference Attacks and Defenses in Federated Learning: A Survey. (1%)
Li Bai; Haibo Hu; Qingqing Ye; Haoyang Li; Leixia Wang; Jianliang Xu
  Federated learning is a decentralized machine learning approach where clients
train models locally and share model updates to develop a global model. This
enables low-resource devices to collaboratively build a high-quality model
without requiring direct access to the raw training data. However, despite only
sharing model updates, federated learning still faces several privacy
vulnerabilities. One of the key threats is membership inference attacks, which
target clients' privacy by determining whether a specific example is part of
the training set. These attacks can compromise sensitive information in
real-world applications, such as medical diagnoses within a healthcare system.
Although there has been extensive research on membership inference attacks, a
comprehensive and up-to-date survey specifically focused on it within federated
learning is still absent. To fill this gap, we categorize and summarize
membership inference attacks and their corresponding defense strategies based
on their characteristics in this setting. We introduce a unique taxonomy of
existing attack research and provide a systematic overview of various
countermeasures. For these studies, we thoroughly analyze the strengths and
weaknesses of different approaches. Finally, we identify and discuss key future
research directions for readers interested in advancing the field.


http://arxiv.org/abs/2412.05767
DeMem: Privacy-Enhanced Robust Adversarial Learning via De-Memorization. (76%)
Xiaoyu Luo; Qiongxiu Li
  Adversarial robustness, the ability of a model to withstand manipulated
inputs that cause errors, is essential for ensuring the trustworthiness of
machine learning models in real-world applications. However, previous studies
have shown that enhancing adversarial robustness through adversarial training
increases vulnerability to privacy attacks. While differential privacy can
mitigate these attacks, it often compromises robustness against both natural
and adversarial samples. Our analysis reveals that differential privacy
disproportionately impacts low-risk samples, causing an unintended performance
drop. To address this, we propose DeMem, which selectively targets high-risk
samples, achieving a better balance between privacy protection and model
robustness. DeMem is versatile and can be seamlessly integrated into various
adversarial training techniques. Extensive evaluations across multiple training
methods and datasets demonstrate that DeMem significantly reduces privacy
leakage while maintaining robustness against both natural and adversarial
samples. These results confirm DeMem's effectiveness and broad applicability in
enhancing privacy without compromising robustness.


http://arxiv.org/abs/2412.05592
From Flexibility to Manipulation: The Slippery Slope of XAI Evaluation. (47%)
Kristoffer Wickstrøm; Marina Marie-Claire Höhne; Anna Hedström
  The lack of ground truth explanation labels is a fundamental challenge for
quantitative evaluation in explainable artificial intelligence (XAI). This
challenge becomes especially problematic when evaluation methods have numerous
hyperparameters that must be specified by the user, as there is no ground truth
to determine an optimal hyperparameter selection. It is typically not feasible
to do an exhaustive search of hyperparameters so researchers typically make a
normative choice based on similar studies in the literature, which provides
great flexibility for the user. In this work, we illustrate how this
flexibility can be exploited to manipulate the evaluation outcome. We frame
this manipulation as an adversarial attack on the evaluation where seemingly
innocent changes in hyperparameter setting significantly influence the
evaluation outcome. We demonstrate the effectiveness of our manipulation across
several datasets with large changes in evaluation outcomes across several
explanation methods and models. Lastly, we propose a mitigation strategy based
on ranking across hyperparameters that aims to provide robustness towards such
manipulation. This work highlights the difficulty of conducting reliable XAI
evaluation and emphasizes the importance of a holistic and transparent approach
to evaluation in XAI.


http://arxiv.org/abs/2412.05676
Nearly Solved? Robust Deepfake Detection Requires More than Visual Forensics. (33%)
Guy Levy; Nathan Liebmann
  Deepfakes are on the rise, with increased sophistication and prevalence
allowing for high-profile social engineering attacks. Detecting them in the
wild is therefore important as ever, giving rise to new approaches breaking
benchmark records in this task. In line with previous work, we show that
recently developed state-of-the-art detectors are susceptible to classical
adversarial attacks, even in a highly-realistic black-box setting, putting
their usability in question. We argue that crucial 'robust features' of
deepfakes are in their higher semantics, and follow that with evidence that a
detector based on a semantic embedding model is less susceptible to black-box
perturbation attacks. We show that large visuo-lingual models like GPT-4o can
perform zero-shot deepfake detection better than current state-of-the-art
methods, and introduce a novel attack based on high-level semantic
manipulation. Finally, we argue that hybridising low- and high-level detectors
can improve adversarial robustness, based on their complementary strengths and
weaknesses.


http://arxiv.org/abs/2412.05232
LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds. (15%)
James Beetham; Souradip Chakraborty; Mengdi Wang; Furong Huang; Amrit Singh Bedi; Mubarak Shah
  Many existing jailbreak techniques rely on solving discrete combinatorial
optimization, while more recent approaches involve training LLMs to generate
multiple adversarial prompts. However, both approaches require significant
computational resources to produce even a single adversarial prompt. We
hypothesize that the inefficiency of current approaches stems from an
inadequate characterization of the jailbreak problem. To address this gap, we
formulate the jailbreak problem in terms of alignment. By starting from an
available safety-aligned model, we leverage an unsafe reward to guide the safe
model towards generating unsafe outputs using alignment techniques (e.g.,
reinforcement learning from human feedback), effectively performing
jailbreaking via alignment. We propose a novel jailbreak method called LIAR
(LeveragIng Alignment to jailbReak). To demonstrate the simplicity and
effectiveness of our approach, we employ a best-of-N method to solve the
alignment problem. LIAR offers significant advantages: lower computational
requirements without additional training, fully black-box operation,
competitive attack success rates, and more human-readable prompts. We provide
theoretical insights into the possibility of jailbreaking a safety-aligned
model, revealing inherent vulnerabilities in current alignment strategies for
LLMs. We also provide sub-optimality guarantees for the proposed \algo.
Experimentally, we achieve ASR comparable to the SoTA with a 10x improvement to
perplexity and a Time-to-Attack measured in seconds rather than tens of hours.


http://arxiv.org/abs/2412.05538
Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models. (8%)
Hao Cheng; Erjia Xiao; Jiayan Yang; Jiahang Cao; Qiang Zhang; Jize Zhang; Kaidi Xu; Jindong Gu; Renjing Xu
  Current image generation models can effortlessly produce high-quality, highly
realistic images, but this also increases the risk of misuse. In various
Text-to-Image or Image-to-Image tasks, attackers can generate a series of
images containing inappropriate content by simply editing the language modality
input. To mitigate this security concern, numerous guarding or defensive
strategies have been proposed, with a particular emphasis on safeguarding
language modality. However, in practical applications, threats in the vision
modality, particularly in tasks involving the editing of real-world images,
present heightened security risks as they can easily infringe upon the rights
of the image owner. Therefore, this paper employs a method named typographic
attack to reveal that various image generation models are also susceptible to
threats within the vision modality. Furthermore, we also evaluate the defense
performance of various existing methods when facing threats in the vision
modality and uncover their ineffectiveness. Finally, we propose the Vision
Modal Threats in Image Generation Models (VMT-IGMs) dataset, which would serve
as a baseline for evaluating the vision modality vulnerability of various image
generation models.


http://arxiv.org/abs/2412.05351
Towards Predicting the Success of Transfer-based Attacks by Quantifying Shared Feature Representations. (2%)
Ashley S. Dale; Mei Qiu; Foo Bin Che; Thomas Bsaibes; Lauren Christopher; Paul Salama
  Much effort has been made to explain and improve the success of
transfer-based attacks (TBA) on black-box computer vision models. This work
provides the first attempt at a priori prediction of attack success by
identifying the presence of vulnerable features within target models. Recent
work by Chen and Liu (2024) proposed the manifold attack model, a unifying
framework proposing that successful TBA exist in a common manifold space. Our
work experimentally tests the common manifold space hypothesis by a new
methodology: first, projecting feature vectors from surrogate and target
feature extractors trained on ImageNet onto the same low-dimensional manifold;
second, quantifying any observed structure similarities on the manifold; and
finally, by relating these observed similarities to the success of the TBA. We
find that shared feature representation moderately correlates with increased
success of TBA (\r{ho}= 0.56). This method may be used to predict whether an
attack will transfer without information of the model weights, training,
architecture or details of the attack. The results confirm the presence of
shared feature representations between two feature extractors of different
sizes and complexities, and demonstrate the utility of datasets from different
target domains as test signals for interpreting black-box feature
representations.


http://arxiv.org/abs/2412.05010
Backdooring Outlier Detection Methods: A Novel Attack Approach. (2%)
ZeinabSadat Taghavi; Hossein Mirzaei
  There have been several efforts in backdoor attacks, but these have primarily
focused on the closed-set performance of classifiers (i.e., classification).
This has left a gap in addressing the threat to classifiers' open-set
performance, referred to as outlier detection in the literature. Reliable
outlier detection is crucial for deploying classifiers in critical real-world
applications such as autonomous driving and medical image analysis. First, we
show that existing backdoor attacks fall short in affecting the open-set
performance of classifiers, as they have been specifically designed to confuse
intra-closed-set decision boundaries. In contrast, an effective backdoor attack
for outlier detection needs to confuse the decision boundary between the closed
and open sets. Motivated by this, in this study, we propose BATOD, a novel
Backdoor Attack targeting the Outlier Detection task. Specifically, we design
two categories of triggers to shift inlier samples to outliers and vice versa.
We evaluate BATOD using various real-world datasets and demonstrate its
superior ability to degrade the open-set performance of classifiers compared to
previous attacks, both before and after applying defenses.


http://arxiv.org/abs/2412.04245
Intriguing Properties of Robust Classification. (96%)
Bernd Prach; Christoph H. Lampert
  Despite extensive research since the community learned about adversarial
examples 10 years ago, we still do not know how to train high-accuracy
classifiers that are guaranteed to be robust to small perturbations of their
inputs. Previous works often argued that this might be because no classifier
exists that is robust and accurate at the same time. However, in computer
vision this assumption does not match reality where humans are usually accurate
and robust on most tasks of interest. We offer an alternative explanation and
show that in certain settings robust generalization is only possible with
unrealistically large amounts of data. More precisely we find a setting where a
robust classifier exists, it is easy to learn an accurate classifier, yet it
requires an exponential amount of data to learn a robust classifier. Based on
this theoretical result, we explore how well robust classifiers generalize on
datasets such as CIFAR-10. We come to the conclusion that on this datasets, the
limitation of current robust models also lies in the generalization, and that
they require a lot of data to do well on the test set. We also show that the
problem is not in the expressiveness or generalization capabilities of current
architectures, and that there are low magnitude features in the data which are
useful for non-robust generalization but are not available for robust
classifiers.


http://arxiv.org/abs/2412.04163
On the Lack of Robustness of Binary Function Similarity Systems. (92%)
Gianluca Capozzi; Tong Tang; Jie Wan; Ziqi Yang; Daniele Cono D'Elia; Luna Giuseppe Antonio Di; Lorenzo Cavallaro; Leonardo Querzoni
  Binary function similarity, which often relies on learning-based algorithms
to identify what functions in a pool are most similar to a given query
function, is a sought-after topic in different communities, including machine
learning, software engineering, and security. Its importance stems from the
impact it has in facilitating several crucial tasks, from reverse engineering
and malware analysis to automated vulnerability detection. Whereas recent work
cast light around performance on this long-studied problem, the research
landscape remains largely lackluster in understanding the resiliency of the
state-of-the-art machine learning models against adversarial attacks. As
security requires to reason about adversaries, in this work we assess the
robustness of such models through a simple yet effective black-box greedy
attack, which modifies the topology and the content of the control flow of the
attacked functions. We demonstrate that this attack is successful in
compromising all the models, achieving average attack success rates of 57.06%
and 95.81% depending on the problem settings (targeted and untargeted attacks).
Our findings are insightful: top performance on clean data does not necessarily
relate to top robustness properties, which explicitly highlights
performance-robustness trade-offs one should consider when deploying such
models, calling for further research.


http://arxiv.org/abs/2412.04776
Megatron: Evasive Clean-Label Backdoor Attacks against Vision Transformer. (76%)
Xueluan Gong; Bowei Tian; Meng Xue; Shuike Li; Yanjiao Chen; Qian Wang
  Vision transformers have achieved impressive performance in various
vision-related tasks, but their vulnerability to backdoor attacks is
under-explored. A handful of existing works focus on dirty-label attacks with
wrongly-labeled poisoned training samples, which may fail if a benign model
trainer corrects the labels. In this paper, we propose Megatron, an evasive
clean-label backdoor attack against vision transformers, where the attacker
injects the backdoor without manipulating the data-labeling process. To
generate an effective trigger, we customize two loss terms based on the
attention mechanism used in transformer networks, i.e., latent loss and
attention diffusion loss. The latent loss aligns the last attention layer
between triggered samples and clean samples of the target label. The attention
diffusion loss emphasizes the attention diffusion area that encompasses the
trigger. A theoretical analysis is provided to underpin the rationale behind
the attention diffusion loss. Extensive experiments on CIFAR-10, GTSRB,
CIFAR-100, and Tiny ImageNet demonstrate the effectiveness of Megatron.
Megatron can achieve attack success rates of over 90% even when the position of
the trigger is slightly shifted during testing. Furthermore, Megatron achieves
better evasiveness than baselines regarding both human visual inspection and
defense strategies (i.e., DBAVT, BAVT, Beatrix, TeCo, and SAGE).


http://arxiv.org/abs/2412.03908
Can Targeted Clean-Label Poisoning Attacks Generalize? (13%)
Zhizhen Chen; Subrat Kishore Dutta; Zhengyu Zhao; Chenhao Lin; Chao Shen; Xiao Zhang
  Targeted poisoning attacks aim to compromise the model's prediction on
specific target samples. In a common clean-label setting, they are achieved by
slightly perturbing a subset of training samples given access to those specific
targets. Despite continuous efforts, it remains unexplored whether such attacks
can generalize to unknown variations of those targets. In this paper, we take
the first step to systematically study this generalization problem. Observing
that the widely adopted, cosine similarity-based attack exhibits limited
generalizability, we propose a well-generalizable attack that leverages both
the direction and magnitude of model gradients. In particular, we explore
diverse target variations, such as an object with varied viewpoints and an
animal species with distinct appearances. Extensive experiments across various
generalization scenarios demonstrate that our method consistently achieves the
best attack effectiveness. For example, our method outperforms the cosine
similarity-based attack by 20.95% in attack success rate with similar overall
accuracy, averaged over four models on two image benchmark datasets. The code
is available at https://github.com/jiaangk/generalizable_tcpa


http://arxiv.org/abs/2412.03993
LaserGuider: A Laser Based Physical Backdoor Attack against Deep Neural Networks. (8%)
Yongjie Xu; Guangke Chen; Fu Song; Yuqi Chen
  Backdoor attacks embed hidden associations between triggers and targets in
deep neural networks (DNNs), causing them to predict the target when a trigger
is present while maintaining normal behavior otherwise. Physical backdoor
attacks, which use physical objects as triggers, are feasible but lack remote
control, temporal stealthiness, flexibility, and mobility. To overcome these
limitations, in this work, we propose a new type of backdoor triggers utilizing
lasers that feature long-distance transmission and instant-imaging properties.
Based on the laser-based backdoor triggers, we present a physical backdoor
attack, called LaserGuider, which possesses remote control ability and achieves
high temporal stealthiness, flexibility, and mobility. We also introduce a
systematic approach to optimize laser parameters for improving attack
effectiveness. Our evaluation on traffic sign recognition DNNs, critical in
autonomous vehicles, demonstrates that LaserGuider with three different
laser-based triggers achieves over 90% attack success rate with negligible
impact on normal inputs. Additionally, we release LaserMark, the first dataset
of real world traffic signs stamped with physical laser spots, to support
further research in backdoor attacks and defenses.


http://arxiv.org/abs/2412.03876
Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimization. (3%)
Jiangweizhi Peng; Zhiwei Tang; Gaowen Liu; Charles Fleming; Mingyi Hong
  Text-to-Image (T2I) diffusion models are widely recognized for their ability
to generate high-quality and diverse images based on text prompts. However,
despite recent advances, these models are still prone to generating unsafe
images containing sensitive or inappropriate content, which can be harmful to
users. Current efforts to prevent inappropriate image generation for diffusion
models are easy to bypass and vulnerable to adversarial attacks. How to ensure
that T2I models align with specific safety goals remains a significant
challenge. In this work, we propose a novel, training-free approach, called
Prompt-Noise Optimization (PNO), to mitigate unsafe image generation. Our
method introduces a novel optimization framework that leverages both the
continuous prompt embedding and the injected noise trajectory in the sampling
process to generate safe images. Extensive numerical results demonstrate that
our framework achieves state-of-the-art performance in suppressing toxic image
generations and demonstrates robustness to adversarial attacks, without needing
to tune the model parameters. Furthermore, compared with existing methods, PNO
uses comparable generation time while offering the best tradeoff between the
conflicting goals of safe generation and prompt-image alignment.


http://arxiv.org/abs/2412.04415
Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation. (2%)
Xuying Li; Zhuo Li; Yuji Kosuga; Yasuhiro Yoshida; Victor Bian
  AI agents, powered by large language models (LLMs), have transformed
human-computer interactions by enabling seamless, natural, and context-aware
communication. While these advancements offer immense utility, they also
inherit and amplify inherent safety risks such as bias, fairness,
hallucinations, privacy breaches, and a lack of transparency. This paper
investigates a critical vulnerability: adversarial attacks targeting the LLM
core within AI agents. Specifically, we test the hypothesis that a deceptively
simple adversarial prefix, such as \textit{Ignore the document}, can compel
LLMs to produce dangerous or unintended outputs by bypassing their contextual
safeguards. Through experimentation, we demonstrate a high attack success rate
(ASR), revealing the fragility of existing LLM defenses. These findings
emphasize the urgent need for robust, multi-layered security measures tailored
to mitigate vulnerabilities at the LLM level and within broader agent-based
architectures.


http://arxiv.org/abs/2412.03539
NODE-AdvGAN: Improving the transferability and perceptual similarity of adversarial examples by dynamic-system-driven adversarial generative model. (99%)
Xinheng Xie; Yue Wu; Cuiyu He
  Understanding adversarial examples is crucial for improving model robustness,
as they introduce imperceptible perturbations to deceive models. Effective
adversarial examples, therefore, offer the potential to train more robust
models by eliminating model singularities. We propose NODE-AdvGAN, a novel
approach that treats adversarial generation as a continuous process and employs
a Neural Ordinary Differential Equation (NODE) to simulate generator dynamics.
By mimicking the iterative nature of traditional gradient-based methods,
NODE-AdvGAN generates smoother and more precise perturbations that preserve
high perceptual similarity when added to benign images. We also propose a new
training strategy, NODE-AdvGAN-T, which enhances transferability in black-box
attacks by tuning the noise parameters during training. Experiments demonstrate
that NODE-AdvGAN and NODE-AdvGAN-T generate more effective adversarial examples
that achieve higher attack success rates while preserving better perceptual
quality than baseline models.


http://arxiv.org/abs/2412.03235
Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (99%)
Sravanti Addepalli; Yerram Varun; Arun Suggala; Karthikeyan Shanmugam; Prateek Jain
  Large Language Models (LLMs) are known to be susceptible to crafted
adversarial attacks or jailbreaks that lead to the generation of objectionable
content despite being aligned to human preferences using safety fine-tuning
methods. While the large dimensionality of input token space makes it
inevitable to find adversarial prompts that can jailbreak these models, we aim
to evaluate whether safety fine-tuned LLMs are safe against natural prompts
which are semantically related to toxic seed prompts that elicit safe responses
after alignment. We surprisingly find that popular aligned LLMs such as GPT-4
can be compromised using naive prompts that are NOT even crafted with an
objective of jailbreaking the model. Furthermore, we empirically show that
given a seed prompt that elicits a toxic response from an unaligned model, one
can systematically generate several semantically related natural prompts that
can jailbreak aligned LLMs. Towards this, we propose a method of Response
Guided Question Augmentation (ReG-QA) to evaluate the generalization of safety
aligned LLMs to natural prompts, that first generates several toxic answers
given a seed question using an unaligned LLM (Q to A), and further leverages an
LLM to generate questions that are likely to produce these answers (A to Q). We
interestingly find that safety fine-tuned LLMs such as GPT-4o are vulnerable to
producing natural jailbreak questions from unsafe content (without denial) and
can thus be used for the latter (A to Q) step. We obtain attack success rates
that are comparable to/ better than leading adversarial attack methods on the
JailbreakBench leaderboard, while being significantly more stable against
defenses such as Smooth-LLM and Synonym Substitution, which are effective
against existing all attacks on the leaderboard.


http://arxiv.org/abs/2412.03051
Less is More: A Stealthy and Efficient Adversarial Attack Method for DRL-based Autonomous Driving Policies. (98%)
Junchao Fan; Xuyang Lei; Xiaolin Chang; Jelena Mišić; Vojislav B. Mišić
  Despite significant advancements in deep reinforcement learning (DRL)-based
autonomous driving policies, these policies still exhibit vulnerability to
adversarial attacks. This vulnerability poses a formidable challenge to the
practical deployment of these policies in autonomous driving. Designing
effective adversarial attacks is an indispensable prerequisite for enhancing
the robustness of these policies. In view of this, we present a novel stealthy
and efficient adversarial attack method for DRL-based autonomous driving
policies. Specifically, we introduce a DRL-based adversary designed to trigger
safety violations (e.g., collisions) by injecting adversarial samples at
critical moments. We model the attack as a mixed-integer optimization problem
and formulate it as a Markov decision process. Then, we train the adversary to
learn the optimal policy for attacking at critical moments without domain
knowledge. Furthermore, we introduce attack-related information and a
trajectory clipping method to enhance the learning capability of the adversary.
Finally, we validate our method in an unprotected left-turn scenario across
different traffic densities. The experimental results show that our method
achieves more than 90% collision rate within three attacks in most cases.
Furthermore, our method achieves more than 130% improvement in attack
efficiency compared to the unlimited attack method.


http://arxiv.org/abs/2412.03441
PBP: Post-training Backdoor Purification for Malware Classifiers. (76%)
Dung Thuy Nguyen; Ngoc N. Tran; Taylor T. Johnson; Kevin Leach
  In recent years, the rise of machine learning (ML) in cybersecurity has
brought new challenges, including the increasing threat of backdoor poisoning
attacks on ML malware classifiers. For instance, adversaries could inject
malicious samples into public malware repositories, contaminating the training
data and potentially misclassifying malware by the ML model. Current
countermeasures predominantly focus on detecting poisoned samples by leveraging
disagreements within the outputs of a diverse set of ensemble models on
training data points. However, these methods are not suitable for scenarios
where Machine Learning-as-a-Service (MLaaS) is used or when users aim to remove
backdoors from a model after it has been trained. Addressing this scenario, we
introduce PBP, a post-training defense for malware classifiers that mitigates
various types of backdoor embeddings without assuming any specific backdoor
embedding mechanism. Our method exploits the influence of backdoor attacks on
the activation distribution of neural networks, independent of the
trigger-embedding method. In the presence of a backdoor attack, the activation
distribution of each layer is distorted into a mixture of distributions. By
regulating the statistics of the batch normalization layers, we can guide a
backdoored model to perform similarly to a clean one. Our method demonstrates
substantial advantages over several state-of-the-art methods, as evidenced by
experiments on two datasets, two types of backdoor methods, and various attack
configurations. Notably, our approach requires only a small portion of the
training data -- only 1\% -- to purify the backdoor and reduce the attack
success rate from 100\% to almost 0\%, a 100-fold improvement over the baseline
methods. Our code is available at
\url{https://github.com/judydnguyen/pbp-backdoor-purification-official}.


http://arxiv.org/abs/2412.03154
Testing Neural Network Verifiers: A Soundness Benchmark with Hidden Counterexamples. (13%)
Xingjian Zhou; Hongji Xu; Andy Xu; Zhouxing Shi; Cho-Jui Hsieh; Huan Zhang
  In recent years, many neural network (NN) verifiers have been developed to
formally verify certain properties of neural networks such as robustness.
Although many benchmarks have been constructed to evaluate the performance of
NN verifiers, they typically lack a ground-truth for hard instances where no
current verifier can verify and no counterexample can be found, which makes it
difficult to check the soundness of a new verifier if it claims to verify hard
instances which no other verifier can do. We propose to develop a soundness
benchmark for NN verification. Our benchmark contains instances with
deliberately inserted counterexamples while we also try to hide the
counterexamples from regular adversarial attacks which can be used for finding
counterexamples. We design a training method to produce neural networks with
such hidden counterexamples. Our benchmark aims to be used for testing the
soundness of NN verifiers and identifying falsely claimed verifiability when it
is known that hidden counterexamples exist. We systematically construct our
benchmark and generate instances across diverse model architectures, activation
functions, input sizes, and perturbation radii. We demonstrate that our
benchmark successfully identifies bugs in state-of-the-art NN verifiers, as
well as synthetic bugs, providing a crucial step toward enhancing the
reliability of testing NN verifiers. Our code is available at
https://github.com/MVP-Harry/SoundnessBench and our benchmark is available at
https://huggingface.co/datasets/SoundnessBench/SoundnessBench.


http://arxiv.org/abs/2412.03283
Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models. (12%)
Andreas Müller; Denis Lukovnikov; Jonas Thietke; Asja Fischer; Erwin Quiring
  Integrating watermarking into the generation process of latent diffusion
models (LDMs) simplifies detection and attribution of generated content.
Semantic watermarks, such as Tree-Rings and Gaussian Shading, represent a novel
class of watermarking techniques that are easy to implement and highly robust
against various perturbations. However, our work demonstrates a fundamental
security vulnerability of semantic watermarks. We show that attackers can
leverage unrelated models, even with different latent spaces and architectures
(UNet vs DiT), to perform powerful and realistic forgery attacks. Specifically,
we design two watermark forgery attacks. The first imprints a targeted
watermark into real images by manipulating the latent representation of an
arbitrary image in an unrelated LDM to get closer to the latent representation
of a watermarked image. We also show that this technique can be used for
watermark removal. The second attack generates new images with the target
watermark by inverting a watermarked image and re-generating it with an
arbitrary prompt. Both attacks just need a single reference image with the
target watermark. Overall, our findings question the applicability of semantic
watermarks by revealing that attackers can easily forge or remove these
watermarks under realistic conditions.


http://arxiv.org/abs/2412.03682
Designing DNNs for a trade-off between robustness and processing performance in embedded devices. (11%)
Jon Gutiérrez-Zaballa; Koldo Basterretxea; Javier Echanobe
  Machine learning-based embedded systems employed in safety-critical
applications such as aerospace and autonomous driving need to be robust against
perturbations produced by soft errors. Soft errors are an increasing concern in
modern digital processors since smaller transistor geometries and lower
voltages give electronic devices a higher sensitivity to background radiation.
The resilience of deep neural network (DNN) models to perturbations in their
parameters is determined, to a large extent, by the structure of the model
itself, and also by the selected numerical representation and used arithmetic
precision. When compression techniques such as model pruning and model
quantization are applied to reduce memory footprint and computational
complexity for deployment, both model structure and numerical representation
are modified and thus, soft error robustness also changes. In this sense,
although the choice of activation functions (AFs) in DNN models is frequently
ignored, it conditions not only their accuracy and trainability, but also
compressibility rates and numerical robustness. This paper investigates the
suitability of using bounded AFs to improve model robustness against DNN
parameter perturbations, assessing at the same time the impact of this choice
on deployment in terms of model accuracy, compressibility, and computational
burden. In particular, we analyze encoder-decoder fully convolutional models
aimed at performing semantic segmentation tasks on hyperspectral images for
scene understanding in autonomous driving. Deployment characterization is
performed experimentally on an AMD-Xilinx's KV260 SoM.


http://arxiv.org/abs/2412.03453
Pre-trained Multiple Latent Variable Generative Models are good defenders against Adversarial Attacks. (4%)
Dario Serez; Marco Cristani; Bue Alessio Del; Vittorio Murino; Pietro Morerio
  Attackers can deliberately perturb classifiers' input with subtle noise,
altering final predictions. Among proposed countermeasures, adversarial
purification employs generative networks to preprocess input images, filtering
out adversarial noise. In this study, we propose specific generators, defined
Multiple Latent Variable Generative Models (MLVGMs), for adversarial
purification. These models possess multiple latent variables that naturally
disentangle coarse from fine features. Taking advantage of these properties, we
autoencode images to maintain class-relevant information, while discarding and
re-sampling any detail, including adversarial noise. The procedure is
completely training-free, exploring the generalization abilities of pre-trained
MLVGMs on the adversarial purification downstream task. Despite the lack of
large models, trained on billions of samples, we show that smaller MLVGMs are
already competitive with traditional methods, and can be used as foundation
models. Official code released at https://github.com/SerezD/gen_adversarial.


http://arxiv.org/abs/2412.03630
Evaluating Single Event Upsets in Deep Neural Networks for Semantic Segmentation: an embedded system perspective. (1%)
Jon Gutiérrez-Zaballa; Koldo Basterretxea; Javier Echanobe
  As the deployment of artifical intelligence (AI) algorithms at edge devices
becomes increasingly prevalent, enhancing the robustness and reliability of
autonomous AI-based perception and decision systems is becoming as relevant as
precision and performance, especially in applications areas considered
safety-critical such as autonomous driving and aerospace. This paper delves
into the robustness assessment in embedded Deep Neural Networks (DNNs),
particularly focusing on the impact of parameter perturbations produced by
single event upsets (SEUs) on convolutional neural networks (CNN) for image
semantic segmentation. By scrutinizing the layer-by-layer and bit-by-bit
sensitivity of various encoder-decoder models to soft errors, this study
thoroughly investigates the vulnerability of segmentation DNNs to SEUs and
evaluates the consequences of techniques like model pruning and parameter
quantization on the robustness of compressed models aimed at embedded
implementations. The findings offer valuable insights into the mechanisms
underlying SEU-induced failures that allow for evaluating the robustness of
DNNs once trained in advance. Moreover, based on the collected data, we propose
a set of practical lightweight error mitigation techniques with no memory or
computational cost suitable for resource-constrained deployments. The code used
to perform the fault injection (FI) campaign is available at
https://github.com/jonGuti13/TensorFI2 , while the code to implement proposed
techniques is available at https://github.com/jonGuti13/parameterProtection .


http://arxiv.org/abs/2412.02270
Sustainable Self-evolution Adversarial Training. (99%)
Wenxuan Wang; Chenglei Wang; Huihui Qi; Menghao Ye; Xuelin Qian; Peng Wang; Yanning Zhang
  With the wide application of deep neural network models in various computer
vision tasks, there has been a proliferation of adversarial example generation
strategies aimed at deeply exploring model security. However, existing
adversarial training defense models, which rely on single or limited types of
attacks under a one-time learning process, struggle to adapt to the dynamic and
evolving nature of attack methods. Therefore, to achieve defense performance
improvements for models in long-term applications, we propose a novel
Sustainable Self-Evolution Adversarial Training (SSEAT) framework.
Specifically, we introduce a continual adversarial defense pipeline to realize
learning from various kinds of adversarial examples across multiple stages.
Additionally, to address the issue of model catastrophic forgetting caused by
continual learning from ongoing novel attacks, we propose an adversarial data
replay module to better select more diverse and key relearning data.
Furthermore, we design a consistency regularization strategy to encourage
current defense models to learn more from previously trained ones, guiding them
to retain more past knowledge and maintain accuracy on clean samples. Extensive
experiments have been conducted to verify the efficacy of the proposed SSEAT
defense method, which demonstrates superior defense performance and
classification accuracy compared to competitors.


http://arxiv.org/abs/2412.02803
Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objects. (99%)
Abdurrahman Zeybey; Mehmet Ergezer; Tommy Nguyen
  3D Gaussian Splatting has advanced radiance field reconstruction, enabling
high-quality view synthesis and fast rendering in 3D modeling. While
adversarial attacks on object detection models are well-studied for 2D images,
their impact on 3D models remains underexplored. This work introduces the
Masked Iterative Fast Gradient Sign Method (M-IFGSM), designed to generate
adversarial noise targeting the CLIP vision-language model. M-IFGSM
specifically alters the object of interest by focusing perturbations on masked
regions, degrading the performance of CLIP's zero-shot object detection
capability when applied to 3D models. Using eight objects from the Common
Objects 3D (CO3D) dataset, we demonstrate that our method effectively reduces
the accuracy and confidence of the model, with adversarial noise being nearly
imperceptible to human observers. The top-1 accuracy in original model renders
drops from 95.4\% to 12.5\% for train images and from 91.2\% to 35.4\% for test
images, with confidence levels reflecting this shift from true classification
to misclassification, underscoring the risks of adversarial attacks on 3D
models in applications such as autonomous driving, robotics, and surveillance.
The significance of this research lies in its potential to expose
vulnerabilities in modern 3D vision models, including radiance fields,
prompting the development of more robust defenses and security measures in
critical real-world applications.


http://arxiv.org/abs/2412.02343
Multi-Granularity Tibetan Textual Adversarial Attack Method Based on Masked Language Model. (98%)
Xi Cao; Nuo Qun; Quzong Gesang; Yulei Zhu; Trashi Nyima
  In social media, neural network models have been applied to hate speech
detection, sentiment analysis, etc., but neural network models are susceptible
to adversarial attacks. For instance, in a text classification task, the
attacker elaborately introduces perturbations to the original texts that hardly
alter the original semantics in order to trick the model into making different
predictions. By studying textual adversarial attack methods, the robustness of
language models can be evaluated and then improved. Currently, most of the
research in this field focuses on English, and there is also a certain amount
of research on Chinese. However, there is little research targeting Chinese
minority languages. With the rapid development of artificial intelligence
technology and the emergence of Chinese minority language models, textual
adversarial attacks become a new challenge for the information processing of
Chinese minority languages. In response to this situation, we propose a
multi-granularity Tibetan textual adversarial attack method based on masked
language models called TSTricker. We utilize the masked language models to
generate candidate substitution syllables or words, adopt the scoring mechanism
to determine the substitution order, and then conduct the attack method on
several fine-tuned victim models. The experimental results show that TSTricker
reduces the accuracy of the classification models by more than 28.70% and makes
the classification models change the predictions of more than 90.60% of the
samples, which has an evidently higher attack effect than the baseline method.


http://arxiv.org/abs/2412.02323
Pay Attention to the Robustness of Chinese Minority Language Models! Syllable-level Textual Adversarial Attack on Tibetan Script. (98%)
Xi Cao; Dolma Dawa; Nuo Qun; Trashi Nyima
  The textual adversarial attack refers to an attack method in which the
attacker adds imperceptible perturbations to the original texts by elaborate
design so that the NLP (natural language processing) model produces false
judgments. This method is also used to evaluate the robustness of NLP models.
Currently, most of the research in this field focuses on English, and there is
also a certain amount of research on Chinese. However, to the best of our
knowledge, there is little research targeting Chinese minority languages.
Textual adversarial attacks are a new challenge for the information processing
of Chinese minority languages. In response to this situation, we propose a
Tibetan syllable-level black-box textual adversarial attack called TSAttacker
based on syllable cosine distance and scoring mechanism. And then, we conduct
TSAttacker on six models generated by fine-tuning two PLMs (pre-trained
language models) for three downstream tasks. The experiment results show that
TSAttacker is effective and generates high-quality adversarial samples. In
addition, the robustness of the involved models still has much room for
improvement.


http://arxiv.org/abs/2412.02171
Can't Slow me Down: Learning Robust and Hardware-Adaptive Object Detectors against Latency Attacks for Edge Devices. (93%)
Tianyi Zhejiang University, Hangzhou, China Wang; Zichen Zhejiang University, Hangzhou, China Wang; Cong Zhejiang University, Hangzhou, China Wang; Yuanchao Zhejiang University, Hangzhou, China Shu; Ruilong Zhejiang University, Hangzhou, China Deng; Peng Zhejiang University, Hangzhou, China Cheng; Jiming Zhejiang University, Hangzhou, China Chen
  Object detection is a fundamental enabler for many real-time downstream
applications such as autonomous driving, augmented reality and supply chain
management. However, the algorithmic backbone of neural networks is brittle to
imperceptible perturbations in the system inputs, which were generally known as
misclassifying attacks. By targeting the real-time processing capability, a new
class of latency attacks are reported recently. They exploit new attack
surfaces in object detectors by creating a computational bottleneck in the
post-processing module, that leads to cascading failure and puts the real-time
downstream tasks at risks. In this work, we take an initial attempt to defend
against this attack via background-attentive adversarial training that is also
cognizant of the underlying hardware capabilities. We first draw system-level
connections between latency attack and hardware capacity across heterogeneous
GPU devices. Based on the particular adversarial behaviors, we utilize
objectness loss as a proxy and build background attention into the adversarial
training pipeline, and achieve a reasonable balance between clean and robust
accuracy. The extensive experiments demonstrate the defense effectiveness of
restoring real-time processing capability from $13$ FPS to $43$ FPS on Jetson
Orin NX, with a better trade-off between the clean and robust accuracy.


http://arxiv.org/abs/2412.02795
Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attacks. (80%)
Zijiao Yang; Xiangxi Shi; Eric Slyman; Stefan Lee
  Assistive embodied agents that can be instructed in natural language to
perform tasks in open-world environments have the potential to significantly
impact labor tasks like manufacturing or in-home care -- benefiting the lives
of those who come to depend on them. In this work, we consider how this benefit
might be hijacked by local modifications in the appearance of the agent's
operating environment. Specifically, we take the popular Vision-and-Language
Navigation (VLN) task as a representative setting and develop a whitebox
adversarial attack that optimizes a 3D attack object's appearance to induce
desired behaviors in pretrained VLN agents that observe it in the environment.
We demonstrate that the proposed attack can cause VLN agents to ignore their
instructions and execute alternative actions after encountering the attack
object -- even for instructions and agent paths not considered when optimizing
the attack. For these novel settings, we find our attacks can induce
early-termination behaviors or divert an agent along an attacker-defined
multi-step trajectory. Under both conditions, environmental attacks
significantly reduce agent capabilities to successfully follow user
instructions.


http://arxiv.org/abs/2412.02371
TSCheater: Generating High-Quality Tibetan Adversarial Texts via Visual Similarity. (76%)
Xi Cao; Quzong Gesang; Yuan Sun; Nuo Qun; Tashi Nyima
  Language models based on deep neural networks are vulnerable to textual
adversarial attacks. While rich-resource languages like English are receiving
focused attention, Tibetan, a cross-border language, is gradually being studied
due to its abundant ancient literature and critical language strategy.
Currently, there are several Tibetan adversarial text generation methods, but
they do not fully consider the textual features of Tibetan script and
overestimate the quality of generated adversarial texts. To address this issue,
we propose a novel Tibetan adversarial text generation method called TSCheater,
which considers the characteristic of Tibetan encoding and the feature that
visually similar syllables have similar semantics. This method can also be
transferred to other abugidas, such as Devanagari script. We utilize a
self-constructed Tibetan syllable visual similarity database called TSVSDB to
generate substitution candidates and adopt a greedy algorithm-based scoring
mechanism to determine substitution order. After that, we conduct the method on
eight victim language models. Experimentally, TSCheater outperforms existing
methods in attack effectiveness, perturbation magnitude, semantic similarity,
visual similarity, and human acceptance. Finally, we construct the first
Tibetan adversarial robustness evaluation benchmark called AdvTS, which is
generated by existing methods and proofread by humans.


http://arxiv.org/abs/2412.02576
The Efficacy of Transfer-based No-box Attacks on Image Watermarking: A Pragmatic Analysis. (61%)
Qilong Wu; Varun Chandrasekaran
  Watermarking approaches are widely used to identify if images being
circulated are authentic or AI-generated. Determining the robustness of image
watermarking methods in the ``no-box'' setting, where the attacker is assumed
to have no knowledge about the watermarking model, is an interesting problem.
Our main finding is that evading the no-box setting is challenging: the success
of optimization-based transfer attacks (involving training surrogate models)
proposed in prior work~\cite{hu2024transfer} depends on impractical
assumptions, including (i) aligning the architecture and training
configurations of both the victim and attacker's surrogate watermarking models,
as well as (ii) a large number of surrogate models with potentially large
computational requirements. Relaxing these assumptions i.e., moving to a more
pragmatic threat model results in a failed attack, with an evasion rate at most
$21.1\%$. We show that when the configuration is mostly aligned, a simple
non-optimization attack we propose, OFT, with one single surrogate model can
already exceed the success of optimization-based efforts. Under the same
$\ell_\infty$ norm perturbation budget of $0.25$, prior
work~\citet{hu2024transfer} is comparable to or worse than OFT in $11$ out of
$12$ configurations and has a limited advantage on the remaining one. The code
used for all our experiments is available at
\url{https://github.com/Ardor-Wu/transfer}.


http://arxiv.org/abs/2412.03002
AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations? (61%)
Shouwei Ruan; Hanqing Liu; Yao Huang; Xiaoqi Wang; Caixin Kang; Hang Su; Yinpeng Dong; Xingxing Wei
  Vision Language Models (VLMs) have exhibited remarkable generalization
capabilities, yet their robustness in dynamic real-world scenarios remains
largely unexplored. To systematically evaluate VLMs' robustness to real-world
3D variations, we propose AdvDreamer, the first framework capable of generating
physically reproducible Adversarial 3D Transformation (Adv-3DT) samples from
single-view observations. In AdvDreamer, we integrate three key innovations:
Firstly, to characterize real-world 3D variations with limited prior knowledge
precisely, we design a zero-shot Monocular Pose Manipulation pipeline built
upon generative 3D priors. Secondly, to ensure the visual quality of worst-case
Adv-3DT samples, we propose a Naturalness Reward Model that provides continuous
naturalness regularization during adversarial optimization, effectively
preventing convergence to hallucinated or unnatural elements. Thirdly, to
enable systematic evaluation across diverse VLM architectures and
visual-language tasks, we introduce the Inverse Semantic Probability loss as
the adversarial optimization objective, which solely operates in the
fundamental visual-textual alignment space. Based on the captured Adv-3DT
samples with high aggressiveness and transferability, we establish MM3DTBench,
the first VQA benchmark dataset tailored to evaluate VLM robustness under
challenging 3D variations. Extensive evaluations of representative VLMs with
varying architectures reveal that real-world 3D variations can pose severe
threats to model performance across various tasks.


http://arxiv.org/abs/2412.02535
Defending Against Diverse Attacks in Federated Learning Through Consensus-Based Bi-Level Optimization. (22%)
Nicolás García Trillos; Aditya Kumar Akash; Sixu Li; Konstantin Riedl; Yuhua Zhu
  Adversarial attacks pose significant challenges in many machine learning
applications, particularly in the setting of distributed training and federated
learning, where malicious agents seek to corrupt the training process with the
goal of jeopardizing and compromising the performance and reliability of the
final models. In this paper, we address the problem of robust federated
learning in the presence of such attacks by formulating the training task as a
bi-level optimization problem. We conduct a theoretical analysis of the
resilience of consensus-based bi-level optimization (CB$^2$O), an interacting
multi-particle metaheuristic optimization method, in adversarial settings.
Specifically, we provide a global convergence analysis of CB$^2$O in mean-field
law in the presence of malicious agents, demonstrating the robustness of
CB$^2$O against a diverse range of attacks. Thereby, we offer insights into how
specific hyperparameter choices enable to mitigate adversarial effects. On the
practical side, we extend CB$^2$O to the clustered federated learning setting
by proposing FedCB$^2$O, a novel interacting multi-particle system, and design
a practical algorithm that addresses the demands of real-world applications.
Extensive experiments demonstrate the robustness of the FedCB$^2$O algorithm
against label-flipping attacks in decentralized clustered federated learning
scenarios, showcasing its effectiveness in practical contexts.


http://arxiv.org/abs/2412.02479
OODFace: Benchmarking Robustness of Face Recognition under Common Corruptions and Appearance Variations. (11%)
Caixin Kang; Yubo Chen; Shouwei Ruan; Shiji Zhao; Ruochen Zhang; Jiayi Wang; Shan Fu; Xingxing Wei
  With the rise of deep learning, facial recognition technology has seen
extensive research and rapid development. Although facial recognition is
considered a mature technology, we find that existing open-source models and
commercial algorithms lack robustness in certain complex Out-of-Distribution
(OOD) scenarios, raising concerns about the reliability of these systems. In
this paper, we introduce OODFace, which explores the OOD challenges faced by
facial recognition models from two perspectives: common corruptions and
appearance variations. We systematically design 30 OOD scenarios across 9 major
categories tailored for facial recognition. By simulating these challenges on
public datasets, we establish three robustness benchmarks: LFW-C/V, CFP-FP-C/V,
and YTF-C/V. We then conduct extensive experiments on 19 facial recognition
models and 3 commercial APIs, along with extended physical experiments on face
masks to assess their robustness. Next, we explore potential solutions from two
perspectives: defense strategies and Vision-Language Models (VLMs). Based on
the results, we draw several key insights, highlighting the vulnerability of
facial recognition systems to OOD data and suggesting possible solutions.
Additionally, we offer a unified toolkit that includes all corruption and
variation types, easily extendable to other datasets. We hope that our
benchmarks and findings can provide guidance for future improvements in facial
recognition model robustness.


http://arxiv.org/abs/2412.02454
Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retraining. (2%)
Zongru Wu; Pengzhou Cheng; Lingyong Fang; Zhuosheng Zhang; Gongshen Liu
  Backdoor attacks remain significant security threats to generative large
language models (LLMs). Since generative LLMs output sequences of
high-dimensional token logits instead of low-dimensional classification logits,
most existing backdoor defense methods designed for discriminative models like
BERT are ineffective for generative LLMs. Inspired by the observed differences
in learning behavior between backdoor and clean mapping in the frequency space,
we transform gradients of each training sample, directly influencing parameter
updates, into the frequency space. Our findings reveal a distinct separation
between the gradients of backdoor and clean samples in the frequency space.
Based on this phenomenon, we propose Gradient Clustering in the Frequency Space
for Backdoor Sample Filtering (GraCeFul), which leverages sample-wise gradients
in the frequency space to effectively identify backdoor samples without
requiring retraining LLMs. Experimental results show that GraCeFul outperforms
baselines significantly. Notably, GraCeFul exhibits remarkable computational
efficiency, achieving nearly 100% recall and F1 scores in identifying backdoor
samples, reducing the average success rate of various backdoor attacks to 0%
with negligible drops in clean accuracy across multiple free-style question
answering datasets. Additionally, GraCeFul generalizes to Llama-2 and Vicuna.
The codes are publicly available at https://github.com/ZrW00/GraceFul.


http://arxiv.org/abs/2412.01527
Traversing the Subspace of Adversarial Patches. (83%)
Jens Bayer; Stefan Becker; David Münch; Michael Arens; Jürgen Beyerer
  Despite ongoing research on the topic of adversarial examples in deep
learning for computer vision, some fundamentals of the nature of these attacks
remain unclear. As the manifold hypothesis posits, high-dimensional data tends
to be part of a low-dimensional manifold. To verify the thesis with adversarial
patches, this paper provides an analysis of a set of adversarial patches and
investigates the reconstruction abilities of three different dimensionality
reduction methods. Quantitatively, the performance of reconstructed patches in
an attack setting is measured and the impact of sampled patches from the latent
space during adversarial training is investigated. The evaluation is performed
on two publicly available datasets for person detection. The results indicate
that more sophisticated dimensionality reduction methods offer no advantages
over a simple principal component analysis.


http://arxiv.org/abs/2412.01440
DiffPatch: Generating Customizable Adversarial Patches using Diffusion Model. (82%)
Zhixiang Wang; Guangnan Ye; Xiaosen Wang; Siheng Chen; Zhibo Wang; Xingjun Ma; Yu-Gang Jiang
  Physical adversarial patches printed on clothing can easily allow individuals
to evade person detectors. However, most existing adversarial patch generation
methods prioritize attack effectiveness over stealthiness, resulting in patches
that are aesthetically unpleasing. Although existing methods using generative
adversarial networks or diffusion models can produce more natural-looking
patches, they often struggle to balance stealthiness with attack effectiveness
and lack flexibility for user customization. To address these challenges, we
propose a novel diffusion-based customizable patch generation framework termed
DiffPatch, specifically tailored for creating naturalistic and customizable
adversarial patches. Our approach enables users to utilize a reference image as
the source, rather than starting from random noise, and incorporates masks to
craft naturalistic patches of various shapes, not limited to squares. To
prevent the original semantics from being lost during the diffusion process, we
employ Null-text inversion to map random noise samples to a single input image
and generate patches through Incomplete Diffusion Optimization (IDO). Notably,
while maintaining a natural appearance, our method achieves a comparable attack
performance to state-of-the-art non-naturalistic patches when using similarly
sized attacks. Using DiffPatch, we have created a physical adversarial T-shirt
dataset, AdvPatch-1K, specifically targeting YOLOv5s. This dataset includes
over a thousand images across diverse scenarios, validating the effectiveness
of our attack in real-world environments. Moreover, it provides a valuable
resource for future research.


http://arxiv.org/abs/2412.01363
Exploring the Robustness of AI-Driven Tools in Digital Forensics: A Preliminary Study. (74%)
Silvia Lucia Sanna; Leonardo Regano; Davide Maiorca; Giorgio Giacinto
  Nowadays, many tools are used to facilitate forensic tasks about data
extraction and data analysis. In particular, some tools leverage Artificial
Intelligence (AI) to automatically label examined data into specific categories
(\ie, drugs, weapons, nudity). However, this raises a serious concern about the
robustness of the employed AI algorithms against adversarial attacks. Indeed,
some people may need to hide specific data to AI-based digital forensics tools,
thus manipulating the content so that the AI system does not recognize the
offensive/prohibited content and marks it at as suspicious to the analyst. This
could be seen as an anti-forensics attack scenario. For this reason, we
analyzed two of the most important forensics tools employing AI for data
classification: Magnet AI, used by Magnet Axiom, and Excire Photo AI, used by
X-Ways Forensics. We made preliminary tests using about $200$ images, other
$100$ sent in $3$ chats about pornography and teenage nudity, drugs and weapons
to understand how the tools label them. Moreover, we loaded some deepfake
images (images generated by AI forging real ones) of some actors to understand
if they would be classified in the same category as the original images. From
our preliminary study, we saw that the AI algorithm is not robust enough, as we
expected since these topics are still open research problems. For example, some
sexual images were not categorized as nudity, and some deepfakes were
categorized as the same real person, while the human eye can see the clear
nudity image or catch the difference between the deepfakes. Building on these
results and other state-of-the-art works, we provide some suggestions for
improving how digital forensics analysis tool leverage AI and their robustness
against adversarial attacks or different scenarios than the trained one.


http://arxiv.org/abs/2412.01756
Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios. (69%)
Sangyeon Yoon; Wonje Jeung; Albert No
  Auditing Differentially Private Stochastic Gradient Descent (DP-SGD) in the
final model setting is challenging and often results in empirical lower bounds
that are significantly looser than theoretical privacy guarantees. We introduce
a novel auditing method that achieves tighter empirical lower bounds without
additional assumptions by crafting worst-case adversarial samples through
loss-based input-space auditing. Our approach surpasses traditional
canary-based heuristics and is effective in both white-box and black-box
scenarios. Specifically, with a theoretical privacy budget of $\varepsilon =
10.0$, our method achieves empirical lower bounds of $6.68$ in white-box
settings and $4.51$ in black-box settings, compared to the baseline of $4.11$
for MNIST. Moreover, we demonstrate that significant privacy auditing results
can be achieved using in-distribution (ID) samples as canaries, obtaining an
empirical lower bound of $4.33$ where traditional methods produce near-zero
leakage detection. Our work offers a practical framework for reliable and
accurate privacy auditing in differentially private machine learning.


http://arxiv.org/abs/2412.01646
Robust and Transferable Backdoor Attacks Against Deep Image Compression With Selective Frequency Prior. (67%)
Yi Yu; Yufei Wang; Wenhan Yang; Lanqing Guo; Shijian Lu; Ling-Yu Duan; Yap-Peng Tan; Alex C. Kot
  Recent advancements in deep learning-based compression techniques have
surpassed traditional methods. However, deep neural networks remain vulnerable
to backdoor attacks, where pre-defined triggers induce malicious behaviors.
This paper introduces a novel frequency-based trigger injection model for
launching backdoor attacks with multiple triggers on learned image compression
models. Inspired by the widely used DCT in compression codecs, triggers are
embedded in the DCT domain. We design attack objectives tailored to diverse
scenarios, including: 1) degrading compression quality in terms of bit-rate and
reconstruction accuracy; 2) targeting task-driven measures like face
recognition and semantic segmentation. To improve training efficiency, we
propose a dynamic loss function that balances loss terms with fewer
hyper-parameters, optimizing attack objectives effectively. For advanced
scenarios, we evaluate the attack's resistance to defensive preprocessing and
propose a two-stage training schedule with robust frequency selection to
enhance resilience. To improve cross-model and cross-domain transferability for
downstream tasks, we adjust the classification boundary in the attack loss
during training. Experiments show that our trigger injection models, combined
with minor modifications to encoder parameters, successfully inject multiple
backdoors and their triggers into a single compression model, demonstrating
strong performance and versatility. (*Due to the notification of arXiv "The
Abstract field cannot be longer than 1,920 characters", the appeared Abstract
is shortened. For the full Abstract, please download the Article.)


http://arxiv.org/abs/2412.01495
Adversarial Attacks on Hyperbolic Networks. (26%)
Spengler Max van; Jan Zahálka; Pascal Mettes
  As hyperbolic deep learning grows in popularity, so does the need for
adversarial robustness in the context of such a non-Euclidean geometry. To this
end, this paper proposes hyperbolic alternatives to the commonly used FGM and
PGD adversarial attacks. Through interpretable synthetic benchmarks and
experiments on existing datasets, we show how the existing and newly proposed
attacks differ. Moreover, we investigate the differences in adversarial
robustness between Euclidean and fully hyperbolic networks. We find that these
networks suffer from different types of vulnerabilities and that the newly
proposed hyperbolic attacks cannot address these differences. Therefore, we
conclude that the shifts in adversarial robustness are due to the models
learning distinct patterns resulting from their different geometries.


http://arxiv.org/abs/2412.02156
Compromising the Intelligence of Modern DNNs: On the Effectiveness of Targeted RowPress. (13%)
Ranyang Zhou; Jacqueline T. Liu; Sabbir Ahmed; Shaahin Angizi; Adnan Siraj Rakin
  Recent advancements in side-channel attacks have revealed the vulnerability
of modern Deep Neural Networks (DNNs) to malicious adversarial weight attacks.
The well-studied RowHammer attack has effectively compromised DNN performance
by inducing precise and deterministic bit-flips in the main memory (e.g.,
DRAM). Similarly, RowPress has emerged as another effective strategy for
flipping targeted bits in DRAM. However, the impact of RowPress on deep
learning applications has yet to be explored in the existing literature,
leaving a fundamental research question unanswered: How does RowPress compare
to RowHammer in leveraging bit-flip attacks to compromise DNN performance? This
paper is the first to address this question and evaluate the impact of RowPress
on DNN applications. We conduct a comparative analysis utilizing a novel
DRAM-profile-aware attack designed to capture the distinct bit-flip patterns
caused by RowHammer and RowPress. Eleven widely-used DNN architectures trained
on different benchmark datasets deployed on a Samsung DRAM chip conclusively
demonstrate that they suffer from a drastically more rapid performance
degradation under the RowPress attack compared to RowHammer. The difference in
the underlying attack mechanism of RowHammer and RowPress also renders existing
RowHammer mitigation mechanisms ineffective under RowPress. As a result,
RowPress introduces a new vulnerability paradigm for DNN compute platforms and
unveils the urgent need for corresponding protective measures.


http://arxiv.org/abs/2412.01154
R.I.P.: A Simple Black-box Attack on Continual Test-time Adaptation. (5%)
Trung-Hieu Hoang; Duc Minh Vo; Minh N. Do
  Test-time adaptation (TTA) has emerged as a promising solution to tackle the
continual domain shift in machine learning by allowing model parameters to
change at test time, via self-supervised learning on unlabeled testing data. At
the same time, it unfortunately opens the door to unforeseen vulnerabilities
for degradation over time. Through a simple theoretical continual TTA model, we
successfully identify a risk in the sampling process of testing data that could
easily degrade the performance of a continual TTA model. We name this risk as
Reusing of Incorrect Prediction (RIP) that TTA attackers can employ or as a
result of the unintended query from general TTA users. The risk posed by RIP is
also highly realistic, as it does not require prior knowledge of model
parameters or modification of testing samples. This simple requirement makes
RIP as the first black-box TTA attack algorithm that stands out from existing
white-box attempts. We extensively benchmark the performance of the most recent
continual TTA approaches when facing the RIP attack, providing insights on its
success, and laying out potential roadmaps that could enhance the resilience of
future continual TTA systems.


http://arxiv.org/abs/2412.01975
Reactive Synthesis of Sensor Revealing Strategies in Hypergames on Graphs. (1%)
Sumukha Udupa; Ahmed Hemida; Charles A. Kamhoua; Jie Fu
  In many security applications of cyber-physical systems, a system designer
must guarantee that critical missions are satisfied against attacks in the
sensors and actuators of the CPS. Traditional security design of CPSs often
assume that attackers have complete knowledge of the system. In this article,
we introduce a class of deception techniques and study how to leverage
asymmetric information created by deception to strengthen CPS security.
Consider an adversarial interaction between a CPS defender and an attacker, who
can perform sensor jamming attacks. To mitigate such attacks, the defender
introduces asymmetrical information by deploying a "hidden sensor," whose
presence is initially undisclosed but can be revealed if queried. We introduce
hypergames on graphs to model this game with asymmetric information. Building
on the solution concept called subjective rationalizable strategies in
hypergames, we identify two stages in the game: An initial game stage where the
defender commits to a strategy perceived rationalizable by the attacker until
he deviates from the equilibrium in the attacker's perceptual game; Upon the
deviation, a delay-attack game stage starts where the defender plays against
the attacker, who has a bounded delay in attacking the sensor being revealed.
Based on backward induction, we develop an algorithm that determines, for any
given state, if the defender can benefit from hiding a sensor and revealing it
later. If the answer is affirmative, the algorithm outputs a sensor revealing
strategy to determine when to reveal the sensor during dynamic interactions. We
demonstrate the effectiveness of our deceptive strategies through two case
studies related to CPS security applications.


http://arxiv.org/abs/2412.01127
Precision Profile Pollution Attack on Sequential Recommenders via Influence Function. (1%)
Xiaoyu Du; Yingying Chen; Yang Zhang; Jinhui Tang
  Sequential recommendation approaches have demonstrated remarkable proficiency
in modeling user preferences. Nevertheless, they are susceptible to profile
pollution attacks (PPA), wherein items are introduced into a user's interaction
history deliberately to influence the recommendation list. Since retraining the
model for each polluted item is time-consuming, recent PPAs estimate item
influence based on gradient directions to identify the most effective attack
candidates. However, the actual item representations diverge significantly from
the gradients, resulting in disparate outcomes.To tackle this challenge, we
introduce an INFluence Function-based Attack approach INFAttack that offers a
more accurate estimation of the influence of polluting items. Specifically, we
calculate the modifications to the original model using the influence function
when generating polluted sequences by introducing specific items. Subsequently,
we choose the sequence that has been most significantly influenced to
substitute the original sequence, thus promoting the target item. Comprehensive
experiments conducted on five real-world datasets illustrate that INFAttack
surpasses all baseline methods and consistently delivers stable attack
performance for both popular and unpopular items.


http://arxiv.org/abs/2412.00696
Intermediate Outputs Are More Sensitive Than You Think. (61%)
Tao Huang; Qingyu Huang; Jiayang Meng
  The increasing reliance on deep computer vision models that process sensitive
data has raised significant privacy concerns, particularly regarding the
exposure of intermediate results in hidden layers. While traditional privacy
risk assessment techniques focus on protecting overall model outputs, they
often overlook vulnerabilities within these intermediate representations.
Current privacy risk assessment techniques typically rely on specific attack
simulations to assess risk, which can be computationally expensive and
incomplete. This paper introduces a novel approach to measuring privacy risks
in deep computer vision models based on the Degrees of Freedom (DoF) and
sensitivity of intermediate outputs, without requiring adversarial attack
simulations. We propose a framework that leverages DoF to evaluate the amount
of information retained in each layer and combines this with the rank of the
Jacobian matrix to assess sensitivity to input variations. This dual analysis
enables systematic measurement of privacy risks at various model layers. Our
experimental validation on real-world datasets demonstrates the effectiveness
of this approach in providing deeper insights into privacy risks associated
with intermediate representations.


http://arxiv.org/abs/2412.01101
Hiding Faces in Plain Sight: Defending DeepFakes by Disrupting Face Detection. (16%)
Delong Zhu; Yuezun Li; Baoyuan Wu; Jiaran Zhou; Zhibo Wang; Siwei Lyu
  This paper investigates the feasibility of a proactive DeepFake defense
framework, {\em FacePosion}, to prevent individuals from becoming victims of
DeepFake videos by sabotaging face detection. The motivation stems from the
reliance of most DeepFake methods on face detectors to automatically extract
victim faces from videos for training or synthesis (testing). Once the face
detectors malfunction, the extracted faces will be distorted or incorrect,
subsequently disrupting the training or synthesis of the DeepFake model. To
achieve this, we adapt various adversarial attacks with a dedicated design for
this purpose and thoroughly analyze their feasibility. Based on FacePoison, we
introduce {\em VideoFacePoison}, a strategy that propagates FacePoison across
video frames rather than applying them individually to each frame. This
strategy can largely reduce the computational overhead while retaining the
favorable attack performance. Our method is validated on five face detectors,
and extensive experiments against eleven different DeepFake models demonstrate
the effectiveness of disrupting face detectors to hinder DeepFake generation.


http://arxiv.org/abs/2412.00797
Online Poisoning Attack Against Reinforcement Learning under Black-box Environments. (11%)
Jianhui Li; Bokang Zhang; Junfeng Wu
  This paper proposes an online environment poisoning algorithm tailored for
reinforcement learning agents operating in a black-box setting, where an
adversary deliberately manipulates training data to lead the agent toward a
mischievous policy. In contrast to prior studies that primarily investigate
white-box settings, we focus on a scenario characterized by \textit{unknown}
environment dynamics to the attacker and a \textit{flexible} reinforcement
learning algorithm employed by the targeted agent. We first propose an attack
scheme that is capable of poisoning the reward functions and state transitions.
The poisoning task is formalized as a constrained optimization problem,
following the framework of \cite{ma2019policy}. Given the transition
probabilities are unknown to the attacker in a black-box environment, we apply
a stochastic gradient descent algorithm, where the exact gradients are
approximated using sample-based estimates. A penalty-based method along with a
bilevel reformulation is then employed to transform the problem into an
unconstrained counterpart and to circumvent the double-sampling issue. The
algorithm's effectiveness is validated through a maze environment.


http://arxiv.org/abs/2412.00404
Hard-Label Black-Box Attacks on 3D Point Clouds. (99%)
Daizong Liu; Yunbo Tao; Pan Zhou; Wei Hu
  With the maturity of depth sensors in various 3D safety-critical
applications, 3D point cloud models have been shown to be vulnerable to
adversarial attacks. Almost all existing 3D attackers simply follow the
white-box or black-box setting to iteratively update coordinate perturbations
based on back-propagated or estimated gradients. However, these methods are
hard to deploy in real-world scenarios (no model details are provided) as they
severely rely on parameters or output logits of victim models. To this end, we
propose point cloud attacks from a more practical setting, i.e., hard-label
black-box attack, in which attackers can only access the prediction label of 3D
input. We introduce a novel 3D attack method based on a new spectrum-aware
decision boundary algorithm to generate high-quality adversarial samples. In
particular, we first construct a class-aware model decision boundary, by
developing a learnable spectrum-fusion strategy to adaptively fuse point clouds
of different classes in the spectral domain, aiming to craft their intermediate
samples without distorting the original geometry. Then, we devise an iterative
coordinate-spectrum optimization method with curvature-aware boundary search to
move the intermediate sample along the decision boundary for generating
adversarial point clouds with trivial perturbations. Experiments demonstrate
that our attack competitively outperforms existing white/black-box attackers in
terms of attack performance and adversary quality.


http://arxiv.org/abs/2412.00621
Exposing LLM Vulnerabilities: Adversarial Scam Detection and Performance. (69%)
Chen-Wei Chang; Shailik Sarkar; Shutonu Mitra; Qi Zhang; Hossein Salemi; Hemant Purohit; Fengxiu Zhang; Michin Hong; Jin-Hee Cho; Chang-Tien Lu
  Can we trust Large Language Models (LLMs) to accurately predict scam? This
paper investigates the vulnerabilities of LLMs when facing adversarial scam
messages for the task of scam detection. We addressed this issue by creating a
comprehensive dataset with fine-grained labels of scam messages, including both
original and adversarial scam messages. The dataset extended traditional binary
classes for the scam detection task into more nuanced scam types. Our analysis
showed how adversarial examples took advantage of vulnerabilities of a LLM,
leading to high misclassification rate. We evaluated the performance of LLMs on
these adversarial scam messages and proposed strategies to improve their
robustness.


http://arxiv.org/abs/2412.00537
Exact Certification of (Graph) Neural Networks Against Label Poisoning. (22%)
Mahalakshmi Sabanayagam; Lukas Gosch; Stephan Günnemann; Debarghya Ghoshdastidar
  Machine learning models are highly vulnerable to label flipping, i.e., the
adversarial modification (poisoning) of training labels to compromise
performance. Thus, deriving robustness certificates is important to guarantee
that test predictions remain unaffected and to understand worst-case robustness
behavior. However, for Graph Neural Networks (GNNs), the problem of certifying
label flipping has so far been unsolved. We change this by introducing an exact
certification method, deriving both sample-wise and collective certificates.
Our method leverages the Neural Tangent Kernel (NTK) to capture the training
dynamics of wide networks enabling us to reformulate the bilevel optimization
problem representing label flipping into a Mixed-Integer Linear Program (MILP).
We apply our method to certify a broad range of GNN architectures in node
classification tasks. Thereby, concerning the worst-case robustness to label
flipping: $(i)$ we establish hierarchies of GNNs on different benchmark graphs;
$(ii)$ quantify the effect of architectural choices such as activations, depth
and skip-connections; and surprisingly, $(iii)$ uncover a novel phenomenon of
the robustness plateauing for intermediate perturbation budgets across all
investigated datasets and architectures. While we focus on GNNs, our
certificates are applicable to sufficiently wide NNs in general through their
NTK. Thus, our work presents the first exact certificate to a poisoning attack
ever derived for neural networks, which could be of independent interest.


http://arxiv.org/abs/2412.00473
Jailbreak Large Vision-Language Models Through Multi-Modal Linkage. (12%)
Yu Wang; Xiaofei Zhou; Yichen Wang; Geyuan Zhang; Tianxing He
  With the significant advancement of Large Vision-Language Models (VLMs),
concerns about their potential misuse and abuse have grown rapidly. Previous
studies have highlighted VLMs' vulnerability to jailbreak attacks, where
carefully crafted inputs can lead the model to produce content that violates
ethical and legal standards. However, existing methods struggle against
state-of-the-art VLMs like GPT-4o, due to the over-exposure of harmful content
and lack of stealthy malicious guidance. In this work, we propose a novel
jailbreak attack framework: Multi-Modal Linkage (MML) Attack. Drawing
inspiration from cryptography, MML utilizes an encryption-decryption process
across text and image modalities to mitigate over-exposure of malicious
information. To align the model's output with malicious intent covertly, MML
employs a technique called "evil alignment", framing the attack within a video
game production scenario. Comprehensive experiments demonstrate MML's
effectiveness. Specifically, MML jailbreaks GPT-4o with attack success rates of
97.80% on SafeBench, 98.81% on MM-SafeBench and 99.07% on HADES-Dataset. Our
code is available at https://github.com/wangyu-ovo/MML.


http://arxiv.org/abs/2411.19853
Towards Class-wise Robustness Analysis. (99%)
Tejaswini Medi; Julia Grabinski; Margret Keuper
  While being very successful in solving many downstream tasks, the application
of deep neural networks is limited in real-life scenarios because of their
susceptibility to domain shifts such as common corruptions, and adversarial
attacks. The existence of adversarial examples and data corruption
significantly reduces the performance of deep classification models.
Researchers have made strides in developing robust neural architectures to
bolster decisions of deep classifiers. However, most of these works rely on
effective adversarial training methods, and predominantly focus on overall
model robustness, disregarding class-wise differences in robustness, which are
critical. Exploiting weakly robust classes is a potential avenue for attackers
to fool the image recognition models. Therefore, this study investigates
class-to-class biases across adversarially trained robust classification models
to understand their latent space structures and analyze their strong and weak
class-wise properties. We further assess the robustness of classes against
common corruptions and adversarial attacks, recognizing that class
vulnerability extends beyond the number of correct classifications for a
specific class. We find that the number of false positives of classes as
specific target classes significantly impacts their vulnerability to attacks.
Through our analysis on the Class False Positive Score, we assess a fair
evaluation of how susceptible each class is to misclassification.


http://arxiv.org/abs/2411.19479
FLARE: Towards Universal Dataset Purification against Backdoor Attacks. (81%)
Linshan Hou; Wei Luo; Zhongyun Hua; Songhua Chen; Leo Yu Zhang; Yiming Li
  Deep neural networks (DNNs) are susceptible to backdoor attacks, where
adversaries poison datasets with adversary-specified triggers to implant hidden
backdoors, enabling malicious manipulation of model predictions. Dataset
purification serves as a proactive defense by removing malicious training
samples to prevent backdoor injection at its source. We first reveal that the
current advanced purification methods rely on a latent assumption that the
backdoor connections between triggers and target labels in backdoor attacks are
simpler to learn than the benign features. We demonstrate that this assumption,
however, does not always hold, especially in all-to-all (A2A) and untargeted
(UT) attacks. As a result, purification methods that analyze the separation
between the poisoned and benign samples in the input-output space or the final
hidden layer space are less effective. We observe that this separability is not
confined to a single layer but varies across different hidden layers. Motivated
by this understanding, we propose FLARE, a universal purification method to
counter various backdoor attacks. FLARE aggregates abnormal activations from
all hidden layers to construct representations for clustering. To enhance
separation, FLARE develops an adaptive subspace selection algorithm to isolate
the optimal space for dividing an entire dataset into two clusters. FLARE
assesses the stability of each cluster and identifies the cluster with higher
stability as poisoned. Extensive evaluations on benchmark datasets demonstrate
the effectiveness of FLARE against 22 representative backdoor attacks,
including all-to-one (A2O), all-to-all (A2A), and untargeted (UT) attacks, and
its robustness to adaptive attacks.


http://arxiv.org/abs/2412.00324
Robust Table Integration in Data Lakes. (56%)
Daomin Ji; Hui Luo; Zhifeng Bao; Shane Culpepper
  In this paper, we investigate the challenge of integrating tables from data
lakes, focusing on three core tasks: 1) pairwise integrability judgment, which
determines whether a tuple pair in a table is integrable, accounting for any
occurrences of semantic equivalence or typographical errors; 2) integrable set
discovery, which aims to identify all integrable sets in a table based on
pairwise integrability judgments established in the first task; 3) multi-tuple
conflict resolution, which resolves conflicts among multiple tuples during
integration. We train a binary classifier to address the task of pairwise
integrability judgment. Given the scarcity of labeled data, we propose a
self-supervised adversarial contrastive learning algorithm to perform
classification, which incorporates data augmentation methods and adversarial
examples to autonomously generate new training data. Upon the output of
pairwise integrability judgment, each integrable set is considered as a
community, a densely connected sub-graph where nodes and edges correspond to
tuples in the table and their pairwise integrability, respectively. We proceed
to investigate various community detection algorithms to address the integrable
set discovery objective. Moving forward to tackle multi-tuple conflict
resolution, we introduce an novel in-context learning methodology. This
approach capitalizes on the knowledge embedded within pretrained large language
models to effectively resolve conflicts that arise when integrating multiple
tuples. Notably, our method minimizes the need for annotated data. Since no
suitable test collections are available for our tasks, we develop our own
benchmarks using two real-word dataset repositories: Real and Join. We conduct
extensive experiments on these benchmarks to validate the robustness and
applicability of our methodologies in the context of integrating tables within
data lakes.


http://arxiv.org/abs/2411.19508
On the Adversarial Robustness of Instruction-Tuned Large Language Models for Code. (38%)
Md Imran Hossen; Xiali Hei
  The advent of instruction-tuned Large Language Models designed for coding
tasks (Code LLMs) has transformed software engineering practices. However,
their robustness against various input challenges remains a critical concern.
This study introduces DegradePrompter, a novel method designed to
systematically evaluate the robustness of instruction-tuned Code LLMs. We
assess the impact of diverse input challenges on the functionality and
correctness of generated code using rigorous metrics and established
benchmarks. Our comprehensive evaluation includes five state-of-the-art
open-source models and three production-grade closed-source models, revealing
varying degrees of robustness. Open-source models demonstrate an increased
susceptibility to input perturbations, resulting in declines in functional
correctness ranging from 12% to 34%. In contrast, commercial models demonstrate
relatively greater resilience, with performance degradation ranging from 3% to
24%. To enhance the robustness of the models against these vulnerabilities, we
investigate a straightforward yet effective mitigation strategy. Our findings
highlight the need for robust defense mechanisms and comprehensive evaluations
during both the development and deployment phases to ensure the resilience and
reliability of automated code generation systems.


http://arxiv.org/abs/2411.19841
Parallel Stacked Aggregated Network for Voice Authentication in IoT-Enabled Smart Devices. (10%)
Awais Khan; Ijaz Ul Haq; Khalid Mahmood Malik
  Voice authentication on IoT-enabled smart devices has gained prominence in
recent years due to increasing concerns over user privacy and security. The
current authentication systems are vulnerable to different voice-spoofing
attacks (e.g., replay, voice cloning, and audio deepfakes) that mimic
legitimate voices to deceive authentication systems and enable fraudulent
activities (e.g., impersonation, unauthorized access, financial fraud, etc.).
Existing solutions are often designed to tackle a single type of attack,
leading to compromised performance against unseen attacks. On the other hand,
existing unified voice anti-spoofing solutions, not designed specifically for
IoT, possess complex architectures and thus cannot be deployed on IoT-enabled
smart devices. Additionally, most of these unified solutions exhibit
significant performance issues, including higher equal error rates or lower
accuracy for specific attacks. To overcome these issues, we present the
parallel stacked aggregation network (PSA-Net), a lightweight framework
designed as an anti-spoofing defense system for voice-controlled smart IoT
devices. The PSA-Net processes raw audios directly and eliminates the need for
dataset-dependent handcrafted features or pre-computed spectrograms.
Furthermore, PSA-Net employs a split-transform-aggregate approach, which
involves the segmentation of utterances, the extraction of intrinsic
differentiable embeddings through convolutions, and the aggregation of them to
distinguish legitimate from spoofed audios. In contrast to existing deep
Resnet-oriented solutions, we incorporate cardinality as an additional
dimension in our network, which enhances the PSA-Net ability to generalize
across diverse attacks. The results show that the PSA-Net achieves more
consistent performance for different attacks that exist in current
anti-spoofing solutions.


http://arxiv.org/abs/2411.19688
SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Tasks. (2%)
Kim-Celine Kahl; Selen Erkan; Jeremias Traub; Carsten T. Lüth; Klaus Maier-Hein; Lena Maier-Hein; Paul F. Jaeger
  Vision-Language Models (VLMs) have great potential in medical tasks, like
Visual Question Answering (VQA), where they could act as interactive assistants
for both patients and clinicians. Yet their robustness to distribution shifts
on unseen data remains a critical concern for safe deployment. Evaluating such
robustness requires a controlled experimental setup that allows for systematic
insights into the model's behavior. However, we demonstrate that current setups
fail to offer sufficiently thorough evaluations, limiting their ability to
accurately assess model robustness. To address this gap, our work introduces a
novel framework, called SURE-VQA, centered around three key requirements to
overcome the current pitfalls and systematically analyze the robustness of
VLMs: 1) Since robustness on synthetic shifts does not necessarily translate to
real-world shifts, robustness should be measured on real-world shifts that are
inherent to the VQA data; 2) Traditional token-matching metrics often fail to
capture underlying semantics, necessitating the use of large language models
(LLMs) for more accurate semantic evaluation; 3) Model performance often lacks
interpretability due to missing sanity baselines, thus meaningful baselines
should be reported that allow assessing the multimodal impact on the VLM. To
demonstrate the relevance of this framework, we conduct a study on the
robustness of various fine-tuning methods across three medical datasets with
four different types of distribution shifts. Our study reveals several
important findings: 1) Sanity baselines that do not utilize image data can
perform surprisingly well; 2) We confirm LoRA as the best-performing PEFT
method; 3) No PEFT method consistently outperforms others in terms of
robustness to shifts. Code is provided at https://github.com/IML-DKFZ/sure-vqa.


http://arxiv.org/abs/2412.00341
Fusing Physics-Driven Strategies and Cross-Modal Adversarial Learning: Toward Multi-Domain Applications. (1%)
Hana Satou; Alan Mitkiy
  The convergence of cross-modal adversarial learning and physics-driven
methods represents a cutting-edge direction for tackling challenges in complex
multi-modal tasks and scientific computing. This review focuses on
systematically analyzing how these two approaches can be synergistically
integrated to enhance performance and robustness across diverse application
domains. By addressing key obstacles such as modality discrepancies, limited
data availability, and insufficient model robustness, this paper highlights the
role of physics-based optimization frameworks in facilitating efficient and
interpretable adversarial perturbation generation. The review also explores
significant advancements in cross-modal adversarial learning, including
applications in tasks such as image cross-modal retrieval (e.g., infrared and
RGB matching), scientific computing (e.g., solving partial differential
equations), and optimization under physical consistency constraints in vision
systems. By examining theoretical foundations and experimental outcomes, this
study demonstrates the potential of combining these approaches to handle
complex scenarios and improve the security of multi-modal systems. Finally, we
outline future directions, proposing a novel framework that unifies physical
principles with adversarial optimization, providing a pathway for researchers
to develop robust and adaptable cross-modal learning methods with both
theoretical and practical significance.


http://arxiv.org/abs/2412.00114
SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments. (84%)
Yue Cao; Yun Xing; Jie Zhang; Di Lin; Tianwei Zhang; Ivor Tsang; Yang Liu; Qing Guo
  Large vision-language models (LVLMs) have shown remarkable capabilities in
interpreting visual content. While existing works demonstrate these models'
vulnerability to deliberately placed adversarial texts, such texts are often
easily identifiable as anomalous. In this paper, we present the first approach
to generate scene-coherent typographic adversarial attacks that mislead
advanced LVLMs while maintaining visual naturalness through the capability of
the LLM-based agent. Our approach addresses three critical questions: what
adversarial text to generate, where to place it within the scene, and how to
integrate it seamlessly. We propose a training-free, multi-modal LLM-driven
scene-coherent typographic adversarial planning (SceneTAP) that employs a
three-stage process: scene understanding, adversarial planning, and seamless
integration. The SceneTAP utilizes chain-of-thought reasoning to comprehend the
scene, formulate effective adversarial text, strategically plan its placement,
and provide detailed instructions for natural integration within the image.
This is followed by a scene-coherent TextDiffuser that executes the attack
using a local diffusion mechanism. We extend our method to real-world scenarios
by printing and placing generated patches in physical environments,
demonstrating its practical implications. Extensive experiments show that our
scene-coherent adversarial text successfully misleads state-of-the-art LVLMs,
including ChatGPT-4o, even after capturing new images of physical setups. Our
evaluations demonstrate a significant increase in attack success rates while
maintaining visual naturalness and contextual appropriateness. This work
highlights vulnerabilities in current vision-language models to sophisticated,
scene-coherent adversarial attacks and provides insights into potential defense
mechanisms.


http://arxiv.org/abs/2411.19335
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning. (69%)
Shenghui Li; Edith C. -H. Ngai; Fanghua Ye; Thiemo Voigt
  Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a
promising paradigm for privacy-preserving and efficient adaptation of
Pre-trained Language Models (PLMs) in Federated Learning (FL) settings. It
preserves data privacy by keeping the data decentralized and training the model
on local devices, ensuring that raw data never leaves the user's device.
Moreover, the integration of PEFT methods such as LoRA significantly reduces
the number of trainable parameters compared to fine-tuning the entire model,
thereby minimizing communication costs and computational overhead. Despite its
potential, the security implications of FedPEFT remain underexplored. This
paper introduces a novel security threat to FedPEFT, termed PEFT-as-an-Attack
(PaaA), which exposes how PEFT can be exploited as an attack vector to
circumvent PLMs' safety alignment and generate harmful content in response to
malicious prompts. Our evaluation of PaaA reveals that with less than 1% of the
model's parameters set as trainable, and a small subset of clients acting
maliciously, the attack achieves an approximate 80% attack success rate using
representative PEFT methods such as LoRA. To mitigate this threat, we further
investigate potential defense strategies, including Robust Aggregation Schemes
(RASs) and Post-PEFT Safety Alignment (PPSA). However, our empirical analysis
highlights the limitations of these defenses, i.e., even the most advanced
RASs, such as DnC and ClippedClustering, struggle to defend against PaaA in
scenarios with highly heterogeneous data distributions. Similarly, while PPSA
can reduce attack success rates to below 10%, it severely degrades the model's
accuracy on the target task. Our results underscore the urgent need for more
effective defense mechanisms that simultaneously ensure security and maintain
the performance of the FedPEFT paradigm.


http://arxiv.org/abs/2411.18956
Random Sampling for Diffusion-based Adversarial Purification. (26%)
Jiancheng Zhang; Peiran Dong; Yongyong Chen; Yin-Ping Zhao; Song Guo
  Denoising Diffusion Probabilistic Models (DDPMs) have gained great attention
in adversarial purification. Current diffusion-based works focus on designing
effective condition-guided mechanisms while ignoring a fundamental problem,
i.e., the original DDPM sampling is intended for stable generation, which may
not be the optimal solution for adversarial purification. Inspired by the
stability of the Denoising Diffusion Implicit Model (DDIM), we propose an
opposite sampling scheme called random sampling. In brief, random sampling will
sample from a random noisy space during each diffusion process, while DDPM and
DDIM sampling will continuously sample from the adjacent or original noisy
space. Thus, random sampling obtains more randomness and achieves stronger
robustness against adversarial attacks. Correspondingly, we also introduce a
novel mediator conditional guidance to guarantee the consistency of the
prediction under the purified image and clean image input. To expand awareness
of guided diffusion purification, we conduct a detailed evaluation with
different sampling methods and our random sampling achieves an impressive
improvement in multiple settings. Leveraging mediator-guided random sampling,
we also establish a baseline method named DiffAP, which significantly
outperforms state-of-the-art (SOTA) approaches in performance and defensive
stability. Remarkably, under strong attack, our DiffAP even achieves a more
than 20% robustness advantage with 10$\times$ sampling acceleration.


http://arxiv.org/abs/2412.04495
Artificial intelligence and cybersecurity in banking sector: opportunities and risks. (12%)
Ana Kovacevic; Sonja D. Radenkovic; Dragana Nikolic
  The rapid advancements in artificial intelligence (AI) have presented new
opportunities for enhancing efficiency and economic competitiveness across
various industries, espcially in banking. Machine learning (ML), as a subset of
artificial intelligence, enables systems to adapt and learn from vast datasets,
revolutionizing decision-making processes, fraud detection, and customer
service automation. However, these innovations also introduce new challenges,
particularly in the realm of cybersecurity. Adversarial attacks, such as data
poisoning and evasion attacks, represent critical threats to machine learning
models, exploiting vulnerabilities to manipulate outcomes or compromise
sensitive information. Furthermore, this study highlights the dual-use nature
of AI tools, which can be used by malicious users. To address these challenges,
the paper emphasizes the importance of developing machine learning models with
key characteristics such as security, trust, resilience and robustness. These
features are essential to mitigating risks and ensuring the secure deployment
of AI technologies in banking sectors, where the protection of financial data
is paramount. The findings underscore the urgent need for enhanced
cybersecurity frameworks and continuous improvements in defensive mechanisms.
By exploring both opportunities and risks, this paper aims to guide the
responsible integration of AI in the banking sector, paving the way for
innovation while safeguarding against emerging threats.


http://arxiv.org/abs/2411.19117
Understanding and Improving Training-Free AI-Generated Image Detections with Vision Foundation Models. (11%)
Chung-Ting Tsai; Ching-Yun Ko; I-Hsin Chung; Yu-Chiang Frank Wang; Pin-Yu Chen
  The rapid advancement of generative models has introduced serious risks,
including deepfake techniques for facial synthesis and editing. Traditional
approaches rely on training classifiers and enhancing generalizability through
various feature extraction techniques. Meanwhile, training-free detection
methods address issues like limited data and overfitting by directly leveraging
statistical properties from vision foundation models to distinguish between
real and fake images. The current leading training-free approach, RIGID,
utilizes DINOv2 sensitivity to perturbations in image space for detecting fake
images, with fake image embeddings exhibiting greater sensitivity than those of
real images. This observation prompts us to investigate how detection
performance varies across model backbones, perturbation types, and datasets.
Our experiments reveal that detection performance is closely linked to model
robustness, with self-supervised (SSL) models providing more reliable
representations. While Gaussian noise effectively detects general objects, it
performs worse on facial images, whereas Gaussian blur is more effective due to
potential frequency artifacts. To further improve detection, we introduce
Contrastive Blur, which enhances performance on facial images, and MINDER
(MINimum distance DetEctoR), which addresses noise type bias, balancing
performance across domains. Beyond performance gains, our work offers valuable
insights for both the generative and detection communities, contributing to a
deeper understanding of model robustness property utilized for deepfake
detection.


http://arxiv.org/abs/2411.19075
LADDER: Multi-objective Backdoor Attack via Evolutionary Algorithm. (2%)
Dazhuang Liu; Yanqi Qiao; Rui Wang; Kaitai Liang; Georgios Smaragdakis
  Current black-box backdoor attacks in convolutional neural networks formulate
attack objective(s) as single-objective optimization problems in single domain.
Designing triggers in single domain harms semantics and trigger robustness as
well as introduces visual and spectral anomaly. This work proposes a
multi-objective black-box backdoor attack in dual domains via evolutionary
algorithm (LADDER), the first instance of achieving multiple attack objectives
simultaneously by optimizing triggers without requiring prior knowledge about
victim model. In particular, we formulate LADDER as a multi-objective
optimization problem (MOP) and solve it via multi-objective evolutionary
algorithm (MOEA). MOEA maintains a population of triggers with trade-offs among
attack objectives and uses non-dominated sort to drive triggers toward optimal
solutions. We further apply preference-based selection to MOEA to exclude
impractical triggers. We state that LADDER investigates a new dual-domain
perspective for trigger stealthiness by minimizing the anomaly between clean
and poisoned samples in the spectral domain. Lastly, the robustness against
preprocessing operations is achieved by pushing triggers to low-frequency
regions. Extensive experiments comprehensively showcase that LADDER achieves
attack effectiveness of at least 99%, attack robustness with 90.23% (50.09%
higher than state-of-the-art attacks on average), superior natural stealthiness
(1.12x to 196.74x improvement) and excellent spectral stealthiness (8.45x
enhancement) as compared to current stealthy attacks by the average $l_2$-norm
across 5 public datasets.


http://arxiv.org/abs/2411.19027
Enhancing Neural Network Robustness Against Fault Injection Through Non-linear Weight Transformations. (2%)
Ninnart Fuengfusin; Hakaru Tamukoh
  Deploying deep neural networks (DNNs) in real-world environments poses
challenges due to faults that can manifest in physical hardware from radiation,
aging, and temperature fluctuations. To address this, previous works have
focused on protecting DNNs via activation range restriction using clipped ReLU
and finding the optimal clipping threshold. However, this work instead focuses
on constraining DNN weights by applying saturated activation functions (SAFs):
Tanh, Arctan, and others. SAFs prevent faults from causing DNN weights to
become excessively large, which can lead to model failure. These methods not
only enhance the robustness of DNNs against fault injections but also improve
DNN performance by a small margin. Before deployment, DNNs are trained with
weights constrained by SAFs. During deployment, the weights without applied SAF
are written to mediums with faults. When read, weights with faults are applied
with SAFs and are used for inference. We demonstrate our proposed method across
three datasets (CIFAR10, CIFAR100, ImageNet 2012) and across three datatypes
(32-bit floating point (FP32), 16-bit floating point, and 8-bit fixed point).
We show that our method enables FP32 ResNet18 with ImageNet 2012 to operate at
a bit-error rate of 0.00001 with minor accuracy loss, while without the
proposed method, the FP32 DNN only produces random guesses. Furthermore, to
accelerate the training process, we demonstrate that an ImageNet 2012
pre-trained ResNet18 can be adapted to SAF by training for a few epochs with a
slight improvement in Top-1 accuracy while still ensuring robustness against
fault injection.


http://arxiv.org/abs/2411.18275
Visual Adversarial Attack on Vision-Language Models for Autonomous Driving. (99%)
Tianyuan Zhang; Lu Wang; Xinwei Zhang; Yitong Zhang; Boyi Jia; Siyuan Liang; Shengshan Hu; Qiang Fu; Aishan Liu; Xianglong Liu
  Vision-language models (VLMs) have significantly advanced autonomous driving
(AD) by enhancing reasoning capabilities. However, these models remain highly
vulnerable to adversarial attacks. While existing research has primarily
focused on general VLM attacks, the development of attacks tailored to the
safety-critical AD context has been largely overlooked. In this paper, we take
the first step toward designing adversarial attacks specifically targeting VLMs
in AD, exposing the substantial risks these attacks pose within this critical
domain. We identify two unique challenges for effective adversarial attacks on
AD VLMs: the variability of textual instructions and the time-series nature of
visual scenarios. To this end, we propose ADvLM, the first visual adversarial
attack framework specifically designed for VLMs in AD. Our framework introduces
Semantic-Invariant Induction, which uses a large language model to create a
diverse prompt library of textual instructions with consistent semantic
content, guided by semantic entropy. Building on this, we introduce
Scenario-Associated Enhancement, an approach where attention mechanisms select
key frames and perspectives within driving scenarios to optimize adversarial
perturbations that generalize across the entire scenario. Extensive experiments
on several AD VLMs over multiple benchmarks show that ADvLM achieves
state-of-the-art attack effectiveness. Moreover, real-world attack studies
further validate its applicability and potential in practice.


http://arxiv.org/abs/2411.18776
Fall Leaf Adversarial Attack on Traffic Sign Classification. (99%)
Anthony Etim; Jakub Szefer
  Adversarial input image perturbation attacks have emerged as a significant
threat to machine learning algorithms, particularly in image classification
setting. These attacks involve subtle perturbations to input images that cause
neural networks to misclassify the input images, even though the images remain
easily recognizable to humans. One critical area where adversarial attacks have
been demonstrated is in automotive systems where traffic sign classification
and recognition is critical, and where misclassified images can cause
autonomous systems to take wrong actions. This work presents a new class of
adversarial attacks. Unlike existing work that has focused on adversarial
perturbations that leverage human-made artifacts to cause the perturbations,
such as adding stickers, paint, or shining flashlights at traffic signs, this
work leverages nature-made artifacts: tree leaves. By leveraging nature-made
artifacts, the new class of attacks has plausible deniability: a fall leaf
stuck to a street sign could come from a near-by tree, rather than be placed
there by an malicious human attacker. To evaluate the new class of the
adversarial input image perturbation attacks, this work analyses how fall
leaves can cause misclassification in street signs. The work evaluates various
leaves from different species of trees, and considers various parameters such
as size, color due to tree leaf type, and rotation. The work demonstrates high
success rate for misclassification. The work also explores the correlation
between successful attacks and how they affect the edge detection, which is
critical in many image classification algorithms.


http://arxiv.org/abs/2411.18688
Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment. (67%)
Soumya Suvra Ghosal; Souradip Chakraborty; Vaibhav Singh; Tianrui Guan; Mengdi Wang; Ahmad Beirami; Furong Huang; Alvaro Velasquez; Dinesh Manocha; Amrit Singh Bedi
  With the widespread deployment of Multimodal Large Language Models (MLLMs)
for visual-reasoning tasks, improving their safety has become crucial. Recent
research indicates that despite training-time safety alignment, these models
remain vulnerable to jailbreak attacks. In this work, we first highlight an
important safety gap to describe that alignment achieved solely through safety
training may be insufficient against jailbreak attacks. To address this
vulnerability, we propose Immune, an inference-time defense framework that
leverages a safe reward model through controlled decoding to defend against
jailbreak attacks. Additionally, we provide a mathematical characterization of
Immune, offering provable guarantees against jailbreaks. Extensive evaluations
on diverse jailbreak benchmarks using recent MLLMs reveal that Immune
effectively enhances model safety while preserving the model's original
capabilities. For instance, against text-based jailbreak attacks on LLaVA-1.6,
Immune reduces the attack success rate by 57.82% and 16.78% compared to the
base MLLM and state-of-the-art defense strategy, respectively.


http://arxiv.org/abs/2411.18280
Neutralizing Backdoors through Information Conflicts for Large Language Models. (26%)
Chen Chen; Yuchen Sun; Xueluan Gong; Jiaxin Gao; Kwok-Yan Lam
  Large language models (LLMs) have seen significant advancements, achieving
superior performance in various Natural Language Processing (NLP) tasks, from
understanding to reasoning. However, they remain vulnerable to backdoor
attacks, where models behave normally for standard queries but generate harmful
responses or unintended output when specific triggers are activated. Existing
backdoor defenses often suffer from drawbacks that they either focus on
detection without removal, rely on rigid assumptions about trigger properties,
or prove to be ineffective against advanced attacks like multi-trigger
backdoors. In this paper, we present a novel method to eliminate backdoor
behaviors from LLMs through the construction of information conflicts using
both internal and external mechanisms. Internally, we leverage a lightweight
dataset to train a conflict model, which is then merged with the backdoored
model to neutralize malicious behaviors by embedding contradictory information
within the model's parametric memory. Externally, we incorporate convincing
contradictory evidence into the prompt to challenge the model's internal
backdoor knowledge. Experimental results on classification and conversational
tasks across 4 widely used LLMs demonstrate that our method outperforms 8
state-of-the-art backdoor defense baselines. We can reduce the attack success
rate of advanced backdoor attacks by up to 98% while maintaining over 90% clean
data accuracy. Furthermore, our method has proven to be robust against adaptive
backdoor attacks. The code will be open-sourced upon publication.


http://arxiv.org/abs/2411.18269
Hidden Data Privacy Breaches in Federated Learning. (22%)
Xueluan Gong; Yuji Wang; Shuaike Li; Mengyuan Sun; Songze Li; Qian Wang; Kwok-Yan Lam; Chen Chen
  Federated Learning (FL) emerged as a paradigm for conducting machine learning
across broad and decentralized datasets, promising enhanced privacy by
obviating the need for direct data sharing. However, recent studies show that
attackers can steal private data through model manipulation or gradient
analysis. Existing attacks are constrained by low theft quantity or
low-resolution data, and they are often detected through anomaly monitoring in
gradients or weights. In this paper, we propose a novel data-reconstruction
attack leveraging malicious code injection, supported by two key techniques,
i.e., distinctive and sparse encoding design and block partitioning. Unlike
conventional methods that require detectable changes to the model, our method
stealthily embeds a hidden model using parameter sharing to systematically
extract sensitive data. The Fibonacci-based index design ensures efficient,
structured retrieval of memorized data, while the block partitioning method
enhances our method's capability to handle high-resolution images by dividing
them into smaller, manageable units. Extensive experiments on 4 datasets
confirmed that our method is superior to the five state-of-the-art
data-reconstruction attacks under the five respective detection methods. Our
method can handle large-scale and high-resolution data without being detected
or mitigated by state-of-the-art data reconstruction defense methods. In
contrast to baselines, our method can be directly applied to both FedAVG and
FedSGD scenarios, underscoring the need for developers to devise new defenses
against such vulnerabilities. We will open-source our code upon acceptance.


http://arxiv.org/abs/2411.18479
SoK: Watermarking for AI-Generated Content. (3%)
Xuandong Zhao; Sam Gunn; Miranda Christ; Jaiden Fairoze; Andres Fabrega; Nicholas Carlini; Sanjam Garg; Sanghyun Hong; Milad Nasr; Florian Tramer; Somesh Jha; Lei Li; Yu-Xiang Wang; Dawn Song
  As the outputs of generative AI (GenAI) techniques improve in quality, it
becomes increasingly challenging to distinguish them from human-created
content. Watermarking schemes are a promising approach to address the problem
of distinguishing between AI and human-generated content. These schemes embed
hidden signals within AI-generated content to enable reliable detection. While
watermarking is not a silver bullet for addressing all risks associated with
GenAI, it can play a crucial role in enhancing AI safety and trustworthiness by
combating misinformation and deception. This paper presents a comprehensive
overview of watermarking techniques for GenAI, beginning with the need for
watermarking from historical and regulatory perspectives. We formalize the
definitions and desired properties of watermarking schemes and examine the key
objectives and threat models for existing approaches. Practical evaluation
strategies are also explored, providing insights into the development of robust
watermarking techniques capable of resisting various attacks. Additionally, we
review recent representative works, highlight open challenges, and discuss
potential directions for this emerging field. By offering a thorough
understanding of watermarking in GenAI, this work aims to guide researchers in
advancing watermarking methods and applications, and support policymakers in
addressing the broader implications of GenAI.


http://arxiv.org/abs/2411.18207
From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects. (1%)
Zizhao Li; Zhengkang Xiang; Joseph West; Kourosh Khoshelham
  Traditional object detection methods operate under the closed-set assumption,
where models can only detect a fixed number of objects predefined in the
training set. Recent works on open vocabulary object detection (OVD) enable the
detection of objects defined by an unbounded vocabulary, which reduces the cost
of training models for specific tasks. However, OVD heavily relies on accurate
prompts provided by an ''oracle'', which limits their use in critical
applications such as driving scene perception. OVD models tend to misclassify
near-out-of-distribution (NOOD) objects that have similar semantics to known
classes, and ignore far-out-of-distribution (FOOD) objects. To address theses
limitations, we propose a framework that enables OVD models to operate in open
world settings, by identifying and incrementally learning novel objects. To
detect FOOD objects, we propose Open World Embedding Learning (OWEL) and
introduce the concept of Pseudo Unknown Embedding which infers the location of
unknown classes in a continuous semantic space based on the information of
known classes. We also propose Multi-Scale Contrastive Anchor Learning (MSCAL),
which enables the identification of misclassified unknown objects by promoting
the intra-class consistency of object embeddings at different scales. The
proposed method achieves state-of-the-art performance in common open world
object detection and autonomous driving benchmarks.


http://arxiv.org/abs/2411.17959
Adversarial Training in Low-Label Regimes with Margin-Based Interpolation. (99%)
Tian Ye; Rajgopal Kannan; Viktor Prasanna
  Adversarial training has emerged as an effective approach to train robust
neural network models that are resistant to adversarial attacks, even in
low-label regimes where labeled data is scarce. In this paper, we introduce a
novel semi-supervised adversarial training approach that enhances both
robustness and natural accuracy by generating effective adversarial examples.
Our method begins by applying linear interpolation between clean and
adversarial examples to create interpolated adversarial examples that cross
decision boundaries by a controlled margin. This sample-aware strategy tailors
adversarial examples to the characteristics of each data point, enabling the
model to learn from the most informative perturbations. Additionally, we
propose a global epsilon scheduling strategy that progressively adjusts the
upper bound of perturbation strengths during training. The combination of these
strategies allows the model to develop increasingly complex decision boundaries
with better robustness and natural accuracy. Empirical evaluations show that
our approach effectively enhances performance against various adversarial
attacks, such as PGD and AutoAttack.


http://arxiv.org/abs/2411.17283
BadScan: An Architectural Backdoor Attack on Visual State Space Models. (98%)
Om Suhas Deshmukh; Sankalp Nagaonkar; Achyut Mani Tripathi; Ashish Mishra
  The newly introduced Visual State Space Model (VMamba), which employs
\textit{State Space Mechanisms} (SSM) to interpret images as sequences of
patches, has shown exceptional performance compared to Vision Transformers
(ViT) across various computer vision tasks. However, recent studies have
highlighted that deep models are susceptible to adversarial attacks. One common
approach is to embed a trigger in the training data to retrain the model,
causing it to misclassify data samples into a target class, a phenomenon known
as a backdoor attack. In this paper, we first evaluate the robustness of the
VMamba model against existing backdoor attacks. Based on this evaluation, we
introduce a novel architectural backdoor attack, termed BadScan, designed to
deceive the VMamba model. This attack utilizes bit plane slicing to create
visually imperceptible backdoored images. During testing, if a trigger is
detected by performing XOR operations between the $k^{th}$ bit planes of the
modified triggered patches, the traditional 2D selective scan (SS2D) mechanism
in the visual state space (VSS) block of VMamba is replaced with our newly
designed BadScan block, which incorporates four newly developed scanning
patterns. We demonstrate that the BadScan backdoor attack represents a
significant threat to visual state space models and remains effective even
after complete retraining from scratch. Experimental results on two widely used
image classification datasets, CIFAR-10, and ImageNet-1K, reveal that while
visual state space models generally exhibit robustness against current backdoor
attacks, the BadScan attack is particularly effective, achieving a higher
Triggered Accuracy Ratio (TAR) in misleading the VMamba model and its variants.


http://arxiv.org/abs/2411.17936
Stealthy Multi-Task Adversarial Attacks. (92%)
Jiacheng Guo; Tianyun Zhang; Lei Li; Haochen Yang; Hongkai Yu; Minghai Qin
  Deep Neural Networks exhibit inherent vulnerabilities to adversarial attacks,
which can significantly compromise their outputs and reliability. While
existing research primarily focuses on attacking single-task scenarios or
indiscriminately targeting all tasks in multi-task environments, we investigate
selectively targeting one task while preserving performance in others within a
multi-task framework. This approach is motivated by varying security priorities
among tasks in real-world applications, such as autonomous driving, where
misinterpreting critical objects (e.g., signs, traffic lights) poses a greater
security risk than minor depth miscalculations. Consequently, attackers may
hope to target security-sensitive tasks while avoiding non-critical tasks from
being compromised, thus evading being detected before compromising crucial
functions. In this paper, we propose a method for the stealthy multi-task
attack framework that utilizes multiple algorithms to inject imperceptible
noise into the input. This novel method demonstrates remarkable efficacy in
compromising the target task while simultaneously maintaining or even enhancing
performance across non-targeted tasks - a criterion hitherto unexplored in the
field. Additionally, we introduce an automated approach for searching the
weighting factors in the loss function, further enhancing attack efficiency.
Experimental results validate our framework's ability to successfully attack
the target task while preserving the performance of non-targeted tasks. The
automated loss function weight searching method demonstrates comparable
efficacy to manual tuning, establishing a state-of-the-art multi-task attack
framework.


http://arxiv.org/abs/2411.17468
Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers. (82%)
Fatemeh Nourilenjan Nokabadi; Jean-Francois Lalonde; Christian Gagné
  Adversarial perturbations aim to deceive neural networks into predicting
inaccurate results. For visual object trackers, adversarial attacks have been
developed to generate perturbations by manipulating the outputs. However,
transformer trackers predict a specific bounding box instead of an object
candidate list, which limits the applicability of many existing attack
scenarios. To address this issue, we present a novel white-box approach to
attack visual object trackers with transformer backbones using only one
bounding box. From the tracker predicted bounding box, we generate a list of
adversarial bounding boxes and compute the adversarial loss for those bounding
boxes. Experimental results demonstrate that our simple yet effective attack
outperforms existing attacks against several robust transformer trackers,
including TransT-M, ROMTrack, and MixFormer, on popular benchmark tracking
datasets such as GOT-10k, UAV123, and VOT2022STS.


http://arxiv.org/abs/2411.18648
MADE: Graph Backdoor Defense with Masked Unlearning. (82%)
Xiao Lin; Mingjie Li; Yisen Wang
  Graph Neural Networks (GNNs) have garnered significant attention from
researchers due to their outstanding performance in handling graph-related
tasks, such as social network analysis, protein design, and so on. Despite
their widespread application, recent research has demonstrated that GNNs are
vulnerable to backdoor attacks, implemented by injecting triggers into the
training datasets. Trained on the poisoned data, GNNs will predict target
labels when attaching trigger patterns to inputs. This vulnerability poses
significant security risks for applications of GNNs in sensitive domains, such
as drug discovery. While there has been extensive research into backdoor
defenses for images, strategies to safeguard GNNs against such attacks remain
underdeveloped. Furthermore, we point out that conventional backdoor defense
methods designed for images cannot work well when directly implemented on graph
data. In this paper, we first analyze the key difference between image backdoor
and graph backdoor attacks. Then we tackle the graph defense problem by
presenting a novel approach called MADE, which devises an adversarial mask
generation mechanism that selectively preserves clean sub-graphs and further
leverages masks on edge weights to eliminate the influence of triggers
effectively. Extensive experiments across various graph classification tasks
demonstrate the effectiveness of MADE in significantly reducing the attack
success rate (ASR) while maintaining a high classification accuracy.


http://arxiv.org/abs/2411.18000
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models. (75%)
Shuyang Hao; Bryan Hooi; Jun Liu; Kai-Wei Chang; Zi Huang; Yujun Cai
  Despite inheriting security measures from underlying language models,
Vision-Language Models (VLMs) may still be vulnerable to safety alignment
issues. Through empirical analysis, we uncover two critical findings:
scenario-matched images can significantly amplify harmful outputs, and contrary
to common assumptions in gradient-based attacks, minimal loss values do not
guarantee optimal attack effectiveness. Building on these insights, we
introduce MLAI (Multi-Loss Adversarial Images), a novel jailbreak framework
that leverages scenario-aware image generation for semantic alignment, exploits
flat minima theory for robust adversarial image selection, and employs
multi-image collaborative attacks for enhanced effectiveness. Extensive
experiments demonstrate MLAI's significant impact, achieving attack success
rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2, substantially outperforming
existing methods by margins of 34.37% and 12.77% respectively. Furthermore,
MLAI shows considerable transferability to commercial black-box VLMs, achieving
up to 60.11% success rate. Our work reveals fundamental visual vulnerabilities
in current VLMs safety mechanisms and underscores the need for stronger
defenses. Warning: This paper contains potentially harmful example text.


http://arxiv.org/abs/2411.18027
Privacy-preserving Robotic-based Multi-factor Authentication Scheme for Secure Automated Delivery System. (9%)
Yang Yang; Aryan Mohammadi Pasikhani; Prosanta Gope; Biplab Sikdar
  Package delivery is a critical aspect of various industries, but it often
incurs high financial costs and inefficiencies when relying solely on human
resources. The last-mile transport problem, in particular, contributes
significantly to the expenditure of human resources in major companies.
Robot-based delivery systems have emerged as a potential solution for last-mile
delivery to address this challenge. However, robotic delivery systems still
face security and privacy issues, like impersonation, replay, man-in-the-middle
attacks (MITM), unlinkability, and identity theft. In this context, we propose
a privacy-preserving multi-factor authentication scheme specifically designed
for robot delivery systems. Additionally, AI-assisted robotic delivery systems
are susceptible to machine learning-based attacks (e.g. FGSM, PGD, etc.). We
introduce the \emph{first} transformer-based audio-visual fusion defender to
tackle this issue, which effectively provides resilience against adversarial
samples. Furthermore, we provide a rigorous formal analysis of the proposed
protocol and also analyse the protocol security using a popular symbolic proof
tool called ProVerif and Scyther. Finally, we present a real-world
implementation of the proposed robotic system with the computation cost and
energy consumption analysis. Code and pre-trained models are available at:
https://drive.google.com/drive/folders/18B2YbxtV0Pyj5RSFX-ZzCGtFOyorBHil


http://arxiv.org/abs/2411.17453
PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning. (2%)
Zhen Sun; Tianshuo Cong; Yule Liu; Chenhao Lin; Xinlei He; Rongmao Chen; Xingshuo Han; Xinyi Huang
  Fine-tuning is an essential process to improve the performance of Large
Language Models (LLMs) in specific domains, with Parameter-Efficient
Fine-Tuning (PEFT) gaining popularity due to its capacity to reduce
computational demands through the integration of low-rank adapters. These
lightweight adapters, such as LoRA, can be shared and utilized on open-source
platforms. However, adversaries could exploit this mechanism to inject
backdoors into these adapters, resulting in malicious behaviors like incorrect
or harmful outputs, which pose serious security risks to the community.
Unfortunately, few of the current efforts concentrate on analyzing the backdoor
patterns or detecting the backdoors in the adapters.
  To fill this gap, we first construct (and will release) PADBench, a
comprehensive benchmark that contains 13,300 benign and backdoored adapters
fine-tuned with various datasets, attack strategies, PEFT methods, and LLMs.
Moreover, we propose PEFTGuard, the first backdoor detection framework against
PEFT-based adapters. Extensive evaluation upon PADBench shows that PEFTGuard
outperforms existing detection methods, achieving nearly perfect detection
accuracy (100%) in most cases. Notably, PEFTGuard exhibits zero-shot
transferability on three aspects, including different attacks, PEFT methods,
and adapter ranks. In addition, we consider various adaptive attacks to
demonstrate the high robustness of PEFTGuard. We further explore several
possible backdoor mitigation defenses, finding fine-mixing to be the most
effective method. We envision our benchmark and method can shed light on future
LLM backdoor detection research.


http://arxiv.org/abs/2411.18028
Improved Parallel Derandomization via Finite Automata with Applications. (1%)
Jeff Giliberti; David G. Harris
  One main genre of algorithmic derandomization comes from the construction of
probability distributions with small support that fool a randomized algorithm.
This is especially well-suited to parallelization, i.e. NC algorithms. A
significant abstraction of these methods can be formulated in terms of fooling
polynomial-space statistical tests computed via finite automata (Sivakumar
2002); this encompasses $k$-wise independence, sums of random variables, and
many other properties.
  We describe new parallel algorithms to fool general finite-state automata
with significantly reduced processor complexity. The analysis is also
simplified because we can cleanly separate the problem-specific optimizations
from the general lattice discrepancy problems at the core of the
automaton-fooling construction. We illustrate with improved applications to the
Gale-Berlekamp Switching Game and to approximate MAX-CUT via SDP rounding.


http://arxiv.org/abs/2411.17585
Multi-Objective Reinforcement Learning for Automated Resilient Cyber Defence. (1%)
Ross O'Driscoll; Claudia Hagen; Joe Bater; James M. Adams
  Cyber-attacks pose a security threat to military command and control
networks, Intelligence, Surveillance, and Reconnaissance (ISR) systems, and
civilian critical national infrastructure. The use of artificial intelligence
and autonomous agents in these attacks increases the scale, range, and
complexity of this threat and the subsequent disruption they cause. Autonomous
Cyber Defence (ACD) agents aim to mitigate this threat by responding at machine
speed and at the scale required to address the problem. Sequential
decision-making algorithms such as Deep Reinforcement Learning (RL) provide a
promising route to create ACD agents. These algorithms focus on a single
objective such as minimizing the intrusion of red agents on the network, by
using a handcrafted weighted sum of rewards. This approach removes the ability
to adapt the model during inference, and fails to address the many competing
objectives present when operating and protecting these networks. Conflicting
objectives, such as restoring a machine from a back-up image, must be carefully
balanced with the cost of associated down-time, or the disruption to network
traffic or services that might result. Instead of pursing a Single-Objective RL
(SORL) approach, here we present a simple example of a multi-objective network
defence game that requires consideration of both defending the network against
red-agents and maintaining critical functionality of green-agents. Two
Multi-Objective Reinforcement Learning (MORL) algorithms, namely
Multi-Objective Proximal Policy Optimization (MOPPO), and Pareto-Conditioned
Networks (PCN), are used to create two trained ACD agents whose performance is
compared on our Multi-Objective Cyber Defence game. The benefits and
limitations of MORL ACD agents in comparison to SORL ACD agents are discussed
based on the investigations of this game.


http://arxiv.org/abs/2411.17911
Passive Deepfake Detection Across Multi-modalities: A Comprehensive Survey. (1%)
Hong-Hanh Nguyen-Le; Van-Tuan Tran; Dinh-Thuc Nguyen; Nhien-An Le-Khac
  In recent years, deepfakes (DFs) have been utilized for malicious purposes,
such as individual impersonation, misinformation spreading, and artists style
imitation, raising questions about ethical and security concerns. In this
survey, we provide a comprehensive review and comparison of passive DF
detection across multiple modalities, including image, video, audio, and
multi-modal, to explore the inter-modality relationships between them. Beyond
detection accuracy, we extend our analysis to encompass crucial performance
dimensions essential for real-world deployment: generalization capabilities
across novel generation techniques, robustness against adversarial
manipulations and postprocessing techniques, attribution precision in
identifying generation sources, and resilience under real-world operational
conditions. Additionally, we analyze the advantages and limitations of existing
datasets, benchmarks, and evaluation metrics for passive DF detection. Finally,
we propose future research directions that address these unexplored and
emerging issues in the field of passive DF detection. This survey offers
researchers and practitioners a comprehensive resource for understanding the
current landscape, methodological approaches, and promising future directions
in this rapidly evolving field.


http://arxiv.org/abs/2411.16622
Imperceptible Adversarial Examples in the Physical World. (99%)
Weilin Xu; Sebastian Szyller; Cory Cornelius; Luis Murillo Rojas; Marius Arvinte; Alvaro Velasquez; Jason Martin; Nageen Himayat
  Adversarial examples in the digital domain against deep learning-based
computer vision models allow for perturbations that are imperceptible to human
eyes. However, producing similar adversarial examples in the physical world has
been difficult due to the non-differentiable image distortion functions in
visual sensing systems. The existing algorithms for generating physically
realizable adversarial examples often loosen their definition of adversarial
examples by allowing unbounded perturbations, resulting in obvious or even
strange visual patterns. In this work, we make adversarial examples
imperceptible in the physical world using a straight-through estimator (STE,
a.k.a. BPDA). We employ STE to overcome the non-differentiability -- applying
exact, non-differentiable distortions in the forward pass of the
backpropagation step, and using the identity function in the backward pass. Our
differentiable rendering extension to STE also enables imperceptible
adversarial patches in the physical world. Using printout photos, and
experiments in the CARLA simulator, we show that STE enables fast generation of
$\ell_\infty$ bounded adversarial examples despite the non-differentiable
distortions. To the best of our knowledge, this is the first work demonstrating
imperceptible adversarial examples bounded by small $\ell_\infty$ norms in the
physical world that force zero classification accuracy in the global
perturbation threat model and cause near-zero ($4.22\%$) AP50 in object
detection in the patch perturbation threat model. We urge the community to
re-evaluate the threat of adversarial examples in the physical world.


http://arxiv.org/abs/2411.16598
DiffBreak: Is Diffusion-Based Purification Robust? (99%)
Andre Kassis; Urs Hengartner; Yaoliang Yu
  Diffusion-based purification (DBP) has become a cornerstone defense against
adversarial examples (AEs), regarded as robust due to its use of diffusion
models (DMs) that project AEs onto the natural data manifold. We refute this
core claim, theoretically proving that gradient-based attacks effectively
target the DM rather than the classifier, causing DBP's outputs to align with
adversarial distributions. This prompts a reassessment of DBP's robustness,
attributing it to two critical flaws: incorrect gradients and inappropriate
evaluation protocols that test only a single random purification of the AE. We
show that with proper accounting for stochasticity and resubmission risk, DBP
collapses. To support this, we introduce DiffBreak, the first reliable toolkit
for differentiation through DBP, eliminating gradient flaws that previously
further inflated robustness estimates. We also analyze the current defense
scheme used for DBP where classification relies on a single purification,
pinpointing its inherent invalidity. We provide a statistically grounded
majority-vote (MV) alternative that aggregates predictions across multiple
purified copies, showing partial but meaningful robustness gain. We then
propose a novel adaptation of an optimization method against deepfake
watermarking, crafting systemic perturbations that defeat DBP even under MV,
challenging DBP's viability.


http://arxiv.org/abs/2411.16782
Scaling Laws for Black box Adversarial Attacks. (99%)
Chuan Liu; Huanran Chen; Yichi Zhang; Yinpeng Dong; Jun Zhu
  Adversarial examples usually exhibit good cross-model transferability,
enabling attacks on black-box models with limited information about their
architectures and parameters, which are highly threatening in commercial
black-box scenarios. Model ensembling is an effective strategy to improve the
transferability of adversarial examples by attacking multiple surrogate models.
However, since prior studies usually adopt few models in the ensemble, there
remains an open question of whether scaling the number of models can further
improve black-box attacks. Inspired by the scaling law of large foundation
models, we investigate the scaling laws of black-box adversarial attacks in
this work. Through theoretical analysis and empirical evaluations, we conclude
with clear scaling laws that using more surrogate models enhances adversarial
transferability. Comprehensive experiments verify the claims on standard image
classifiers, diverse defended models and multimodal large language models using
various adversarial attack methods. Specifically, by scaling law, we achieve
90%+ transfer attack success rate on even proprietary models like GPT-4o.
Further visualization indicates that there is also a scaling law on the
interpretability and semantics of adversarial perturbations.


http://arxiv.org/abs/2411.16437
Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack. (81%)
Xide Xu; Muhammad Atif Butt; Sandesh Kamath; Bogdan Raducanu
  The growing demand for customized visual content has led to the rise of
personalized text-to-image (T2I) diffusion models. Despite their remarkable
potential, they pose significant privacy risk when misused for malicious
purposes. In this paper, we propose a novel and efficient adversarial attack
method, Concept Protection by Selective Attention Manipulation (CoPSAM) which
targets only the cross-attention layers of a T2I diffusion model. For this
purpose, we carefully construct an imperceptible noise to be added to clean
samples to get their adversarial counterparts. This is obtained during the
fine-tuning process by maximizing the discrepancy between the corresponding
cross-attention maps of the user-specific token and the class-specific token,
respectively. Experimental validation on a subset of CelebA-HQ face images
dataset demonstrates that our approach outperforms existing methods. Besides
this, our method presents two important advantages derived from the qualitative
evaluation: (i) we obtain better protection results for lower noise levels than
our competitors; and (ii) we protect the content from unauthorized use thereby
protecting the individual's identity from potential misuse.


http://arxiv.org/abs/2411.17746
UVCG: Leveraging Temporal Consistency for Universal Video Protection. (54%)
KaiZhou Li; Jindong Gu; Xinchun Yu; Junjie Cao; Yansong Tang; Xiao-Ping Zhang
  The security risks of AI-driven video editing have garnered significant
attention. Although recent studies indicate that adding perturbations to images
can protect them from malicious edits, directly applying image-based methods to
perturb each frame in a video becomes ineffective, as video editing techniques
leverage the consistency of inter-frame information to restore individually
perturbed content. To address this challenge, we leverage the temporal
consistency of video content to propose a straightforward and efficient, yet
highly effective and broadly applicable approach, Universal Video Consistency
Guard (UVCG). UVCG embeds the content of another video(target video) within a
protected video by introducing continuous, imperceptible perturbations which
has the ability to force the encoder of editing models to map continuous inputs
to misaligned continuous outputs, thereby inhibiting the generation of videos
consistent with the intended textual prompts. Additionally leveraging
similarity in perturbations between adjacent frames, we improve the
computational efficiency of perturbation generation by employing a
perturbation-reuse strategy. We applied UVCG across various versions of Latent
Diffusion Models (LDM) and assessed its effectiveness and generalizability
across multiple LDM-based editing pipelines. The results confirm the
effectiveness, transferability, and efficiency of our approach in safeguarding
video content from unauthorized modifications.


http://arxiv.org/abs/2411.16512
Guarding the Gate: ConceptGuard Battles Concept-Level Backdoors in Concept Bottleneck Models. (50%)
Songning Lai; Yu Huang; Jiayu Yang; Gaoxiang Huang; Wenshuo Chen; Yutao Yue
  The increasing complexity of AI models, especially in deep learning, has
raised concerns about transparency and accountability, particularly in
high-stakes applications like medical diagnostics, where opaque models can
undermine trust. Explainable Artificial Intelligence (XAI) aims to address
these issues by providing clear, interpretable models. Among XAI techniques,
Concept Bottleneck Models (CBMs) enhance transparency by using high-level
semantic concepts. However, CBMs are vulnerable to concept-level backdoor
attacks, which inject hidden triggers into these concepts, leading to
undetectable anomalous behavior. To address this critical security gap, we
introduce ConceptGuard, a novel defense framework specifically designed to
protect CBMs from concept-level backdoor attacks. ConceptGuard employs a
multi-stage approach, including concept clustering based on text distance
measurements and a voting mechanism among classifiers trained on different
concept subgroups, to isolate and mitigate potential triggers. Our
contributions are threefold: (i) we present ConceptGuard as the first defense
mechanism tailored for concept-level backdoor attacks in CBMs; (ii) we provide
theoretical guarantees that ConceptGuard can effectively defend against such
attacks within a certain trigger size threshold, ensuring robustness; and (iii)
we demonstrate that ConceptGuard maintains the high performance and
interpretability of CBMs, crucial for trustworthiness. Through comprehensive
experiments and theoretical proofs, we show that ConceptGuard significantly
enhances the security and trustworthiness of CBMs, paving the way for their
secure deployment in critical applications.


http://arxiv.org/abs/2411.16832
Edit Away and My Face Will not Stay: Personal Biometric Defense against Malicious Generative Editing. (50%)
Hanhui Wang; Yihua Zhang; Ruizheng Bai; Yue Zhao; Sijia Liu; Zhengzhong Tu
  Recent advancements in diffusion models have made generative image editing
more accessible, enabling creative edits but raising ethical concerns,
particularly regarding malicious edits to human portraits that threaten privacy
and identity security. Existing protection methods primarily rely on
adversarial perturbations to nullify edits but often fail against diverse
editing requests. We propose FaceLock, a novel approach to portrait protection
that optimizes adversarial perturbations to destroy or significantly alter
biometric information, rendering edited outputs biometrically unrecognizable.
FaceLock integrates facial recognition and visual perception into perturbation
optimization to provide robust protection against various editing attempts. We
also highlight flaws in commonly used evaluation metrics and reveal how they
can be manipulated, emphasizing the need for reliable assessments of
protection. Experiments show FaceLock outperforms baselines in defending
against malicious edits and is robust against purification techniques. Ablation
studies confirm its stability and broad applicability across diffusion-based
editing algorithms. Our work advances biometric defense and sets the foundation
for privacy-preserving practices in image editing. The code is available at:
https://github.com/taco-group/FaceLock.


http://arxiv.org/abs/2411.16162
Sparse patches adversarial attacks via extrapolating point-wise information. (47%)
Yaniv Nemcovsky; Avi Mendelson; Chaim Baskin
  Sparse and patch adversarial attacks were previously shown to be applicable
in realistic settings and are considered a security risk to autonomous systems.
Sparse adversarial perturbations constitute a setting in which the adversarial
perturbations are limited to affecting a relatively small number of points in
the input. Patch adversarial attacks denote the setting where the sparse
attacks are limited to a given structure, i.e., sparse patches with a given
shape and number. However, previous patch adversarial attacks do not
simultaneously optimize multiple patches' locations and perturbations. This
work suggests a novel approach for sparse patches adversarial attacks via
point-wise trimming dense adversarial perturbations. Our approach enables
simultaneous optimization of multiple sparse patches' locations and
perturbations for any given number and shape. Moreover, our approach is also
applicable for standard sparse adversarial attacks, where we show that it
significantly improves the state-of-the-art over multiple extensive settings. A
reference implementation of the proposed method and the reported experiments is
provided at \url{https://github.com/yanemcovsky/SparsePatches.git}


http://arxiv.org/abs/2411.17026
RED: Robust Environmental Design. (10%)
Jinghan Yang
  The classification of road signs by autonomous systems, especially those
reliant on visual inputs, is highly susceptible to adversarial attacks.
Traditional approaches to mitigating such vulnerabilities have focused on
enhancing the robustness of classification models. In contrast, this paper
adopts a fundamentally different strategy aimed at increasing robustness
through the redesign of road signs themselves. We propose an attacker-agnostic
learning scheme to automatically design road signs that are robust to a wide
array of patch-based attacks. Empirical tests conducted in both digital and
physical environments demonstrate that our approach significantly reduces
vulnerability to patch attacks, outperforming existing techniques.


http://arxiv.org/abs/2411.16154
DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders. (10%)
Sizai Hou; Songze Li; Duanyi Yao
  Self-supervised learning (SSL) is pervasively exploited in training
high-quality upstream encoders with a large amount of unlabeled data. However,
it is found to be susceptible to backdoor attacks merely via polluting a small
portion of training data. The victim encoders associate triggered inputs with
target embeddings, e.g., mapping a triggered cat image to an airplane
embedding, such that the downstream tasks inherit unintended behaviors when the
trigger is activated. Emerging backdoor attacks have shown great threats across
different SSL paradigms such as contrastive learning and CLIP, yet limited
research is devoted to defending against such attacks, and existing defenses
fall short in detecting advanced stealthy backdoors. To address the
limitations, we propose a novel detection mechanism, DeDe, which detects the
activation of backdoor mappings caused by triggered inputs on victim encoders.
Specifically, DeDe trains a decoder for any given SSL encoder using an
auxiliary dataset (which can be out-of-distribution or even slightly poisoned),
so that for any triggered input that misleads the encoder into the target
embedding, the decoder generates an output image significantly different from
the input. DeDe leverages the discrepancy between the input and the decoded
output to identify potential backdoor misbehavior during inference. We
empirically evaluate DeDe on both contrastive learning and CLIP models against
various types of backdoor attacks. Our results demonstrate promising detection
effectiveness over various advanced attacks and superior performance compared
over state-of-the-art detection methods.


http://arxiv.org/abs/2411.16167
BadSFL: Backdoor Attack against Scaffold Federated Learning. (3%)
Xingshuo Han; Xuanye Zhang; Xiang Lan; Haozhao Wang; Shengmin Xu; Shen Ren; Jason Zeng; Ming Wu; Michael Heinrich; Tianwei Zhang
  Federated learning (FL) enables the training of deep learning models on
distributed clients to preserve data privacy. However, this learning paradigm
is vulnerable to backdoor attacks, where malicious clients can upload poisoned
local models to embed backdoors into the global model, leading to
attacker-desired predictions. Existing backdoor attacks mainly focus on FL with
independently and identically distributed (IID) scenarios, while real-world FL
training data are typically non-IID. Current strategies for non-IID backdoor
attacks suffer from limitations in maintaining effectiveness and durability. To
address these challenges, we propose a novel backdoor attack method, BadSFL,
specifically designed for the FL framework using the scaffold aggregation
algorithm in non-IID settings. BadSFL leverages a Generative Adversarial
Network (GAN) based on the global model to complement the training set,
achieving high accuracy on both backdoor and benign samples. It utilizes a
specific feature as the backdoor trigger to ensure stealthiness, and exploits
the Scaffold's control variate to predict the global model's convergence
direction, ensuring the backdoor's persistence. Extensive experiments on three
benchmark datasets demonstrate the high effectiveness, stealthiness, and
durability of BadSFL. Notably, our attack remains effective over 60 rounds in
the global model and up to 3 times longer than existing baseline attacks after
stopping the injection of malicious updates.


http://arxiv.org/abs/2411.16120
Why the Agent Made that Decision: Explaining Deep Reinforcement Learning with Vision Masks. (2%)
Rui Zuo; Zifan Wang; Simon Khan; Garrett Ethan Katz; Qinru Qiu
  Due to the inherent lack of transparency in deep neural networks, it is
challenging for deep reinforcement learning (DRL) agents to gain trust and
acceptance from users, especially in safety-critical applications such as
medical diagnosis and military operations. Existing methods for explaining an
agent's decision either require to retrain the agent using models that support
explanation generation or rely on perturbation-based techniques to reveal the
significance of different input features in the decision making process.
However, retraining the agent may compromise its integrity and performance,
while perturbation-based methods have limited performance and lack knowledge
accumulation or learning capabilities. Moreover, since each perturbation is
performed independently, the joint state of the perturbed inputs may not be
physically meaningful. To address these challenges, we introduce
$\textbf{VisionMask}$, a standalone explanation model trained end-to-end to
identify the most critical regions in the agent's visual input that can explain
its actions. VisionMask is trained in a self-supervised manner without relying
on human-generated labels. Importantly, its training does not alter the agent
model, hence preserving the agent's performance and integrity. We evaluate
VisionMask on Super Mario Bros (SMB) and three Atari games. Compared to
existing methods, VisionMask achieves a 14.9% higher insertion accuracy and a
30.08% higher F1-Score in reproducing original actions from the selected visual
explanations. We also present examples illustrating how VisionMask can be used
for counterfactual analysis.


http://arxiv.org/abs/2411.16817
XAI and Android Malware Models. (2%)
Maithili Kulkarni; Mark Stamp
  Android malware detection based on machine learning (ML) and deep learning
(DL) models is widely used for mobile device security. Such models offer
benefits in terms of detection accuracy and efficiency, but it is often
difficult to understand how such learning models make decisions. As a result,
these popular malware detection strategies are generally treated as black
boxes, which can result in a lack of trust in the decisions made, as well as
making adversarial attacks more difficult to detect. The field of eXplainable
Artificial Intelligence (XAI) attempts to shed light on such black box models.
In this paper, we apply XAI techniques to ML and DL models that have been
trained on a challenging Android malware classification problem. Specifically,
the classic ML models considered are Support Vector Machines (SVM), Random
Forest, and $k$-Nearest Neighbors ($k$-NN), while the DL models we consider are
Multi-Layer Perceptrons (MLP) and Convolutional Neural Networks (CNN). The
state-of-the-art XAI techniques that we apply to these trained models are Local
Interpretable Model-agnostic Explanations (LIME), Shapley Additive exPlanations
(SHAP), PDP plots, ELI5, and Class Activation Mapping (CAM). We obtain global
and local explanation results, and we discuss the utility of XAI techniques in
this problem domain. We also provide a literature review of XAI work related to
Android malware.


http://arxiv.org/abs/2411.16148
Revisiting Marr in Face: The Building of 2D--2.5D--3D Representations in Deep Neural Networks. (1%)
Xiangyu Zhu; Chang Yu; Jiankuo Zhao; Zhaoxiang Zhang; Stan Z. Li; Zhen Lei
  David Marr's seminal theory of vision proposes that the human visual system
operates through a sequence of three stages, known as the 2D sketch, the 2.5D
sketch, and the 3D model. In recent years, Deep Neural Networks (DNN) have been
widely thought to have reached a level comparable to human vision. However, the
mechanisms by which DNNs accomplish this and whether they adhere to Marr's
2D--2.5D--3D construction theory remain unexplored. In this paper, we delve
into the perception task to explore these questions and find evidence
supporting Marr's theory. We introduce a graphics probe, a sub-network crafted
to reconstruct the original image from the network's intermediate layers. The
key to the graphics probe is its flexible architecture that supports image in
both 2D and 3D formats, as well as in a transitional state between them. By
injecting graphics probes into neural networks, and analyzing their behavior in
reconstructing images, we find that DNNs initially encode images as 2D
representations in low-level layers, and finally construct 3D representations
in high-level layers. Intriguingly, in mid-level layers, DNNs exhibit a hybrid
state, building a geometric representation that s sur normals within a narrow
depth range, akin to the appearance of a low-relief sculpture. This stage
resembles the 2.5D representations, providing a view of how DNNs evolve from 2D
to 3D in the perception process. The graphics probe therefore serves as a tool
for peering into the mechanisms of DNN, providing empirical support for Marr's
theory.


http://arxiv.org/abs/2411.15720
Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks. (99%)
Peng Xie; Yequan Bie; Jianda Mao; Yangqiu Song; Yang Wang; Hao Chen; Kani Chen
  Pre-trained vision-language models (VLMs) have showcased remarkable
performance in image and natural language understanding, such as image
captioning and response generation. As the practical applications of
vision-language models become increasingly widespread, their potential safety
and robustness issues raise concerns that adversaries may evade the system and
cause these models to generate toxic content through malicious attacks.
Therefore, evaluating the robustness of open-source VLMs against adversarial
attacks has garnered growing attention, with transfer-based attacks as a
representative black-box attacking strategy. However, most existing
transfer-based attacks neglect the importance of the semantic correlations
between vision and text modalities, leading to sub-optimal adversarial example
generation and attack performance. To address this issue, we present Chain of
Attack (CoA), which iteratively enhances the generation of adversarial examples
based on the multi-modal semantic update using a series of intermediate
attacking steps, achieving superior adversarial transferability and efficiency.
A unified attack success rate computing method is further proposed for
automatic evasion evaluation. Extensive experiments conducted under the most
realistic and high-stakes scenario, demonstrate that our attacking strategy can
effectively mislead models to generate targeted responses using only black-box
attacks without any knowledge of the victim models. The comprehensive
robustness evaluation in our paper provides insight into the vulnerabilities of
VLMs and offers a reference for the safety considerations of future model
developments.


http://arxiv.org/abs/2411.15878
ExAL: An Exploration Enhanced Adversarial Learning Algorithm. (92%)
A Vinil; Aneesh Sreevallabh Chivukula; Pranav Chintareddy
  Adversarial learning is critical for enhancing model robustness, aiming to
defend against adversarial attacks that jeopardize machine learning systems.
Traditional methods often lack efficient mechanisms to explore diverse
adversarial perturbations, leading to limited model resilience. Inspired by
game-theoretic principles, where adversarial dynamics are analyzed through
frameworks like Nash equilibrium, exploration mechanisms in such setups allow
for the discovery of diverse strategies, enhancing system robustness. However,
existing adversarial learning methods often fail to incorporate structured
exploration effectively, reducing their ability to improve model defense
comprehensively. To address these challenges, we propose a novel
Exploration-enhanced Adversarial Learning Algorithm (ExAL), leveraging the
Exponentially Weighted Momentum Particle Swarm Optimizer (EMPSO) to generate
optimized adversarial perturbations. ExAL integrates exploration-driven
mechanisms to discover perturbations that maximize impact on the model's
decision boundary while preserving structural coherence in the data. We
evaluate the performance of ExAL on the MNIST Handwritten Digits and Blended
Malware datasets. Experimental results demonstrate that ExAL significantly
enhances model resilience to adversarial attacks by improving robustness
through adversarial learning.


http://arxiv.org/abs/2411.15921
A Tunable Despeckling Neural Network Stabilized via Diffusion Equation. (64%)
Yi Ran; Zhichang Guo; Jia Li; Yao Li; Martin Burger; Boying Wu
  The removal of multiplicative Gamma noise is a critical research area in the
application of synthetic aperture radar (SAR) imaging, where neural networks
serve as a potent tool. However, real-world data often diverges from
theoretical models, exhibiting various disturbances, which makes the neural
network less effective. Adversarial attacks can be used as a criterion for
judging the adaptability of neural networks to real data, since adversarial
attacks can find the most extreme perturbations that make neural networks
ineffective. In this work, the diffusion equation is designed as a
regularization block to provide sufficient regularity to the whole neural
network, due to its spontaneous dissipative nature. We propose a tunable,
regularized neural network framework that unrolls a shallow denoising neural
network block and a diffusion regularity block into a single network for
end-to-end training. The linear heat equation, known for its inherent
smoothness and low-pass filtering properties, is adopted as the diffusion
regularization block. In our model, a single time step hyperparameter governs
the smoothness of the outputs and can be adjusted dynamically, significantly
enhancing flexibility. The stability and convergence of our model are
theoretically proven. Experimental results demonstrate that the proposed model
effectively eliminates high-frequency oscillations induced by adversarial
attacks. Finally, the proposed model is benchmarked against several
state-of-the-art denoising methods on simulated images, adversarial samples,
and real SAR images, achieving superior performance in both quantitative and
visual evaluations.


http://arxiv.org/abs/2411.16763
Hide in Plain Sight: Clean-Label Backdoor for Auditing Membership Inference. (10%)
Depeng Chen; Hao Chen; Hulin Jin; Jie Cui; Hong Zhong
  Membership inference attacks (MIAs) are critical tools for assessing privacy
risks and ensuring compliance with regulations like the General Data Protection
Regulation (GDPR). However, their potential for auditing unauthorized use of
data remains under explored. To bridge this gap, we propose a novel clean-label
backdoor-based approach for MIAs, designed specifically for robust and stealthy
data auditing. Unlike conventional methods that rely on detectable poisoned
samples with altered labels, our approach retains natural labels, enhancing
stealthiness even at low poisoning rates. Our approach employs an optimal
trigger generated by a shadow model that mimics the target model's behavior.
This design minimizes the feature-space distance between triggered samples and
the source class while preserving the original data labels. The result is a
powerful and undetectable auditing mechanism that overcomes limitations of
existing approaches, such as label inconsistencies and visual artifacts in
poisoned samples. The proposed method enables robust data auditing through
black-box access, achieving high attack success rates across diverse datasets
and model architectures. Additionally, it addresses challenges related to
trigger stealthiness and poisoning durability, establishing itself as a
practical and effective solution for data auditing. Comprehensive experiments
validate the efficacy and generalizability of our approach, outperforming
several baseline methods in both stealth and attack success metrics.


http://arxiv.org/abs/2411.16024
Stealth Attacks Against Moving Target Defense for Smart Grid. (2%)
Ke Sun; Iñaki Esnaola; H. Vincent Poor
  Data injection attacks (DIAs) pose a significant cybersecurity threat to the
Smart Grid by enabling an attacker to compromise the integrity of data
acquisition and manipulate estimated states without triggering bad data
detection procedures. To mitigate this vulnerability, the moving target defense
(MTD) alters branch admittances to mismatch the system information that is
available to an attacker, thereby inducing an imperfect DIA construction that
results in degradation of attack performance. In this paper, we first analyze
the existence of stealth attacks for the case in which the MTD strategy only
changes the admittance of a single branch. Equipped with this initial insight,
we then extend the results to the case in which multiple branches are protected
by the MTD strategy. Remarkably, we show that stealth attacks can be
constructed with information only about which branches are protected, without
knowledge about the particular admittance value changes. Furthermore, we
provide a sufficient protection condition for the MTD strategy via
graph-theoretic tools that guarantee that the system is not vulnerable to DIAs.
Numerical simulations are implemented on IEEE test systems to validate the
obtained results.


http://arxiv.org/abs/2411.15976
DRIVE: Dual-Robustness via Information Variability and Entropic Consistency in Source-Free Unsupervised Domain Adaptation. (2%)
Ruiqiang Xiao; Songning Lai; Yijun Yang; Jiemin Wu; Yutao Yue; Lei Zhu
  Adapting machine learning models to new domains without labeled data,
especially when source data is inaccessible, is a critical challenge in
applications like medical imaging, autonomous driving, and remote sensing. This
task, known as Source-Free Unsupervised Domain Adaptation (SFUDA), involves
adapting a pre-trained model to a target domain using only unlabeled target
data, which can lead to issues such as overfitting, underfitting, and poor
generalization due to domain discrepancies and noise. Existing SFUDA methods
often rely on single-model architectures, struggling with uncertainty and
variability in the target domain. To address these challenges, we propose DRIVE
(Dual-Robustness through Information Variability and Entropy), a novel SFUDA
framework leveraging a dual-model architecture. The two models, initialized
with identical weights, work in parallel to capture diverse target domain
characteristics. One model is exposed to perturbations via projection gradient
descent (PGD) guided by mutual information, focusing on high-uncertainty
regions. We also introduce an entropy-aware pseudo-labeling strategy that
adjusts label weights based on prediction uncertainty, ensuring the model
focuses on reliable data while avoiding noisy regions. The adaptation process
has two stages: the first aligns the models on stable features using a mutual
information consistency loss, and the second dynamically adjusts the
perturbation level based on the loss from the first stage, encouraging the
model to explore a broader range of the target domain while preserving existing
performance. This enhances generalization capabilities and robustness against
interference. Evaluations on standard SFUDA benchmarks show that DRIVE
consistently outperforms previous methods, delivering improved adaptation
accuracy and stability across complex target domains.


http://arxiv.org/abs/2411.15553
Improving Transferable Targeted Attacks with Feature Tuning Mixup. (99%)
Kaisheng Liang; Xuelong Dai; Yanjie Li; Dong Wang; Bin Xiao
  Deep neural networks (DNNs) exhibit vulnerability to adversarial examples
that can transfer across different DNN models. A particularly challenging
problem is developing transferable targeted attacks that can mislead DNN models
into predicting specific target classes. While various methods have been
proposed to enhance attack transferability, they often incur substantial
computational costs while yielding limited improvements. Recent clean feature
mixup methods use random clean features to perturb the feature space but lack
optimization for disrupting adversarial examples, overlooking the advantages of
attack-specific perturbations. In this paper, we propose Feature Tuning Mixup
(FTM), a novel method that enhances targeted attack transferability by
combining both random and optimized noises in the feature space. FTM introduces
learnable feature perturbations and employs an efficient stochastic update
strategy for optimization. These learnable perturbations facilitate the
generation of more robust adversarial examples with improved transferability.
We further demonstrate that attack performance can be enhanced through an
ensemble of multiple FTM-perturbed surrogate models. Extensive experiments on
the ImageNet-compatible dataset across various DNN models demonstrate that our
method achieves significant improvements over state-of-the-art methods while
maintaining low computational cost.


http://arxiv.org/abs/2411.15555
Improving the Transferability of Adversarial Attacks on Face Recognition with Diverse Parameters Augmentation. (99%)
Fengfan Zhou; Bangjie Yin; Hefei Ling; Qianyu Zhou; Wenxuan Wang
  Face Recognition (FR) models are vulnerable to adversarial examples that
subtly manipulate benign face images, underscoring the urgent need to improve
the transferability of adversarial attacks in order to expose the blind spots
of these systems. Existing adversarial attack methods often overlook the
potential benefits of augmenting the surrogate model with diverse
initializations, which limits the transferability of the generated adversarial
examples. To address this gap, we propose a novel method called Diverse
Parameters Augmentation (DPA) attack method, which enhances surrogate models by
incorporating diverse parameter initializations, resulting in a broader and
more diverse set of surrogate models. Specifically, DPA consists of two key
stages: Diverse Parameters Optimization (DPO) and Hard Model Aggregation (HMA).
In the DPO stage, we initialize the parameters of the surrogate model using
both pre-trained and random parameters. Subsequently, we save the models in the
intermediate training process to obtain a diverse set of surrogate models.
During the HMA stage, we enhance the feature maps of the diversified surrogate
models by incorporating beneficial perturbations, thereby further improving the
transferability. Experimental results demonstrate that our proposed attack
method can effectively enhance the transferability of the crafted adversarial
face examples.


http://arxiv.org/abs/2411.16746
LoBAM: LoRA-Based Backdoor Attack on Model Merging. (10%)
Ming Yin; Jingyang Zhang; Jingwei Sun; Minghong Fang; Hai Li; Yiran Chen
  Model merging is an emerging technique that integrates multiple models
fine-tuned on different tasks to create a versatile model that excels in
multiple domains. This scheme, in the meantime, may open up backdoor attack
opportunities where one single malicious model can jeopardize the integrity of
the merged model. Existing works try to demonstrate the risk of such attacks by
assuming substantial computational resources, focusing on cases where the
attacker can fully fine-tune the pre-trained model. Such an assumption,
however, may not be feasible given the increasing size of machine learning
models. In practice where resources are limited and the attacker can only
employ techniques like Low-Rank Adaptation (LoRA) to produce the malicious
model, it remains unclear whether the attack can still work and pose threats.
In this work, we first identify that the attack efficacy is significantly
diminished when using LoRA for fine-tuning. Then, we propose LoBAM, a method
that yields high attack success rate with minimal training resources. The key
idea of LoBAM is to amplify the malicious weights in an intelligent way that
effectively enhances the attack efficacy. We demonstrate that our design can
lead to improved attack success rate through extensive empirical experiments
across various model merging scenarios. Moreover, we show that our method is
highly stealthy and is difficult to detect and defend against.


http://arxiv.org/abs/2411.15673
Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignment. (4%)
Alvi Md Ishmam; Christopher Thomas
  In recent years there has been enormous interest in vision-language models
trained using self-supervised objectives. However, the use of large-scale
datasets scraped from the web for training also makes these models vulnerable
to potential security threats, such as backdooring and poisoning attacks. In
this paper, we propose a method for mitigating such attacks on contrastively
trained vision-language models. Our approach leverages external knowledge
extracted from a language model to prevent models from learning correlations
between image regions which lack strong alignment with external knowledge. We
do this by imposing constraints to enforce that attention paid by the model to
visual regions is proportional to the alignment of those regions with external
knowledge. We conduct extensive experiments using a variety of recent
backdooring and poisoning attacks on multiple datasets and architectures. Our
results clearly demonstrate that our proposed approach is highly effective at
defending against such attacks across multiple settings, while maintaining
model utility and without requiring any changes at inference time


http://arxiv.org/abs/2411.16721
Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks. (99%)
Han Wang; Gang Wang; Huan Zhang
  Vision Language Models (VLMs) can produce unintended and harmful content when
exposed to adversarial attacks, particularly because their vision capabilities
create new vulnerabilities. Existing defenses, such as input preprocessing,
adversarial training, and response evaluation-based methods, are often
impractical for real-world deployment due to their high costs. To address this
challenge, we propose ASTRA, an efficient and effective defense by adaptively
steering models away from adversarial feature directions to resist VLM attacks.
Our key procedures involve finding transferable steering vectors representing
the direction of harmful response and applying adaptive activation steering to
remove these directions at inference time. To create effective steering
vectors, we randomly ablate the visual tokens from the adversarial images and
identify those most strongly associated with jailbreaks. These tokens are then
used to construct steering vectors. During inference, we perform the adaptive
steering method that involves the projection between the steering vectors and
calibrated activation, resulting in little performance drops on benign inputs
while strongly avoiding harmful outputs under adversarial inputs. Extensive
experiments across multiple models and baselines demonstrate our
state-of-the-art performance and high efficiency in mitigating jailbreak risks.
Additionally, ASTRA exhibits good transferability, defending against both
unseen attacks at design time (i.e., structured-based attacks) and adversarial
images from diverse distributions.


http://arxiv.org/abs/2411.14834
Gradient Masking All-at-Once: Ensemble Everything Everywhere Is Not Robust. (99%)
Jie Zhang; Kristina Nikolić; Nicholas Carlini; Florian Tramèr
  Ensemble everything everywhere is a defense to adversarial examples that was
recently proposed to make image classifiers robust. This defense works by
ensembling a model's intermediate representations at multiple noisy image
resolutions, producing a single robust classification. This defense was shown
to be effective against multiple state-of-the-art attacks. Perhaps even more
convincingly, it was shown that the model's gradients are perceptually aligned:
attacks against the model produce noise that perceptually resembles the
targeted class.
  In this short note, we show that this defense is not robust to adversarial
attack. We first show that the defense's randomness and ensembling method cause
severe gradient masking. We then use standard adaptive attack techniques to
reduce the defense's robust accuracy from 48% to 1% on CIFAR-100 and from 62%
to 4% on CIFAR-10, under the $\ell_\infty$-norm threat model with
$\varepsilon=8/255$.


http://arxiv.org/abs/2411.15246
Exploring the Robustness and Transferability of Patch-Based Adversarial Attacks in Quantized Neural Networks. (99%)
Amira Guesmi; Bassem Ouni; Muhammad Shafique
  Quantized neural networks (QNNs) are increasingly used for efficient
deployment of deep learning models on resource-constrained platforms, such as
mobile devices and edge computing systems. While quantization reduces model
size and computational demands, its impact on adversarial robustness-especially
against patch-based attacks-remains inadequately addressed. Patch-based
attacks, characterized by localized, high-visibility perturbations, pose
significant security risks due to their transferability and resilience. In this
study, we systematically evaluate the vulnerability of QNNs to patch-based
adversarial attacks across various quantization levels and architectures,
focusing on factors that contribute to the robustness of these attacks. Through
experiments analyzing feature representations, quantization strength, gradient
alignment, and spatial sensitivity, we find that patch attacks consistently
achieve high success rates across bitwidths and architectures, demonstrating
significant transferability even in heavily quantized models. Contrary to the
expectation that quantization might enhance adversarial defenses, our results
show that QNNs remain highly susceptible to patch attacks due to the
persistence of distinct, localized features within quantized representations.
These findings underscore the need for quantization-aware defenses that address
the specific challenges posed by patch-based attacks. Our work contributes to a
deeper understanding of adversarial robustness in QNNs and aims to guide future
research in developing secure, quantization-compatible defenses for real-world
applications.


http://arxiv.org/abs/2411.14842
Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Language Models. (45%)
Wanqi Yang; Yanda Li; Meng Fang; Yunchao Wei; Tianyi Zhou; Ling Chen
  Adversarial audio attacks pose a significant threat to the growing use of
large language models (LLMs) in voice-based human-machine interactions. While
existing research has primarily focused on model-specific adversarial methods,
real-world applications demand a more generalizable and universal approach to
audio adversarial attacks. In this paper, we introduce the Chat-Audio Attacks
(CAA) benchmark including four distinct types of audio attacks, which aims to
explore the the vulnerabilities of LLMs to these audio attacks in
conversational scenarios. To evaluate the robustness of LLMs, we propose three
evaluation strategies: Standard Evaluation, utilizing traditional metrics to
quantify model performance under attacks; GPT-4o-Based Evaluation, which
simulates real-world conversational complexities; and Human Evaluation,
offering insights into user perception and trust. We evaluate six
state-of-the-art LLMs with voice interaction capabilities, including
Gemini-1.5-Pro, GPT-4o, and others, using three distinct evaluation methods on
the CAA benchmark. Our comprehensive analysis reveals the impact of four types
of audio attacks on the performance of these models, demonstrating that GPT-4o
exhibits the highest level of resilience.


http://arxiv.org/abs/2411.14738
Universal and Context-Independent Triggers for Precise Control of LLM Outputs. (31%)
Jiashuo Liang; Guancheng Li; Yang Yu
  Large language models (LLMs) have been widely adopted in applications such as
automated content generation and even critical decision-making systems.
However, the risk of prompt injection allows for potential manipulation of LLM
outputs. While numerous attack methods have been documented, achieving full
control over these outputs remains challenging, often requiring experienced
attackers to make multiple attempts and depending heavily on the prompt
context. Recent advancements in gradient-based white-box attack techniques have
shown promise in tasks like jailbreaks and system prompt leaks. Our research
generalizes gradient-based attacks to find a trigger that is (1) Universal:
effective irrespective of the target output; (2) Context-Independent: robust
across diverse prompt contexts; and (3) Precise Output: capable of manipulating
LLM inputs to yield any specified output with high accuracy. We propose a novel
method to efficiently discover such triggers and assess the effectiveness of
the proposed attack. Furthermore, we discuss the substantial threats posed by
such attacks to LLM-based applications, highlighting the potential for
adversaries to taking over the decisions and actions made by AI agents.


http://arxiv.org/abs/2411.14865
Benchmarking the Robustness of Optical Flow Estimation to Corruptions. (13%)
Zhonghua Yi; Hao Shi; Qi Jiang; Yao Gao; Ze Wang; Yufan Zhang; Kailun Yang; Kaiwei Wang
  Optical flow estimation is extensively used in autonomous driving and video
editing. While existing models demonstrate state-of-the-art performance across
various benchmarks, the robustness of these methods has been infrequently
investigated. Despite some research focusing on the robustness of optical flow
models against adversarial attacks, there has been a lack of studies
investigating their robustness to common corruptions. Taking into account the
unique temporal characteristics of optical flow, we introduce 7 temporal
corruptions specifically designed for benchmarking the robustness of optical
flow models, in addition to 17 classical single-image corruptions, in which
advanced PSF Blur simulation method is performed. Two robustness benchmarks,
KITTI-FC and GoPro-FC, are subsequently established as the first corruption
robustness benchmark for optical flow estimation, with Out-Of-Domain (OOD) and
In-Domain (ID) settings to facilitate comprehensive studies. Robustness
metrics, Corruption Robustness Error (CRE), Corruption Robustness Error ratio
(CREr), and Relative Corruption Robustness Error (RCRE) are further introduced
to quantify the optical flow estimation robustness. 29 model variants from 15
optical flow methods are evaluated, yielding 10 intriguing observations, such
as 1) the absolute robustness of the model is heavily dependent on the
estimation performance; 2) the corruptions that diminish local information are
more serious than that reduce visual effects. We also give suggestions for the
design and application of optical flow models. We anticipate that our benchmark
will serve as a foundational resource for advancing research in robust optical
flow estimation. The benchmarks and source code will be released at
https://github.com/ZhonghuaYi/optical_flow_robustness_benchmark.


http://arxiv.org/abs/2411.14937
Geminio: Language-Guided Gradient Inversion Attacks in Federated Learning. (5%)
Junjie Shan; Ziqi Zhao; Jialin Lu; Rui Zhang; Siu Ming Yiu; Ka-Ho Chow
  Foundation models that bridge vision and language have made significant
progress, inspiring numerous life-enriching applications. However, their
potential for misuse to introduce new threats remains largely unexplored. This
paper reveals that vision-language models (VLMs) can be exploited to overcome
longstanding limitations in gradient inversion attacks (GIAs) within federated
learning (FL), where an FL server reconstructs private data samples from
gradients shared by victim clients. Current GIAs face challenges in
reconstructing high-resolution images, especially when the victim has a large
local data batch. While focusing reconstruction on valuable samples rather than
the entire batch is promising, existing methods lack the flexibility to allow
attackers to specify their target data. In this paper, we introduce Geminio,
the first approach to transform GIAs into semantically meaningful, targeted
attacks. Geminio enables a brand new privacy attack experience: attackers can
describe, in natural language, the types of data they consider valuable, and
Geminio will prioritize reconstruction to focus on those high-value samples.
This is achieved by leveraging a pretrained VLM to guide the optimization of a
malicious global model that, when shared with and optimized by a victim,
retains only gradients of samples that match the attacker-specified query.
Extensive experiments demonstrate Geminio's effectiveness in pinpointing and
reconstructing targeted samples, with high success rates across complex
datasets under FL and large batch sizes and showing resilience against existing
defenses.


http://arxiv.org/abs/2411.15439
Twin Trigger Generative Networks for Backdoor Attacks against Object Detection. (4%)
Zhiying Li; Zhi Liu; Guanggang Geng; Shreyank N Gowda; Shuyuan Lin; Jian Weng; Xiaobo Jin
  Object detectors, which are widely used in real-world applications, are
vulnerable to backdoor attacks. This vulnerability arises because many users
rely on datasets or pre-trained models provided by third parties due to
constraints on data and resources. However, most research on backdoor attacks
has focused on image classification, with limited investigation into object
detection. Furthermore, the triggers for most existing backdoor attacks on
object detection are manually generated, requiring prior knowledge and
consistent patterns between the training and inference stages. This approach
makes the attacks either easy to detect or difficult to adapt to various
scenarios. To address these limitations, we propose novel twin trigger
generative networks in the frequency domain to generate invisible triggers for
implanting stealthy backdoors into models during training, and visible triggers
for steady activation during inference, making the attack process difficult to
trace. Specifically, for the invisible trigger generative network, we deploy a
Gaussian smoothing layer and a high-frequency artifact classifier to enhance
the stealthiness of backdoor implantation in object detectors. For the visible
trigger generative network, we design a novel alignment loss to optimize the
visible triggers so that they differ from the original patterns but still align
with the malicious activation behavior of the invisible triggers. Extensive
experimental results and analyses prove the possibility of using different
triggers in the training stage and the inference stage, and demonstrate the
attack effectiveness of our proposed visible trigger and invisible trigger
generative networks, significantly reducing the mAP_0.5 of the object detectors
by 70.0% and 84.5%, including YOLOv5 and YOLOv7 with different settings,
respectively.


http://arxiv.org/abs/2411.15306
Heavy-tailed Contamination is Easier than Adversarial Contamination. (1%)
Yeshwanth Cherapanamjeri; Daniel Lee
  A large body of work in the statistics and computer science communities
dating back to Huber (Huber, 1960) has led to statistically and computationally
efficient outlier-robust estimators. Two particular outlier models have
received significant attention: the adversarial and heavy-tailed models. While
the former models outliers as the result of a malicious adversary manipulating
the data, the latter relaxes distributional assumptions on the data allowing
outliers to naturally occur as part of the data generating process. In the
first setting, the goal is to develop estimators robust to the largest fraction
of outliers while in the second, one seeks estimators to combat the loss of
statistical efficiency, where the dependence on the failure probability is
paramount.
  Despite these distinct motivations, the algorithmic approaches to both these
settings have converged, prompting questions on the relationship between the
models. In this paper, we investigate and provide a principled explanation for
this phenomenon. First, we prove that any adversarially robust estimator is
also resilient to heavy-tailed outliers for any statistical estimation problem
with i.i.d data. As a corollary, optimal adversarially robust estimators for
mean estimation, linear regression, and covariance estimation are also optimal
heavy-tailed estimators. Conversely, for arguably the simplest high-dimensional
estimation task of mean estimation, we construct heavy-tailed estimators whose
application to the adversarial setting requires any black-box reduction to
remove almost all the outliers in the data. Taken together, our results imply
that heavy-tailed estimation is likely easier than adversarially robust
estimation opening the door to novel algorithmic approaches for the
heavy-tailed setting. Additionally, confidence intervals obtained for
adversarially robust estimation also hold with high-probability.


http://arxiv.org/abs/2411.15367
Exploiting Watermark-Based Defense Mechanisms in Text-to-Image Diffusion Models for Unauthorized Data Usage. (1%)
Soumil Datta; Shih-Chieh Dai; Leo Yu; Guanhong Tao
  Text-to-image diffusion models, such as Stable Diffusion, have shown
exceptional potential in generating high-quality images. However, recent
studies highlight concerns over the use of unauthorized data in training these
models, which may lead to intellectual property infringement or privacy
violations. A promising approach to mitigate these issues is to apply a
watermark to images and subsequently check if generative models reproduce
similar watermark features. In this paper, we examine the robustness of various
watermark-based protection methods applied to text-to-image models. We observe
that common image transformations are ineffective at removing the watermark
effect. Therefore, we propose RATTAN, that leverages the diffusion process to
conduct controlled image generation on the protected input, preserving the
high-level features of the input while ignoring the low-level details utilized
by watermarks. A small number of generated images are then used to fine-tune
protected models. Our experiments on three datasets and 140 text-to-image
diffusion models reveal that existing state-of-the-art protections are not
robust against RATTAN.


http://arxiv.org/abs/2411.14946
Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach. (1%)
Lars Nieradzik; Henrike Stephani; Janis Keuper
  In this paper, we present an approach for evaluating attribution maps, which
play a central role in interpreting the predictions of convolutional neural
networks (CNNs). We show that the widely used insertion/deletion metrics are
susceptible to distribution shifts that affect the reliability of the ranking.
Our method proposes to replace pixel modifications with adversarial
perturbations, which provides a more robust evaluation framework. By using
smoothness and monotonicity measures, we illustrate the effectiveness of our
approach in correcting distribution shifts. In addition, we conduct the most
comprehensive quantitative and qualitative assessment of attribution maps to
date. Introducing baseline attribution maps as sanity checks, we find that our
metric is the only contender to pass all checks. Using Kendall's $\tau$ rank
correlation coefficient, we show the increased consistency of our metric across
15 dataset-architecture combinations. Of the 16 attribution maps tested, our
results clearly show SmoothGrad to be the best map currently available. This
research makes an important contribution to the development of attribution maps
by providing a reliable and consistent evaluation framework. To ensure
reproducibility, we will provide the code along with our results.


http://arxiv.org/abs/2411.14263
Generating Realistic Adversarial Examples for Business Processes using Variational Autoencoders. (99%)
Alexander Stevens; Jari Peeperkorn; Smedt Johannes De; Weerdt Jochen De
  In predictive process monitoring, predictive models are vulnerable to
adversarial attacks, where input perturbations can lead to incorrect
predictions. Unlike in computer vision, where these perturbations are designed
to be imperceptible to the human eye, the generation of adversarial examples in
predictive process monitoring poses unique challenges. Minor changes to the
activity sequences can create improbable or even impossible scenarios to occur
due to underlying constraints such as regulatory rules or process constraints.
To address this, we focus on generating realistic adversarial examples tailored
to the business process context, in contrast to the imperceptible, pixel-level
changes commonly seen in computer vision adversarial attacks. This paper
introduces two novel latent space attacks, which generate adversaries by adding
noise to the latent space representation of the input data, rather than
directly modifying the input attributes. These latent space methods are
domain-agnostic and do not rely on process-specific knowledge, as we restrict
the generation of adversarial examples to the learned class-specific data
distributions by directly perturbing the latent space representation of the
business process executions. We evaluate these two latent space methods with
six other adversarial attacking methods on eleven real-life event logs and four
predictive models. The first three attacking methods directly permute the
activities of the historically observed business process executions. The fourth
method constrains the adversarial examples to lie within the same data
distribution as the original instances, by projecting the adversarial examples
to the original data distribution.


http://arxiv.org/abs/2411.14424
Learning Fair Robustness via Domain Mixup. (81%)
Meiyu Zhong; Ravi Tandon
  Adversarial training is one of the predominant techniques for training
classifiers that are robust to adversarial attacks. Recent work, however has
found that adversarial training, which makes the overall classifier robust, it
does not necessarily provide equal amount of robustness for all classes. In
this paper, we propose the use of mixup for the problem of learning fair robust
classifiers, which can provide similar robustness across all classes.
Specifically, the idea is to mix inputs from the same classes and perform
adversarial training on mixed up inputs. We present a theoretical analysis of
this idea for the case of linear classifiers and show that mixup combined with
adversarial training can provably reduce the class-wise robustness disparity.
This method not only contributes to reducing the disparity in class-wise
adversarial risk, but also the class-wise natural risk. Complementing our
theoretical analysis, we also provide experimental results on both synthetic
data and the real world dataset (CIFAR-10), which shows improvement in class
wise disparities for both natural and adversarial risks.


http://arxiv.org/abs/2411.14133
GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs. (78%)
Advik Raj Basani; Xiao Zhang
  Large Language Models (LLMs) have shown impressive proficiency across a range
of natural language processing tasks yet remain vulnerable to adversarial
prompts, known as jailbreak attacks, carefully designed to elicit harmful
responses from LLMs. Traditional methods rely on manual heuristics, which
suffer from limited generalizability. While being automatic, optimization-based
attacks often produce unnatural jailbreak prompts that are easy to detect by
safety filters or require high computational overhead due to discrete token
optimization. Witnessing the limitations of existing jailbreak methods, we
introduce Generative Adversarial Suffix Prompter (GASP), a novel framework that
combines human-readable prompt generation with Latent Bayesian Optimization
(LBO) to improve adversarial suffix creation in a fully black-box setting. GASP
leverages LBO to craft adversarial suffixes by efficiently exploring continuous
embedding spaces, gradually optimizing the model to improve attack efficacy
while balancing prompt coherence through a targeted iterative refinement
procedure. Our experiments show that GASP can generate natural jailbreak
prompts, significantly improving attack success rates, reducing training times,
and accelerating inference speed, thus making it an efficient and scalable
solution for red-teaming LLMs.


http://arxiv.org/abs/2411.14243
AnywhereDoor: Multi-Target Backdoor Attacks on Object Detection. (74%)
Jialin Lu; Junjie Shan; Ziqi Zhao; Ka-Ho Chow
  As object detection becomes integral to many safety-critical applications,
understanding its vulnerabilities is essential. Backdoor attacks, in
particular, pose a serious threat by implanting hidden triggers in victim
models, which adversaries can later exploit to induce malicious behaviors
during inference. However, current understanding is limited to single-target
attacks, where adversaries must define a fixed malicious behavior (target)
before training, making inference-time adaptability impossible. Given the large
output space of object detection (including object existence prediction,
bounding box estimation, and classification), the feasibility of flexible,
inference-time model control remains unexplored. This paper introduces
AnywhereDoor, a multi-target backdoor attack for object detection. Once
implanted, AnywhereDoor allows adversaries to make objects disappear, fabricate
new ones, or mislabel them, either across all object classes or specific ones,
offering an unprecedented degree of control. This flexibility is enabled by
three key innovations: (i) objective disentanglement to scale the number of
supported targets; (ii) trigger mosaicking to ensure robustness even against
region-based detectors; and (iii) strategic batching to address object-level
data imbalances that hinder manipulation. Extensive experiments demonstrate
that AnywhereDoor grants attackers a high degree of control, improving attack
success rates by 26% compared to adaptations of existing methods for such
flexible control.


http://arxiv.org/abs/2411.14351
Indiscriminate Disruption of Conditional Inference on Multivariate Gaussians. (50%)
William N. Caballero; Matthew LaRosa; Alexander Fisher; Vahid Tarokh
  The multivariate Gaussian distribution underpins myriad operations-research,
decision-analytic, and machine-learning models (e.g., Bayesian optimization,
Gaussian influence diagrams, and variational autoencoders). However, despite
recent advances in adversarial machine learning (AML), inference for Gaussian
models in the presence of an adversary is notably understudied. Therefore, we
consider a self-interested attacker who wishes to disrupt a decisionmaker's
conditional inference and subsequent actions by corrupting a set of evidentiary
variables. To avoid detection, the attacker also desires the attack to appear
plausible wherein plausibility is determined by the density of the corrupted
evidence. We consider white- and grey-box settings such that the attacker has
complete and incomplete knowledge about the decisionmaker's underlying
multivariate Gaussian distribution, respectively. Select instances are shown to
reduce to quadratic and stochastic quadratic programs, and structural
properties are derived to inform solution methods. We assess the impact and
efficacy of these attacks in three examples, including, real estate evaluation,
interest rate estimation and signals processing. Each example leverages an
alternative underlying model, thereby highlighting the attacks' broad
applicability. Through these applications, we also juxtapose the behavior of
the white- and grey-box attacks to understand how uncertainty and structure
affect attacker behavior.


http://arxiv.org/abs/2411.14718
GraphTheft: Quantifying Privacy Risks in Graph Prompt Learning. (4%)
Jiani Zhu; Xi Lin; Yuxin Qi; Qinghua Mao
  Graph Prompt Learning (GPL) represents an innovative approach in graph
representation learning, enabling task-specific adaptations by fine-tuning
prompts without altering the underlying pre-trained model. Despite its growing
prominence, the privacy risks inherent in GPL remain unexplored. In this study,
we provide the first evaluation of privacy leakage in GPL across three attacker
capabilities: black-box attacks when GPL as a service, and scenarios where node
embeddings and prompt representations are accessible to third parties. We
assess GPL's privacy vulnerabilities through Attribute Inference Attacks (AIAs)
and Link Inference Attacks (LIAs), finding that under any capability, attackers
can effectively infer the properties and relationships of sensitive nodes, and
the success rate of inference on some data sets is as high as 98%. Importantly,
while targeted inference attacks on specific prompts (e.g., GPF-plus) maintain
high success rates, our analysis suggests that the prompt-tuning in GPL does
not significantly elevate privacy risks compared to traditional GNNs. To
mitigate these risks, we explored defense mechanisms, identifying that
Laplacian noise perturbation can substantially reduce inference success, though
balancing privacy protection with model performance remains challenging. This
work highlights critical privacy risks in GPL, offering new insights and
foundational directions for future privacy-preserving strategies in graph
learning.


http://arxiv.org/abs/2411.14502
Global Challenge for Safe and Secure LLMs Track 1. (4%)
Xiaojun Jia; Yihao Huang; Yang Liu; Peng Yan Tan; Weng Kuan Yau; Mun-Thye Mak; Xin Ming Sim; Wee Siong Ng; See Kiong Ng; Hanqing Liu; Lifeng Zhou; Huanqian Yan; Xiaobing Sun; Wei Liu; Long Wang; Yiming Qian; Yong Liu; Junxiao Yang; Zhexin Zhang; Leqi Lei; Renmiao Chen; Yida Lu; Shiyao Cui; Zizhou Wang; Shaohua Li; Yan Wang; Rick Siow Mong Goh; Liangli Zhen; Yingjie Zhang; Zhe Zhao
  This paper introduces the Global Challenge for Safe and Secure Large Language
Models (LLMs), a pioneering initiative organized by AI Singapore (AISG) and the
CyberSG R&D Programme Office (CRPO) to foster the development of advanced
defense mechanisms against automated jailbreaking attacks. With the increasing
integration of LLMs in critical sectors such as healthcare, finance, and public
administration, ensuring these models are resilient to adversarial attacks is
vital for preventing misuse and upholding ethical standards. This competition
focused on two distinct tracks designed to evaluate and enhance the robustness
of LLM security frameworks. Track 1 tasked participants with developing
automated methods to probe LLM vulnerabilities by eliciting undesirable
responses, effectively testing the limits of existing safety protocols within
LLMs. Participants were challenged to devise techniques that could bypass
content safeguards across a diverse array of scenarios, from offensive language
to misinformation and illegal activities. Through this process, Track 1 aimed
to deepen the understanding of LLM vulnerabilities and provide insights for
creating more resilient models.


http://arxiv.org/abs/2411.14681
TrojanEdit: Multimodal Backdoor Attack Against Image Editing Model. (3%)
Ji Guo; Peihong Chen; Wenbo Jiang; Xiaolei Wen; Jiaming He; Jiachen Li; Guoming Lu; Aiguo Chen; Hongwei Li
  Multimodal diffusion models for image editing generate outputs conditioned on
both textual instructions and visual inputs, aiming to modify target regions
while preserving the rest of the image. Although diffusion models have been
shown to be vulnerable to backdoor attacks, existing efforts mainly focus on
unimodal generative models and fail to address the unique challenges in
multimodal image editing. In this paper, we present the first study of backdoor
attacks on multimodal diffusion-based image editing models. We investigate the
use of both textual and visual triggers to embed a backdoor that achieves high
attack success rates while maintaining the model's normal functionality.
However, we identify a critical modality bias. Simply combining triggers from
different modalities leads the model to primarily rely on the stronger one,
often the visual modality, which results in a loss of multimodal behavior and
degrades editing quality. To overcome this issue, we propose TrojanEdit, a
backdoor injection framework that dynamically adjusts the gradient
contributions of each modality during training. This allows the model to learn
a truly multimodal backdoor that activates only when both triggers are present.
Extensive experiments on multiple image editing models show that TrojanEdit
successfully integrates triggers from different modalities, achieving balanced
multimodal backdoor learning while preserving clean editing performance and
ensuring high attack effectiveness.


http://arxiv.org/abs/2411.14215
Evaluating the Robustness of Analogical Reasoning in Large Language Models. (1%)
Martha Lewis; Melanie Mitchell
  LLMs have performed well on several reasoning benchmarks, including ones that
test analogical reasoning abilities. However, there is debate on the extent to
which they are performing general abstract reasoning versus employing
non-robust processes, e.g., that overly rely on similarity to pre-training
data. Here we investigate the robustness of analogy-making abilities previously
claimed for LLMs on three of four domains studied by Webb, Holyoak, and Lu
(2023): letter-string analogies, digit matrices, and story analogies. For each
domain we test humans and GPT models on robustness to variants of the original
analogy problems that test the same abstract reasoning abilities but are likely
dissimilar from tasks in the pre-training data. The performance of a system
that uses robust abstract reasoning should not decline substantially on these
variants.
  On simple letter-string analogies, we find that while the performance of
humans remains high for two types of variants we tested, the GPT models'
performance declines sharply. This pattern is less pronounced as the complexity
of these problems is increased, as both humans and GPT models perform poorly on
both the original and variant problems requiring more complex analogies. On
digit-matrix problems, we find a similar pattern but only on one out of the two
types of variants we tested. On story-based analogy problems, we find that,
unlike humans, the performance of GPT models are susceptible to answer-order
effects, and that GPT models also may be more sensitive than humans to
paraphrasing.
  This work provides evidence that LLMs often lack the robustness of zero-shot
human analogy-making, exhibiting brittleness on most of the variations we
tested. More generally, this work points to the importance of carefully
evaluating AI systems not only for accuracy but also robustness when testing
their cognitive capabilities.


http://arxiv.org/abs/2411.14516
Memory Backdoor Attacks on Neural Networks. (1%)
Eden Luzon; Guy Amit; Roy Weiss; Yisroel Mirsky
  Neural networks, such as image classifiers, are frequently trained on
proprietary and confidential datasets. It is generally assumed that once
deployed, the training data remains secure, as adversaries are limited to query
response interactions with the model, where at best, fragments of arbitrary
data can be inferred without any guarantees on their authenticity. In this
paper, we propose the memory backdoor attack, where a model is covertly trained
to memorize specific training samples and later selectively output them when
triggered with an index pattern. What makes this attack unique is that it (1)
works even when the tasks conflict (making a classifier output images), (2)
enables the systematic extraction of training samples from deployed models and
(3) offers guarantees on the extracted authenticity of the data. We demonstrate
the attack on image classifiers, segmentation models, and a large language
model (LLM). We demonstrate the attack on image classifiers, segmentation
models, and a large language model (LLM). With this attack, it is possible to
hide thousands of images and texts in modern vision architectures and LLMs
respectively, all while maintaining model performance. The memory back door
attack poses a significant threat not only to conventional model deployments
but also to federated learning paradigms and other modern frameworks.
Therefore, we suggest an efficient and effective countermeasure that can be
immediately applied and advocate for further work on the topic.


http://arxiv.org/abs/2411.15210
Towards Million-Scale Adversarial Robustness Evaluation With Stronger Individual Attacks. (98%)
Yong Xie; Weijie Zheng; Hanxun Huang; Guangnan Ye; Xingjun Ma
  As deep learning models are increasingly deployed in safety-critical
applications, evaluating their vulnerabilities to adversarial perturbations is
essential for ensuring their reliability and trustworthiness. Over the past
decade, a large number of white-box adversarial robustness evaluation methods
(i.e., attacks) have been proposed, ranging from single-step to multi-step
methods and from individual to ensemble methods. Despite these advances,
challenges remain in conducting meaningful and comprehensive robustness
evaluations, particularly when it comes to large-scale testing and ensuring
evaluations reflect real-world adversarial risks. In this work, we focus on
image classification models and propose a novel individual attack method,
Probability Margin Attack (PMA), which defines the adversarial margin in the
probability space rather than the logits space. We analyze the relationship
between PMA and existing cross-entropy or logits-margin-based attacks, and show
that PMA can outperform the current state-of-the-art individual methods.
Building on PMA, we propose two types of ensemble attacks that balance
effectiveness and efficiency. Furthermore, we create a million-scale dataset,
CC1M, derived from the existing CC3M dataset, and use it to conduct the first
million-scale white-box adversarial robustness evaluation of
adversarially-trained ImageNet models. Our findings provide valuable insights
into the robustness gaps between individual versus ensemble attacks and
small-scale versus million-scale evaluations.


http://arxiv.org/abs/2411.13136
TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models. (96%)
Xin Wang; Kai Chen; Jiaming Zhang; Jingjing Chen; Xingjun Ma
  Large pre-trained Vision-Language Models (VLMs) such as CLIP have
demonstrated excellent zero-shot generalizability across various downstream
tasks. However, recent studies have shown that the inference performance of
CLIP can be greatly degraded by small adversarial perturbations, especially its
visual modality, posing significant safety threats. To mitigate this
vulnerability, in this paper, we propose a novel defense method called
Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness
of CLIP against visual adversarial attacks. TAPT is a test-time defense method
that learns defensive bimodal (textual and visual) prompts to robustify the
inference process of CLIP. Specifically, it is an unsupervised method that
optimizes the defensive prompts for each test sample by minimizing a multi-view
entropy and aligning adversarial-clean distributions. We evaluate the
effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other
zero-shot datasets, demonstrating that it enhances the zero-shot adversarial
robustness of the original CLIP by at least 48.9% against AutoAttack (AA),
while largely maintaining performance on clean examples. Moreover, TAPT
outperforms existing adversarial prompt tuning methods across various
backbones, achieving an average robustness improvement of at least 36.6%.


http://arxiv.org/abs/2411.13116
Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning. (86%)
Zhi Luo; Xiyuan Yang; Pan Zhou; Di Wang
  Manipulating the interaction trajectories between the intelligent agent and
the environment can control the agent's training and behavior, exposing the
potential vulnerabilities of reinforcement learning (RL). For example, in
Cyber-Physical Systems (CPS) controlled by RL, the attacker can manipulate the
actions of the adopted RL to other actions during the training phase, which
will lead to bad consequences. Existing work has studied action-manipulation
attacks in tabular settings, where the states and actions are discrete. As seen
in many up-and-coming RL applications, such as autonomous driving, continuous
action space is widely accepted, however, its action-manipulation attacks have
not been thoroughly investigated yet. In this paper, we consider this crucial
problem in both white-box and black-box scenarios. Specifically, utilizing the
knowledge derived exclusively from trajectories, we propose a black-box attack
algorithm named LCBT, which uses the Monte Carlo tree search method for
efficient action searching and manipulation. Additionally, we demonstrate that
for an agent whose dynamic regret is sub-linearly related to the total number
of steps, LCBT can teach the agent to converge to target policies with only
sublinear attack cost, i.e., $O\left(\mathcal{R}(T) + MH^3K^E\log
(MT)\right)(0<E<1)$, where $H$ is the number of steps per episode, $K$ is the
total number of episodes, $T=KH$ is the total number of steps, $M$ is the
number of subspaces divided in the state space, and $\mathcal{R}(T)$ is the
bound of the RL algorithm's regret. We conduct our proposed attack methods on
three aggressive algorithms: DDPG, PPO, and TD3 in continuous settings, which
show a promising attack performance.


http://arxiv.org/abs/2411.13778
A Survey on Adversarial Robustness of LiDAR-based Machine Learning Perception in Autonomous Vehicles. (86%)
Junae Kim; Amardeep Kaur
  In autonomous driving, the combination of AI and vehicular technology offers
great potential. However, this amalgamation comes with vulnerabilities to
adversarial attacks. This survey focuses on the intersection of Adversarial
Machine Learning (AML) and autonomous systems, with a specific focus on
LiDAR-based systems. We comprehensively explore the threat landscape,
encompassing cyber-attacks on sensors and adversarial perturbations.
Additionally, we investigate defensive strategies employed in countering these
threats. This paper endeavors to present a concise overview of the challenges
and advances in securing autonomous driving systems against adversarial
threats, emphasizing the need for robust defenses to ensure safety and
security.


http://arxiv.org/abs/2411.15222
Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillation. (68%)
Ke Wuhan University Zhao; Huayang Wuhan University Huang; Miao Wuhan University Li; Yu Wuhan University Wu
  Language-conditioned robotic learning has significantly enhanced robot
adaptability by enabling a single model to execute diverse tasks in response to
verbal commands. Despite these advancements, security vulnerabilities within
this domain remain largely unexplored. This paper addresses this gap by
proposing a novel adversarial prompt attack tailored to language-conditioned
robotic models. Our approach involves crafting a universal adversarial prefix
that induces the model to perform unintended actions when added to any original
prompt. We demonstrate that existing adversarial techniques exhibit limited
effectiveness when directly transferred to the robotic domain due to the
inherent robustness of discretized robotic action spaces. To overcome this
challenge, we propose to optimize adversarial prefixes based on continuous
action representations, circumventing the discretization process. Additionally,
we identify the beneficial impact of intermediate features on adversarial
attacks and leverage the negative gradient of intermediate self-attention
features to further enhance attack efficacy. Extensive experiments on VIMA
models across 13 robot manipulation tasks validate the superiority of our
method over existing approaches and demonstrate its transferability across
different model variants.


http://arxiv.org/abs/2411.13553
AI-generated Image Detection: Passive or Watermark? (31%)
Moyang Guo; Yuepeng Hu; Zhengyuan Jiang; Zeyu Li; Amir Sadovnik; Arka Daw; Neil Gong
  While text-to-image models offer numerous benefits, they also pose
significant societal risks. Detecting AI-generated images is crucial for
mitigating these risks. Detection methods can be broadly categorized into
passive and watermark-based approaches: passive detectors rely on artifacts
present in AI-generated images, whereas watermark-based detectors proactively
embed watermarks into such images. A key question is which type of detector
performs better in terms of effectiveness, robustness, and efficiency. However,
the current literature lacks a comprehensive understanding of this issue. In
this work, we aim to bridge that gap by developing ImageDetectBench, the first
comprehensive benchmark to compare the effectiveness, robustness, and
efficiency of passive and watermark-based detectors. Our benchmark includes
four datasets, each containing a mix of AI-generated and non-AI-generated
images. We evaluate five passive detectors and four watermark-based detectors
against eight types of common perturbations and three types of adversarial
perturbations. Our benchmark results reveal several interesting findings. For
instance, watermark-based detectors consistently outperform passive detectors,
both in the presence and absence of perturbations. Based on these insights, we
provide recommendations for detecting AI-generated images, e.g., when both
types of detectors are applicable, watermark-based detectors should be the
preferred choice. Our code and data are publicly available at
https://github.com/moyangkuo/ImageDetectBench.git.


http://arxiv.org/abs/2411.13459
SoK: A Systems Perspective on Compound AI Threats and Countermeasures. (12%)
Sarbartha Banerjee; Prateek Sahu; Mulong Luo; Anjo Vahldiek-Oberwagner; Neeraja J. Yadwadkar; Mohit Tiwari
  Large language models (LLMs) used across enterprises often use proprietary
models and operate on sensitive inputs and data. The wide range of attack
vectors identified in prior research - targeting various software and hardware
components used in training and inference - makes it extremely challenging to
enforce confidentiality and integrity policies.
  As we advance towards constructing compound AI inference pipelines that
integrate multiple large language models (LLMs), the attack surfaces expand
significantly. Attackers now focus on the AI algorithms as well as the software
and hardware components associated with these systems. While current research
often examines these elements in isolation, we find that combining cross-layer
attack observations can enable powerful end-to-end attacks with minimal
assumptions about the threat model. Given, the sheer number of existing attacks
at each layer, we need a holistic and systemized understanding of different
attack vectors at each layer.
  This SoK discusses different software and hardware attacks applicable to
compound AI systems and demonstrates how combining multiple attack mechanisms
can reduce the threat model assumptions required for an isolated attack. Next,
we systematize the ML attacks in lines with the Mitre Att&ck framework to
better position each attack based on the threat model. Finally, we outline the
existing countermeasures for both software and hardware layers and discuss the
necessity of a comprehensive defense strategy to enable the secure and
high-performance deployment of compound AI systems.


http://arxiv.org/abs/2411.13144
CopyrightMeter: Revisiting Copyright Protection in Text-to-image Models. (12%)
Naen Xu; Changjiang Li; Tianyu Du; Minxi Li; Wenjie Luo; Jiacheng Liang; Yuyuan Li; Xuhong Zhang; Meng Han; Jianwei Yin; Ting Wang
  Text-to-image diffusion models have emerged as powerful tools for generating
high-quality images from textual descriptions. However, their increasing
popularity has raised significant copyright concerns, as these models can be
misused to reproduce copyrighted content without authorization. In response,
recent studies have proposed various copyright protection methods, including
adversarial perturbation, concept erasure, and watermarking techniques.
However, their effectiveness and robustness against advanced attacks remain
largely unexplored. Moreover, the lack of unified evaluation frameworks has
hindered systematic comparison and fair assessment of different approaches. To
bridge this gap, we systematize existing copyright protection methods and
attacks, providing a unified taxonomy of their design spaces. We then develop
CopyrightMeter, a unified evaluation framework that incorporates 17
state-of-the-art protections and 16 representative attacks. Leveraging
CopyrightMeter, we comprehensively evaluate protection methods across multiple
dimensions, thereby uncovering how different design choices impact fidelity,
efficacy, and resilience under attacks. Our analysis reveals several key
findings: (i) most protections (16/17) are not resilient against attacks; (ii)
the "best" protection varies depending on the target priority; (iii) more
advanced attacks significantly promote the upgrading of protections. These
insights provide concrete guidance for developing more robust protection
methods, while its unified evaluation protocol establishes a standard benchmark
for future copyright protection research in text-to-image generation.


http://arxiv.org/abs/2411.13425
WaterPark: A Robustness Assessment of Language Model Watermarking. (1%)
Jiacheng Liang; Zian Wang; Lauren Hong; Shouling Ji; Ting Wang
  Various watermarking methods (``watermarkers'') have been proposed to
identify LLM-generated texts; yet, due to the lack of unified evaluation
platforms, many critical questions remain under-explored: i) What are the
strengths/limitations of various watermarkers, especially their attack
robustness? ii) How do various design choices impact their robustness? iii) How
to optimally operate watermarkers in adversarial environments? To fill this
gap, we systematize existing LLM watermarkers and watermark removal attacks,
mapping out their design spaces. We then develop WaterPark, a unified platform
that integrates 10 state-of-the-art watermarkers and 12 representative attacks.
More importantly, by leveraging WaterPark, we conduct a comprehensive
assessment of existing watermarkers, unveiling the impact of various design
choices on their attack robustness. We further explore the best practices to
operate watermarkers in adversarial environments. We believe our study sheds
light on current LLM watermarking techniques while WaterPark serves as a
valuable testbed to facilitate future research.


http://arxiv.org/abs/2411.12473
NMT-Obfuscator Attack: Ignore a sentence in translation with only one word. (99%)
Sahar Sadrizadeh; César Descalzo; Ljiljana Dolamic; Pascal Frossard
  Neural Machine Translation systems are used in diverse applications due to
their impressive performance. However, recent studies have shown that these
systems are vulnerable to carefully crafted small perturbations to their
inputs, known as adversarial attacks. In this paper, we propose a new type of
adversarial attack against NMT models. In this attack, we find a word to be
added between two sentences such that the second sentence is ignored and not
translated by the NMT model. The word added between the two sentences is such
that the whole adversarial text is natural in the source language. This type of
attack can be harmful in practical scenarios since the attacker can hide
malicious information in the automatic translation made by the target NMT
model. Our experiments show that different NMT models and translation tasks are
vulnerable to this type of attack. Our attack can successfully force the NMT
models to ignore the second part of the input in the translation for more than
50% of all cases while being able to maintain low perplexity for the whole
input.


http://arxiv.org/abs/2411.12575
Stochastic BIQA: Median Randomized Smoothing for Certified Blind Image Quality Assessment. (75%)
Ekaterina Shumitskaya; Mikhail Pautov; Dmitriy Vatolin; Anastasia Antsiferova
  Most modern No-Reference Image-Quality Assessment (NR-IQA) metrics are based
on neural networks vulnerable to adversarial attacks. Attacks on such metrics
lead to incorrect image/video quality predictions, which poses significant
risks, especially in public benchmarks. Developers of image processing
algorithms may unfairly increase the score of a target IQA metric without
improving the actual quality of the adversarial image. Although some empirical
defenses for IQA metrics were proposed, they do not provide theoretical
guarantees and may be vulnerable to adaptive attacks. This work focuses on
developing a provably robust no-reference IQA metric. Our method is based on
Median Smoothing (MS) combined with an additional convolution denoiser with
ranking loss to improve the SROCC and PLCC scores of the defended IQA metric.
Compared with two prior methods on three datasets, our method exhibited
superior SROCC and PLCC scores while maintaining comparable certified
guarantees.


http://arxiv.org/abs/2411.12701
When Backdoors Speak: Understanding LLM Backdoor Attacks Through Model-Generated Explanations. (3%)
Huaizhi Ge; Yiming Li; Qifan Wang; Yongfeng Zhang; Ruixiang Tang
  Large Language Models (LLMs) are vulnerable to backdoor attacks, where hidden
triggers can maliciously manipulate model behavior. While several backdoor
attack methods have been proposed, the mechanisms by which backdoor functions
operate in LLMs remain underexplored. In this paper, we move beyond attacking
LLMs and investigate backdoor functionality through the novel lens of natural
language explanations. Specifically, we leverage LLMs' generative capabilities
to produce human-understandable explanations for their decisions, allowing us
to compare explanations for clean and poisoned samples. We explore various
backdoor attacks and embed the backdoor into LLaMA models for multiple tasks.
Our experiments show that backdoored models produce higher-quality explanations
for clean data compared to poisoned data, while generating significantly more
consistent explanations for poisoned data than for clean data. We further
analyze the explanation generation process, revealing that at the token level,
the explanation token of poisoned samples only appears in the final few
transformer layers of the LLM. At the sentence level, attention dynamics
indicate that poisoned inputs shift attention from the input context when
generating the explanation. These findings deepen our understanding of backdoor
attack mechanisms in LLMs and offer a framework for detecting such
vulnerabilities through explainability techniques, contributing to the
development of more secure LLMs.


http://arxiv.org/abs/2411.12071
Theoretical Corrections and the Leveraging of Reinforcement Learning to Enhance Triangle Attack. (99%)
Nicole Meng; Caleb Manicke; David Chen; Yingjie Lao; Caiwen Ding; Pengyu Hong; Kaleel Mahmood
  Adversarial examples represent a serious issue for the application of machine
learning models in many sensitive domains. For generating adversarial examples,
decision based black-box attacks are one of the most practical techniques as
they only require query access to the model. One of the most recently proposed
state-of-the-art decision based black-box attacks is Triangle Attack (TA). In
this paper, we offer a high-level description of TA and explain potential
theoretical limitations. We then propose a new decision based black-box attack,
Triangle Attack with Reinforcement Learning (TARL). Our new attack addresses
the limits of TA by leveraging reinforcement learning. This creates an attack
that can achieve similar, if not better, attack accuracy than TA with half as
many queries on state-of-the-art classifiers and defenses across ImageNet and
CIFAR-10.


http://arxiv.org/abs/2411.11389
Adapting to Cyber Threats: A Phishing Evolution Network (PEN) Framework for Phishing Generation and Analyzing Evolution Patterns using Large Language Models. (87%)
Fengchao Chen; Tingmin Wu; Van Nguyen; Shuo Wang; Hongsheng Hu; Alsharif Abuadbba; Carsten Rudolph
  Phishing remains a pervasive cyber threat, as attackers craft deceptive
emails to lure victims into revealing sensitive information. While Artificial
Intelligence (AI), particularly deep learning, has become a key component in
defending against phishing attacks, these approaches face critical limitations.
The scarcity of publicly available, diverse, and updated data, largely due to
privacy concerns, constrains their effectiveness. As phishing tactics evolve
rapidly, models trained on limited, outdated data struggle to detect new,
sophisticated deception strategies, leaving systems vulnerable to an
ever-growing array of attacks. Addressing this gap is essential to
strengthening defenses in an increasingly hostile cyber landscape. To address
this gap, we propose the Phishing Evolution Network (PEN), a framework
leveraging large language models (LLMs) and adversarial training mechanisms to
continuously generate high quality and realistic diverse phishing samples, and
analyze features of LLM-provided phishing to understand evolving phishing
patterns. We evaluate the quality and diversity of phishing samples generated
by PEN and find that it produces over 80% realistic phishing samples,
effectively expanding phishing datasets across seven dominant types. These
PEN-generated samples enhance the performance of current phishing detectors,
leading to a 40% improvement in detection accuracy. Additionally, the use of
PEN significantly boosts model robustness, reducing detectors' sensitivity to
perturbations by up to 60%, thereby decreasing attack success rates under
adversarial conditions. When we analyze the phishing patterns that are used in
LLM-generated phishing, the cognitive complexity and the tone of time
limitation are detected with statistically significant differences compared
with existing phishing.


http://arxiv.org/abs/2411.12768
CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization. (75%)
Nay Myat Min; Long H. Pham; Yige Li; Jun Sun
  Large Language Models (LLMs) are vulnerable to backdoor attacks that
manipulate outputs via hidden triggers. Existing defense methods--designed for
vision/text classification tasks--fail for text generation. We propose Internal
Consistency Regularization (CROW), a defense leveraging the observation that
backdoored models exhibit unstable layer-wise hidden representations when
triggered, while clean models show smooth transitions. CROW enforces
consistency across layers via adversarial perturbations and regularization
during finetuning, neutralizing backdoors without requiring clean reference
models or trigger knowledge--only a small clean dataset. Experiments across
Llama-2 (7B, 13B), CodeLlama (7B, 13B), and Mistral-7B demonstrate CROW's
effectiveness: it achieves significant reductions in attack success rates
across diverse backdoor strategies (sentiment steering, targeted refusal, code
injection) while preserving generative performance. CROW's
architecture-agnostic design enables practical deployment.


http://arxiv.org/abs/2411.12220
DeTrigger: A Gradient-Centric Approach to Backdoor Attack Mitigation in Federated Learning. (75%)
Kichang Lee; Yujin Shin; Jonghyuk Yun; Jun Han; JeongGil Ko
  Federated Learning (FL) enables collaborative model training across
distributed devices while preserving local data privacy, making it ideal for
mobile and embedded systems. However, the decentralized nature of FL also opens
vulnerabilities to model poisoning attacks, particularly backdoor attacks,
where adversaries implant trigger patterns to manipulate model predictions. In
this paper, we propose DeTrigger, a scalable and efficient backdoor-robust
federated learning framework that leverages insights from adversarial attack
methodologies. By employing gradient analysis with temperature scaling,
DeTrigger detects and isolates backdoor triggers, allowing for precise model
weight pruning of backdoor activations without sacrificing benign model
knowledge. Extensive evaluations across four widely used datasets demonstrate
that DeTrigger achieves up to 251x faster detection than traditional methods
and mitigates backdoor attacks by up to 98.9%, with minimal impact on global
model accuracy. Our findings establish DeTrigger as a robust and scalable
solution to protect federated learning environments against sophisticated
backdoor threats.


http://arxiv.org/abs/2411.11525
Reliable Poisoned Sample Detection against Backdoor Attacks Enhanced by Sharpness Aware Minimization. (50%)
Mingda Zhang; Mingli Zhu; Zihao Zhu; Baoyuan Wu
  Backdoor attack has been considered as a serious security threat to deep
neural networks (DNNs). Poisoned sample detection (PSD) that aims at filtering
out poisoned samples from an untrustworthy training dataset has shown very
promising performance for defending against data poisoning based backdoor
attacks. However, we observe that the detection performance of many advanced
methods is likely to be unstable when facing weak backdoor attacks, such as low
poisoning ratio or weak trigger strength. To further verify this observation,
we make a statistical investigation among various backdoor attacks and poisoned
sample detections, showing a positive correlation between backdoor effect and
detection performance. It inspires us to strengthen the backdoor effect to
enhance detection performance. Since we cannot achieve that goal via directly
manipulating poisoning ratio or trigger strength, we propose to train one model
using the Sharpness-Aware Minimization (SAM) algorithm, rather than the vanilla
training algorithm. We also provide both empirical and theoretical analysis
about how SAM training strengthens the backdoor effect. Then, this SAM trained
model can be seamlessly integrated with any off-the-shelf PSD method that
extracts discriminative features from the trained model for detection, called
SAM-enhanced PSD. Extensive experiments on several benchmark datasets show the
reliable detection performance of the proposed method against both weak and
strong backdoor attacks, with significant improvements against various attacks
($+34.38\%$ TPR on average), over the conventional PSD methods (i.e., without
SAM enhancement). Overall, this work provides new insights about PSD and
proposes a novel approach that can complement existing detection methods, which
may inspire more in-depth explorations in this field.


http://arxiv.org/abs/2411.11677
Few-shot Model Extraction Attacks against Sequential Recommender Systems. (38%)
Hui Zhang; Fu Liu
  Among adversarial attacks against sequential recommender systems, model
extraction attacks represent a method to attack sequential recommendation
models without prior knowledge. Existing research has primarily concentrated on
the adversary's execution of black-box attacks through data-free model
extraction. However, a significant gap remains in the literature concerning the
development of surrogate models by adversaries with access to few-shot raw data
(10\% even less). That is, the challenge of how to construct a surrogate model
with high functional similarity within the context of few-shot data scenarios
remains an issue that requires resolution.This study addresses this gap by
introducing a novel few-shot model extraction framework against sequential
recommenders, which is designed to construct a superior surrogate model with
the utilization of few-shot data. The proposed few-shot model extraction
framework is comprised of two components: an autoregressive augmentation
generation strategy and a bidirectional repair loss-facilitated model
distillation procedure. Specifically, to generate synthetic data that closely
approximate the distribution of raw data, autoregressive augmentation
generation strategy integrates a probabilistic interaction sampler to extract
inherent dependencies and a synthesis determinant signal module to characterize
user behavioral patterns. Subsequently, bidirectional repair loss, which target
the discrepancies between the recommendation lists, is designed as auxiliary
loss to rectify erroneous predictions from surrogate models, transferring
knowledge from the victim model to the surrogate model effectively. Experiments
on three datasets show that the proposed few-shot model extraction framework
yields superior surrogate models.


http://arxiv.org/abs/2411.11434
CLUE-MARK: Watermarking Diffusion Models using CLWE. (26%)
Kareem Shehata; Aashish Kolluri; Prateek Saxena
  As AI-generated images become widespread, reliable watermarking is essential
for content verification, copyright enforcement, and combating disinformation.
Existing techniques rely on heuristic approaches and lack formal guarantees of
undetectability, making them vulnerable to steganographic attacks that can
expose or erase the watermark. Additionally, these techniques often degrade
output quality by introducing perceptible changes, which is not only
undesirable but an important barrier to adoption in practice.
  In this work, we introduce CLUE-Mark, the first provably undetectable
watermarking scheme for diffusion models. CLUE-Mark requires no changes to the
model being watermarked, is computationally efficient, and because it is
provably undetectable is guaranteed to have no impact on model output quality.
Our approach leverages the Continuous Learning With Errors (CLWE) problem -- a
cryptographically hard lattice problem -- to embed watermarks in the latent
noise vectors used by diffusion models. By proving undetectability via
reduction to a cryptographically hard problem we ensure not only that the
watermark is imperceptible to human observers or adhoc heuristics, but to
\emph{any} efficient detector that does not have the secret key. CLUE-Mark
allows multiple keys to be embedded, enabling traceability of images to
specific users without altering model parameters. Empirical evaluations on
state-of-the-art diffusion models confirm that CLUE-Mark achieves high
recoverability, preserves image quality, and is robust to minor perturbations
such JPEG compression and brightness adjustments. Uniquely, CLUE-Mark cannot be
detected nor removed by recent steganographic attacks.


http://arxiv.org/abs/2411.11407
The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models. (13%)
Xikang Yang; Xuehai Tang; Jizhong Han; Songlin Hu
  The widespread deployment of large language models (LLMs) across various
domains has showcased their immense potential while exposing significant safety
vulnerabilities. A major concern is ensuring that LLM-generated content aligns
with human values. Existing jailbreak techniques reveal how this alignment can
be compromised through specific prompts or adversarial suffixes. In this study,
we introduce a new threat: LLMs' bias toward authority. While this inherent
bias can improve the quality of outputs generated by LLMs, it also introduces a
potential vulnerability, increasing the risk of producing harmful content.
Notably, the biases in LLMs is the varying levels of trust given to different
types of authoritative information in harmful queries. For example, malware
development often favors trust GitHub. To better reveal the risks with LLM, we
propose DarkCite, an adaptive authority citation matcher and generator designed
for a black-box setting. DarkCite matches optimal citation types to specific
risk types and generates authoritative citations relevant to harmful
instructions, enabling more effective jailbreak attacks on aligned LLMs.Our
experiments show that DarkCite achieves a higher attack success rate (e.g.,
LLama-2 at 76% versus 68%) than previous methods. To counter this risk, we
propose an authenticity and harm verification defense strategy, raising the
average defense pass rate (DPR) from 11% to 74%. More importantly, the ability
to link citations to the content they encompass has become a foundational
function in LLMs, amplifying the influence of LLMs' bias toward authority.


http://arxiv.org/abs/2411.11795
Exploring adversarial robustness of JPEG AI: methodology, comparison and new methods. (8%)
Egor Kovalev; Georgii Bychkov; Khaled Abud; Aleksandr Gushchin; Anna Chistyakova; Sergey Lavrushkin; Dmitriy Vatolin; Anastasia Antsiferova
  Adversarial robustness of neural networks is an increasingly important area
of research, combining studies on computer vision models, large language models
(LLMs), and others. With the release of JPEG AI - the first standard for
end-to-end neural image compression (NIC) methods - the question of its
robustness has become critically significant. JPEG AI is among the first
international, real-world applications of neural-network-based models to be
embedded in consumer devices. However, research on NIC robustness has been
limited to open-source codecs and a narrow range of attacks. This paper
proposes a new methodology for measuring NIC robustness to adversarial attacks.
We present the first large-scale evaluation of JPEG AI's robustness, comparing
it with other NIC models. Our evaluation results and code are publicly
available online (link is hidden for a blind review).


http://arxiv.org/abs/2411.13587
Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics. (89%)
Taowen Wang; Cheng Han; James Chenhao Liang; Wenhao Yang; Dongfang Liu; Luna Xinyu Zhang; Qifan Wang; Jiebo Luo; Ruixiang Tang
  Recently in robotics, Vision-Language-Action (VLA) models have emerged as a
transformative approach, enabling robots to execute complex tasks by
integrating visual and linguistic inputs within an end-to-end learning
framework. While VLA models offer significant capabilities, they also introduce
new attack surfaces, making them vulnerable to adversarial attacks. With these
vulnerabilities largely unexplored, this paper systematically quantifies the
robustness of VLA-based robotic systems. Recognizing the unique demands of
robotic execution, our attack objectives target the inherent spatial and
functional characteristics of robotic systems. In particular, we introduce two
untargeted attack objectives that leverage spatial foundations to destabilize
robotic actions, and a targeted attack objective that manipulates the robotic
trajectory. Additionally, we design an adversarial patch generation approach
that places a small, colorful patch within the camera's view, effectively
executing the attack in both digital and physical environments. Our evaluation
reveals a marked degradation in task success rates, with up to a 100\%
reduction across a suite of simulated robotic tasks, highlighting critical
security gaps in current VLA architectures. By unveiling these vulnerabilities
and proposing actionable evaluation metrics, we advance both the understanding
and enhancement of safety for VLA-based robotic systems, underscoring the
necessity for continuously developing robust defense strategies prior to
physical-world deployments.


http://arxiv.org/abs/2411.11114
JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit. (62%)
Zeqing He; Zhibo Wang; Zhixuan Chu; Huiyu Xu; Rui Zheng; Kui Ren; Chun Chen
  Despite the outstanding performance of Large language models (LLMs) in
diverse tasks, they are vulnerable to jailbreak attacks, wherein adversarial
prompts are crafted to bypass their security mechanisms and elicit unexpected
responses.Although jailbreak attacks are prevalent, the understanding of their
underlying mechanisms remains limited. Recent studies have explain typical
jailbreaking behavior (e.g., the degree to which the model refuses to respond)
of LLMs by analyzing the representation shifts in their latent space caused by
jailbreak prompts or identifying key neurons that contribute to the success of
these attacks. However, these studies neither explore diverse jailbreak
patterns nor provide a fine-grained explanation from the failure of circuit to
the changes of representational, leaving significant gaps in uncovering the
jailbreak mechanism. In this paper, we propose JailbreakLens, an interpretation
framework that analyzes jailbreak mechanisms from both representation (which
reveals how jailbreaks alter the model's harmfulness perception) and circuit
perspectives (which uncovers the causes of these deceptions by identifying key
circuits contributing to the vulnerability), tracking their evolution
throughout the entire response generation process. We then conduct an in-depth
evaluation of jailbreak behavior on four mainstream LLMs under seven jailbreak
strategies. Our evaluation finds that jailbreak prompts amplify components that
reinforce affirmative responses while suppressing those that produce refusal.
Although this manipulation shifts model representations toward safe clusters to
deceive the LLM, leading it to provide detailed responses instead of refusals,
it still produce abnormal activation which can be caught in the circuit
analysis.


http://arxiv.org/abs/2411.11200
Countering Backdoor Attacks in Image Recognition: A Survey and Evaluation of Mitigation Strategies. (22%)
Kealan Dunnett; Reza Arablouei; Dimity Miller; Volkan Dedeoglu; Raja Jurdak
  The widespread adoption of deep learning across various industries has
introduced substantial challenges, particularly in terms of model
explainability and security. The inherent complexity of deep learning models,
while contributing to their effectiveness, also renders them susceptible to
adversarial attacks. Among these, backdoor attacks are especially concerning,
as they involve surreptitiously embedding specific triggers within training
data, causing the model to exhibit aberrant behavior when presented with input
containing the triggers. Such attacks often exploit vulnerabilities in
outsourced processes, compromising model integrity without affecting
performance on clean (trigger-free) input data. In this paper, we present a
comprehensive review of existing mitigation strategies designed to counter
backdoor attacks in image recognition. We provide an in-depth analysis of the
theoretical foundations, practical efficacy, and limitations of these
approaches. In addition, we conduct an extensive benchmarking of sixteen
state-of-the-art approaches against eight distinct backdoor attacks, utilizing
three datasets, four model architectures, and three poisoning ratios. Our
results, derived from 122,236 individual experiments, indicate that while many
approaches provide some level of protection, their performance can vary
considerably. Furthermore, when compared to two seminal approaches, most newer
approaches do not demonstrate substantial improvements in overall performance
or consistency across diverse settings. Drawing from these findings, we propose
potential directions for developing more effective and generalizable defensive
mechanisms in the future.


http://arxiv.org/abs/2411.11195
SoK: Unifying Cybersecurity and Cybersafety of Multimodal Foundation Models with an Information Theory Approach. (9%)
Ruoxi Sun; Jiamin Chang; Hammond Pearce; Chaowei Xiao; Bo Li; Qi Wu; Surya Nepal; Minhui Xue
  Multimodal foundation models (MFMs) represent a significant advancement in
artificial intelligence, combining diverse data modalities to enhance learning
and understanding across a wide range of applications. However, this
integration also brings unique safety and security challenges. In this paper,
we conceptualize cybersafety and cybersecurity in the context of multimodal
learning and present a comprehensive Systematization of Knowledge (SoK) to
unify these concepts in MFMs, identifying key threats to these models. We
propose a taxonomy framework grounded in information theory, evaluating and
categorizing threats through the concepts of channel capacity, signal, noise,
and bandwidth. This approach provides a novel framework that unifies model
safety and system security in MFMs, offering a more comprehensive and
actionable understanding of the risks involved. We used this to explore
existing defense mechanisms, and identified gaps in current research -
particularly, a lack of protection for alignment between modalities and a need
for more systematic defense methods. Our work contributes to a deeper
understanding of the security and safety landscape in MFMs, providing
researchers and practitioners with valuable insights for improving the
robustness and reliability of these models.


http://arxiv.org/abs/2411.11006
BackdoorMBTI: A Backdoor Learning Multimodal Benchmark Tool Kit for Backdoor Defense Evaluation. (2%)
Haiyang Yu; Tian Xie; Jiaping Gui; Pengyang Wang; Ping Yi; Yue Wu
  Over the past few years, the emergence of backdoor attacks has presented
significant challenges to deep learning systems, allowing attackers to insert
backdoors into neural networks. When data with a trigger is processed by a
backdoor model, it can lead to mispredictions targeted by attackers, whereas
normal data yields regular results. The scope of backdoor attacks is expanding
beyond computer vision and encroaching into areas such as natural language
processing and speech recognition. Nevertheless, existing backdoor defense
methods are typically tailored to specific data modalities, restricting their
application in multimodal contexts. While multimodal learning proves highly
applicable in facial recognition, sentiment analysis, action recognition,
visual question answering, the security of these models remains a crucial
concern. Specifically, there are no existing backdoor benchmarks targeting
multimodal applications or related tasks.
  In order to facilitate the research in multimodal backdoor, we introduce
BackdoorMBTI, the first backdoor learning toolkit and benchmark designed for
multimodal evaluation across three representative modalities from eleven
commonly used datasets. BackdoorMBTI provides a systematic backdoor learning
pipeline, encompassing data processing, data poisoning, backdoor training, and
evaluation. The generated poison datasets and backdoor models enable detailed
evaluation of backdoor defenses. Given the diversity of modalities,
BackdoorMBTI facilitates systematic evaluation across different data types.
Furthermore, BackdoorMBTI offers a standardized approach to handling practical
factors in backdoor learning, such as issues related to data quality and
erroneous labels. We anticipate that BackdoorMBTI will expedite future research
in backdoor defense methods within a multimodal context. Code is available at
https://github.com/SJTUHaiyangYu/BackdoorMBTI.


http://arxiv.org/abs/2411.11144
CLMIA: Membership Inference Attacks via Unsupervised Contrastive Learning. (2%)
Depeng School of Computer Science and Technology, Anhui University Chen; Xiao School of Computer Science and Technology, Anhui University Liu; Jie School of Computer Science and Technology, Anhui University Cui; Hong School of Computer Science and Technology, Anhui University Zhong
  Since machine learning model is often trained on a limited data set, the
model is trained multiple times on the same data sample, which causes the model
to memorize most of the training set data. Membership Inference Attacks (MIAs)
exploit this feature to determine whether a data sample is used for training a
machine learning model. However, in realistic scenarios, it is difficult for
the adversary to obtain enough qualified samples that mark accurate identity
information, especially since most samples are non-members in real world
applications. To address this limitation, in this paper, we propose a new
attack method called CLMIA, which uses unsupervised contrastive learning to
train an attack model without using extra membership status information.
Meanwhile, in CLMIA, we require only a small amount of data with known
membership status to fine-tune the attack model. Experimental results
demonstrate that CLMIA performs better than existing attack methods for
different datasets and model structures, especially with data with less marked
identity information. In addition, we experimentally find that the attack
performs differently for different proportions of labeled identity information
for member and non-member data. More analysis proves that our attack method
performs better with less labeled identity information, which applies to more
realistic scenarios.


http://arxiv.org/abs/2411.11132
Variational Bayesian Bow tie Neural Networks with Shrinkage. (1%)
Alisa Sheinkman; Sara Wade
  Despite the dominant role of deep models in machine learning, limitations
persist, including overconfident predictions, susceptibility to adversarial
attacks, and underestimation of variability in predictions. The Bayesian
paradigm provides a natural framework to overcome such issues and has become
the gold standard for uncertainty estimation with deep models, also providing
improved accuracy and a framework for tuning critical hyperparameters. However,
exact Bayesian inference is challenging, typically involving variational
algorithms that impose strong independence and distributional assumptions.
Moreover, existing methods are sensitive to the architectural choice of the
network. We address these issues by focusing on a stochastic relaxation of the
standard feed-forward rectified neural network and using sparsity-promoting
priors on the weights of the neural network for increased robustness to
architectural design. Thanks to Polya-Gamma data augmentation tricks, which
render a conditionally linear and Gaussian model, we derive a fast, approximate
variational inference algorithm that avoids distributional assumptions and
independence across layers. Suitable strategies to further improve scalability
and account for multimodality are considered.


http://arxiv.org/abs/2411.10868
Destabilizing a Social Network Model via Intrinsic Feedback Vulnerabilities. (1%)
Lane H. Rogers; Emma J. Reid; Robert A. Bridges
  Social influence plays a significant role in shaping individual sentiments
and actions, particularly in a world of ubiquitous digital interconnection. The
rapid development of generative AI has engendered well-founded concerns
regarding the potential scalable implementation of radicalization techniques in
social media. Motivated by these developments, we present a case study
investigating the effects of small but intentional perturbations on a simple
social network. We employ Taylor's classic model of social influence and tools
from robust control theory (most notably the Dynamical Structure Function
(DSF)), to identify perturbations that qualitatively alter the system's
behavior while remaining as unobtrusive as possible. We examine two such
scenarios: perturbations to an existing link and perturbations that introduce a
new link to the network. In each case, we identify destabilizing perturbations
of minimal norm and simulate their effects. Remarkably, we find that small but
targeted alterations to network structure may lead to the radicalization of all
agents, exhibiting the potential for large-scale shifts in collective behavior
to be triggered by comparatively minuscule adjustments in social influence.
Given that this method of identifying perturbations that are innocuous yet
destabilizing applies to any suitable dynamical system, our findings emphasize
a need for similar analyses to be carried out on real systems (e.g., real
social networks), to identify the places where such dynamics may already exist.


http://arxiv.org/abs/2411.10565
Comparing Robustness Against Adversarial Attacks in Code Generation: LLM-Generated vs. Human-Written. (99%)
Md Abdul Awal; Mrigank Rochan; Chanchal K. Roy
  Thanks to the widespread adoption of Large Language Models (LLMs) in software
engineering research, the long-standing dream of automated code generation has
become a reality on a large scale. Nowadays, LLMs such as GitHub Copilot and
ChatGPT are extensively used in code generation for enterprise and open-source
software development and maintenance. Despite their unprecedented successes in
code generation, research indicates that codes generated by LLMs exhibit
vulnerabilities and security issues. Several studies have been conducted to
evaluate code generated by LLMs, considering various aspects such as security,
vulnerability, code smells, and robustness. While some studies have compared
the performance of LLMs with that of humans in various software engineering
tasks, there's a notable gap in research: no studies have directly compared
human-written and LLM-generated code for their robustness analysis. To fill
this void, this paper introduces an empirical study to evaluate the adversarial
robustness of Pre-trained Models of Code (PTMCs) fine-tuned on code written by
humans and generated by LLMs against adversarial attacks for software clone
detection. These attacks could potentially undermine software security and
reliability. We consider two datasets, two state-of-the-art PTMCs, two
robustness evaluation criteria, and three metrics to use in our experiments.
Regarding effectiveness criteria, PTMCs fine-tuned on human-written code always
demonstrate more robustness than those fine-tuned on LLMs-generated code. On
the other hand, in terms of adversarial code quality, in 75% experimental
combinations, PTMCs fine-tuned on the human-written code exhibit more
robustness than the PTMCs fine-tuned on the LLMs-generated code.


http://arxiv.org/abs/2411.10174
A Hard-Label Cryptanalytic Extraction of Non-Fully Connected Deep Neural Networks using Side-Channel Attacks. (98%)
Benoit Coqueret; Mathieu Carbone; Olivier Sentieys; Gabriel Zaid
  During the past decade, Deep Neural Networks (DNNs) proved their value on a
large variety of subjects. However despite their high value and public
accessibility, the protection of the intellectual property of DNNs is still an
issue and an emerging research field. Recent works have successfully extracted
fully-connected DNNs using cryptanalytic methods in hard-label settings,
proving that it was possible to copy a DNN with high fidelity, i.e., high
similitude in the output predictions. However, the current cryptanalytic
attacks cannot target complex, i.e., not fully connected, DNNs and are limited
to special cases of neurons present in deep networks.
  In this work, we introduce a new end-to-end attack framework designed for
model extraction of embedded DNNs with high fidelity. We describe a new
black-box side-channel attack which splits the DNN in several linear parts for
which we can perform cryptanalytic extraction and retrieve the weights in
hard-label settings. With this method, we are able to adapt cryptanalytic
extraction, for the first time, to non-fully connected DNNs, while maintaining
a high fidelity. We validate our contributions by targeting several
architectures implemented on a microcontroller unit, including a Multi-Layer
Perceptron (MLP) of 1.7 million parameters and a shortened MobileNetv1. Our
framework successfully extracts all of these DNNs with high fidelity (88.4% for
the MobileNetv1 and 93.2% for the MLP). Furthermore, we use the stolen model to
generate adversarial examples and achieve close to white-box performance on the
victim's model (95.8% and 96.7% transfer rate).


http://arxiv.org/abs/2411.10500
Edge-Only Universal Adversarial Attacks in Distributed Learning. (98%)
Giulio Rossolini; Tommaso Baldi; Alessandro Biondi; Giorgio Buttazzo
  Distributed learning frameworks, which partition neural network models across
multiple computing nodes, enhance efficiency in collaborative edge-cloud
systems but may also introduce new vulnerabilities. In this work, we explore
the feasibility of generating universal adversarial attacks when an attacker
has access to the edge part of the model only, which consists in the first
network layers. Unlike traditional universal adversarial perturbations (UAPs)
that require full model knowledge, our approach shows that adversaries can
induce effective mispredictions in the unknown cloud part by leveraging key
features on the edge side. Specifically, we train lightweight classifiers from
intermediate features available at the edge, i.e., before the split point, and
use them in a novel targeted optimization to craft effective UAPs. Our results
on ImageNet demonstrate strong attack transferability to the unknown cloud
part. Additionally, we analyze the capability of an attacker to achieve
targeted adversarial effect with edge-only knowledge, revealing intriguing
behaviors. By introducing the first adversarial attacks with edge-only
knowledge in split inference, this work underscores the importance of
addressing partial model access in adversarial robustness, encouraging further
research in this area.


http://arxiv.org/abs/2411.10498
Prompt-Guided Environmentally Consistent Adversarial Patch. (82%)
Chaoqun Li; Huanqian Yan; Lifeng Zhou; Tairan Chen; Zhuodong Liu; Hang Su
  Adversarial attacks in the physical world pose a significant threat to the
security of vision-based systems, such as facial recognition and autonomous
driving. Existing adversarial patch methods primarily focus on improving attack
performance, but they often produce patches that are easily detectable by
humans and struggle to achieve environmental consistency, i.e., blending
patches into the environment. This paper introduces a novel approach for
generating adversarial patches, which addresses both the visual naturalness and
environmental consistency of the patches. We propose Prompt-Guided
Environmentally Consistent Adversarial Patch (PG-ECAP), a method that aligns
the patch with the environment to ensure seamless integration into the
environment. The approach leverages diffusion models to generate patches that
are both environmental consistency and effective in evading detection. To
further enhance the naturalness and consistency, we introduce two alignment
losses: Prompt Alignment Loss and Latent Space Alignment Loss, ensuring that
the generated patch maintains its adversarial properties while fitting
naturally within its environment. Extensive experiments in both digital and
physical domains demonstrate that PG-ECAP outperforms existing methods in
attack success rate and environmental consistency.


http://arxiv.org/abs/2411.10367
Continual Adversarial Reinforcement Learning (CARL) of False Data Injection detection: forgetting and explainability. (81%)
Pooja Aslami; Kejun Chen; Timothy M. Hansen; Malik Hassanaly
  False data injection attacks (FDIAs) on smart inverters are a growing concern
linked to increased renewable energy production. While data-based FDIA
detection methods are also actively developed, we show that they remain
vulnerable to impactful and stealthy adversarial examples that can be crafted
using Reinforcement Learning (RL). We propose to include such adversarial
examples in data-based detection training procedure via a continual adversarial
RL (CARL) approach. This way, one can pinpoint the deficiencies of data-based
detection, thereby offering explainability during their incremental
improvement. We show that a continual learning implementation is subject to
catastrophic forgetting, and additionally show that forgetting can be addressed
by employing a joint training strategy on all generated FDIA scenarios.


http://arxiv.org/abs/2411.10034
EveGuard: Defeating Vibration-based Side-Channel Eavesdropping with Audio Adversarial Perturbations. (68%)
Jung-Woo Chang; Ke Sun; David Xia; Xinyu Zhang; Farinaz Koushanfar
  Vibrometry-based side channels pose a significant privacy risk, exploiting
sensors like mmWave radars, light sensors, and accelerometers to detect
vibrations from sound sources or proximate objects, enabling speech
eavesdropping. Despite various proposed defenses, these involve costly hardware
solutions with inherent physical limitations. This paper presents EveGuard, a
software-driven defense framework that creates adversarial audio, protecting
voice privacy from side channels without compromising human perception. We
leverage the distinct sensing capabilities of side channels and traditional
microphones where side channels capture vibrations and microphones record
changes in air pressure, resulting in different frequency responses. EveGuard
first proposes a perturbation generator model (PGM) that effectively suppresses
sensor-based eavesdropping while maintaining high audio quality. Second, to
enable end-to-end training of PGM, we introduce a new domain translation task
called Eve-GAN for inferring an eavesdropped signal from a given audio. We
further apply few-shot learning to mitigate the data collection overhead for
Eve-GAN training. Our extensive experiments show that EveGuard achieves a
protection rate of more than 97 percent from audio classifiers and
significantly hinders eavesdropped audio reconstruction. We further validate
the performance of EveGuard across three adaptive attack mechanisms. We have
conducted a user study to verify the perceptual quality of our perturbed audio.


http://arxiv.org/abs/2411.10329
Safe Text-to-Image Generation: Simply Sanitize the Prompt Embedding. (11%)
Huming Qiu; Guanxu Chen; Mi Zhang; Min Yang
  In recent years, text-to-image (T2I) generation models have made significant
progress in generating high-quality images that align with text descriptions.
However, these models also face the risk of unsafe generation, potentially
producing harmful content that violates usage policies, such as explicit
material. Existing safe generation methods typically focus on suppressing
inappropriate content by erasing undesired concepts from visual
representations, while neglecting to sanitize the textual representation.
Although these methods help mitigate the risk of misuse to certain extent,
their robustness remains insufficient when dealing with adversarial attacks.
  Given that semantic consistency between input text and output image is a
fundamental requirement for T2I models, we identify that textual
representations (i.e., prompt embeddings) are likely the primary source of
unsafe generation. To this end, we propose a vision-agnostic safe generation
framework, Embedding Sanitizer (ES), which focuses on erasing inappropriate
concepts from prompt embeddings and uses the sanitized embeddings to guide the
model for safe generation. ES is applied to the output of the text encoder as a
plug-and-play module, enabling seamless integration with different T2I models
as well as other safeguards. In addition, ES's unique scoring mechanism assigns
a score to each token in the prompt to indicate its potential harmfulness, and
dynamically adjusts the sanitization intensity to balance defensive performance
and generation quality. Through extensive evaluation on five prompt benchmarks,
our approach achieves state-of-the-art robustness by sanitizing the source
(prompt embedding) of unsafe generation compared to nine baseline methods. It
significantly outperforms existing safeguards in terms of interpretability and
controllability while maintaining generation quality.


http://arxiv.org/abs/2411.10242
Measuring Non-Adversarial Reproduction of Training Data in Large Language Models. (9%)
Michael Aerni; Javier Rando; Edoardo Debenedetti; Nicholas Carlini; Daphne Ippolito; Florian Tramèr
  Large language models memorize parts of their training data. Memorizing short
snippets and facts is required to answer questions about the world and to be
fluent in any language. But models have also been shown to reproduce long
verbatim sequences of memorized text when prompted by a motivated adversary. In
this work, we investigate an intermediate regime of memorization that we call
non-adversarial reproduction, where we quantify the overlap between model
responses and pretraining data when responding to natural and benign prompts.
For a variety of innocuous prompt categories (e.g., writing a letter or a
tutorial), we show that up to 15% of the text output by popular conversational
language models overlaps with snippets from the Internet. In worst cases, we
find generations where 100% of the content can be found exactly online. For the
same tasks, we find that human-written text has far less overlap with Internet
data. We further study whether prompting strategies can close this reproduction
gap between models and humans. While appropriate prompting can reduce
non-adversarial reproduction on average, we find that mitigating worst-case
reproduction of training data requires stronger defenses -- even for benign
interactions.


http://arxiv.org/abs/2411.10258
MDHP-Net: Detecting an Emerging Time-exciting Threat in IVN. (1%)
Qi Liu; Yanchen Liu; Ruifeng Li; Chenhong Cao; Yufeng Li; Xingyu Li; Peng Wang; Runhan Feng; Shiyang Bu
  The integration of intelligent and connected technologies in modern vehicles,
while offering enhanced functionalities through Electronic Control Unit (ECU)
and interfaces like OBD-II and telematics, also exposes the vehicle's
in-vehicle network (IVN) to potential cyberattacks. Unlike prior work, we
identify a new time-exciting threat model against IVN. These attacks inject
malicious messages that exhibit a time-exciting effect, gradually manipulating
network traffic to disrupt vehicle operations and compromise safety-critical
functions. We systematically analyze the characteristics of the threat:
dynamism, time-exciting impact, and low prior knowledge dependency. To validate
its practicality, we replicate the attack on a real Advanced Driver Assistance
System via Controller Area Network (CAN), exploiting Unified Diagnostic Service
vulnerabilities and proposing four attack strategies. While CAN's integrity
checks mitigate attacks, Ethernet migration (e.g., DoIP/SOME/IP) introduces new
surfaces. We further investigate the feasibility of time-exciting threat under
SOME/IP. To detect time-exciting threat, we introduce MDHP-Net, leveraging
Multi-Dimentional Hawkes Process (MDHP) and temporal and message-wise feature
extracting structures. Meanwhile, to estimate MDHP parameters, we developed the
first GPU-optimized gradient descent solver for MDHP (MDHP-GDS). These modules
significantly improves the detection rate under time-exciting attacks in
multi-ECU IVN system. To address data scarcity, we release STEIA9, the first
open-source dataset for time-exciting attacks, covering 9 Ethernet-based attack
scenarios. Extensive experiments on STEIA9 (9 attack scenarios) show MDHP-Net
outperforms 3 baselines, confirming attack feasibility and detection efficacy.


http://arxiv.org/abs/2411.10029
Toward Robust and Accurate Adversarial Camouflage Generation against Vehicle Detectors. (1%)
Jiawei Zhou; Linye Lyu; Daojing He; Yu Li
  Adversarial camouflage is a widely used physical attack against vehicle
detectors for its superiority in multi-view attack performance. One promising
approach involves using differentiable neural renderers to facilitate
adversarial camouflage optimization through gradient back-propagation. However,
existing methods often struggle to capture environmental characteristics during
the rendering process or produce adversarial textures that can precisely map to
the target vehicle. Moreover, these approaches neglect diverse weather
conditions, reducing the efficacy of generated camouflage across varying
weather scenarios. To tackle these challenges, we propose a robust and accurate
camouflage generation method, namely RAUCA. The core of RAUCA is a novel neural
rendering component, End-to-End Neural Renderer Plus (E2E-NRP), which can
accurately optimize and project vehicle textures and render images with
environmental characteristics such as lighting and weather. In addition, we
integrate a multi-weather dataset for camouflage generation, leveraging the
E2E-NRP to enhance the attack robustness. Experimental results on six popular
object detectors show that RAUCA-final outperforms existing methods in both
simulation and real-world settings.


http://arxiv.org/abs/2411.10507
RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively. (1%)
Yao Lu; Peixin Zhang; Jingyi Wang; Lei Ma; Xiaoniu Yang; Qi Xuan
  Deep learning has revolutionized computing in many real-world applications,
arguably due to its remarkable performance and extreme convenience as an
end-to-end solution. However, deep learning models can be costly to train and
to use, especially for those large-scale models, making it necessary to
optimize the original overly complicated models into smaller ones in scenarios
with limited resources such as mobile applications or simply for resource
saving. The key question in such model optimization is, how can we effectively
identify and measure the redundancy in a deep learning model structure. While
several common metrics exist in the popular model optimization techniques to
measure the performance of models after optimization, they are not able to
quantitatively inform the degree of remaining redundancy. To address the
problem, we present a novel testing approach, i.e., RedTest, which proposes a
novel testing metric called Model Structural Redundancy Score (MSRS) to
quantitatively measure the degree of redundancy in a deep learning model
structure. We first show that MSRS is effective in both revealing and assessing
the redundancy issues in many state-of-the-art models, which urgently calls for
model optimization. Then, we utilize MSRS to assist deep learning model
developers in two practical application scenarios: 1) in Neural Architecture
Search, we design a novel redundancy-aware algorithm to guide the search for
the optimal model structure and demonstrate its effectiveness by comparing it
to existing standard NAS practice; 2) in the pruning of large-scale pre-trained
models, we prune the redundant layers of pre-trained models with the guidance
of layer similarity to derive less redundant ones of much smaller size.
Extensive experimental results demonstrate that removing such redundancy has a
negligible effect on the model utility.


http://arxiv.org/abs/2411.09265
BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation. (99%)
Zheng Zhou; Wenquan Feng; Shuchang Lyu; Guangliang Cheng; Xiaowei Huang; Qi Zhao
  Dataset Distillation (DD) is an emerging technique that compresses
large-scale datasets into significantly smaller synthesized datasets while
preserving high test performance and enabling the efficient training of large
models. However, current research primarily focuses on enhancing evaluation
accuracy under limited compression ratios, often overlooking critical security
concerns such as adversarial robustness. A key challenge in evaluating this
robustness lies in the complex interactions between distillation methods, model
architectures, and adversarial attack strategies, which complicate standardized
assessments. To address this, we introduce BEARD, an open and unified benchmark
designed to systematically assess the adversarial robustness of DD methods,
including DM, IDM, and BACON. BEARD encompasses a variety of adversarial
attacks (e.g., FGSM, PGD, C&W) on distilled datasets like CIFAR-10/100 and
TinyImageNet. Utilizing an adversarial game framework, it introduces three key
metrics: Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive
Robustness-Efficiency Index (CREI). Our analysis includes unified benchmarks,
various Images Per Class (IPC) settings, and the effects of adversarial
training. Results are available on the BEARD Leaderboard, along with a library
providing model and dataset pools to support reproducible research. Access the
code at BEARD.


http://arxiv.org/abs/2411.09220
Transferable Adversarial Attacks against ASR. (89%)
Xiaoxue Gao; Zexin Li; Yiming Chen; Cong Liu; Haizhou Li
  Given the extensive research and real-world applications of automatic speech
recognition (ASR), ensuring the robustness of ASR models against minor input
perturbations becomes a crucial consideration for maintaining their
effectiveness in real-time scenarios. Previous explorations into ASR model
robustness have predominantly revolved around evaluating accuracy on white-box
settings with full access to ASR models. Nevertheless, full ASR model details
are often not available in real-world applications. Therefore, evaluating the
robustness of black-box ASR models is essential for a comprehensive
understanding of ASR model resilience. In this regard, we thoroughly study the
vulnerability of practical black-box attacks in cutting-edge ASR models and
propose to employ two advanced time-domain-based transferable attacks alongside
our differentiable feature extractor. We also propose a speech-aware gradient
optimization approach (SAGO) for ASR, which forces mistranscription with
minimal impact on human imperceptibility through voice activity detection rule
and a speech-aware gradient-oriented optimizer. Our comprehensive experimental
results reveal performance enhancements compared to baseline approaches across
five models on two databases.


http://arxiv.org/abs/2411.09749
RenderBender: A Survey on Adversarial Attacks Using Differentiable Rendering. (83%)
Matthew Hull; Haoran Wang; Matthew Lau; Alec Helbling; Mansi Phute; Chao Zhang; Zsolt Kira; Willian Lunardi; Martin Andreoni; Wenke Lee; Polo Chau
  Differentiable rendering techniques like Gaussian Splatting and Neural
Radiance Fields have become powerful tools for generating high-fidelity models
of 3D objects and scenes. Their ability to produce both physically plausible
and differentiable models of scenes are key ingredient needed to produce
physically plausible adversarial attacks on DNNs. However, the adversarial
machine learning community has yet to fully explore these capabilities, partly
due to differing attack goals (e.g., misclassification, misdetection) and a
wide range of possible scene manipulations used to achieve them (e.g., alter
texture, mesh). This survey contributes the first framework that unifies
diverse goals and tasks, facilitating easy comparison of existing work,
identifying research gaps, and highlighting future directions - ranging from
expanding attack goals and tasks to account for new modalities,
state-of-the-art models, tools, and pipelines, to underscoring the importance
of studying real-world threats in complex scenes.


http://arxiv.org/abs/2411.09259
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey. (69%)
Xuannan Liu; Xing Cui; Peipei Li; Zekun Li; Huaibo Huang; Shuhan Xia; Miaoxuan Zhang; Yueying Zou; Ran He
  The rapid evolution of multimodal foundation models has led to significant
advancements in cross-modal understanding and generation across diverse
modalities, including text, images, audio, and video. However, these models
remain susceptible to jailbreak attacks, which can bypass built-in safety
mechanisms and induce the production of potentially harmful content.
Consequently, understanding the methods of jailbreak attacks and existing
defense mechanisms is essential to ensure the safe deployment of multimodal
generative models in real-world scenarios, particularly in security-sensitive
applications. To provide comprehensive insight into this topic, this survey
reviews jailbreak and defense in multimodal generative models. First, given the
generalized lifecycle of multimodal jailbreak, we systematically explore
attacks and corresponding defense strategies across four levels: input,
encoder, generator, and output. Based on this analysis, we present a detailed
taxonomy of attack methods, defense mechanisms, and evaluation frameworks
specific to multimodal generative models. Additionally, we cover a wide range
of input-output configurations, including modalities such as Any-to-Text,
Any-to-Vision, and Any-to-Any within generative systems. Finally, we highlight
current research challenges and propose potential directions for future
research.The open-source repository corresponding to this work can be found at
https://github.com/liuxuannan/Awesome-Multimodal-Jailbreak.


http://arxiv.org/abs/2411.09359
Your Fixed Watermark is Fragile: Towards Semantic-Aware Watermark for EaaS Copyright Protection. (11%)
Zekun Fei; Biao Yi; Jianing Geng; Ruiqi He; Lihai Nie; Zheli Liu
  Embedding-as-a-Service (EaaS) has emerged as a successful business pattern
but faces significant challenges related to various forms of copyright
infringement, including API misuse and different attacks. Various studies have
proposed backdoor-based watermarking schemes to protect the copyright of EaaS
services. In this paper, we reveal that previous watermarking schemes possess
semantic-independent characteristics and propose the Semantic Perturbation
Attack (SPA). Our theoretical and experimental analyses demonstrate that this
semantic-independent nature makes current watermarking schemes vulnerable to
adaptive attacks that exploit semantic perturbations test to bypass watermark
verification. To address this vulnerability, we propose the Semantic Aware
Watermarking (SAW) scheme, a robust defense mechanism designed to resist SPA,
by injecting a watermark that adapts to the text semantics. Extensive
experimental results across multiple datasets demonstrate that the True
Positive Rate (TPR) for detecting watermarked samples under SPA can reach up to
more than 95%, rendering previous watermarks ineffective. Meanwhile, our
watermarking scheme can resist such attack while ensuring the watermark
verification capability. Our code is available at
https://github.com/Zk4-ps/EaaS-Embedding-Watermark.


http://arxiv.org/abs/2411.09373
Are nuclear masks all you need for improved out-of-domain generalisation? A closer look at cancer classification in histopathology. (1%)
Dhananjay Tomar; Alexander Binder; Andreas Kleppe
  Domain generalisation in computational histopathology is challenging because
the images are substantially affected by differences among hospitals due to
factors like fixation and staining of tissue and imaging equipment. We
hypothesise that focusing on nuclei can improve the out-of-domain (OOD)
generalisation in cancer detection. We propose a simple approach to improve OOD
generalisation for cancer detection by focusing on nuclear morphology and
organisation, as these are domain-invariant features critical in cancer
detection. Our approach integrates original images with nuclear segmentation
masks during training, encouraging the model to prioritise nuclei and their
spatial arrangement. Going beyond mere data augmentation, we introduce a
regularisation technique that aligns the representations of masks and original
images. We show, using multiple datasets, that our method improves OOD
generalisation and also leads to increased robustness to image corruptions and
adversarial attacks. The source code is available at
https://github.com/undercutspiky/SFL/


http://arxiv.org/abs/2411.08933
Confidence-aware Denoised Fine-tuning of Off-the-shelf Models for Certified Robustness. (95%)
Suhyeok Jang; Seojin Kim; Jinwoo Shin; Jongheon Jeong
  The remarkable advances in deep learning have led to the emergence of many
off-the-shelf classifiers, e.g., large pre-trained models. However, since they
are typically trained on clean data, they remain vulnerable to adversarial
attacks. Despite this vulnerability, their superior performance and
transferability make off-the-shelf classifiers still valuable in practice,
demanding further work to provide adversarial robustness for them in a post-hoc
manner. A recently proposed method, denoised smoothing, leverages a denoiser
model in front of the classifier to obtain provable robustness without
additional training. However, the denoiser often creates hallucination, i.e.,
images that have lost the semantics of their originally assigned class, leading
to a drop in robustness. Furthermore, its noise-and-denoise procedure
introduces a significant distribution shift from the original distribution,
causing the denoised smoothing framework to achieve sub-optimal robustness. In
this paper, we introduce Fine-Tuning with Confidence-Aware Denoised Image
Selection (FT-CADIS), a novel fine-tuning scheme to enhance the certified
robustness of off-the-shelf classifiers. FT-CADIS is inspired by the
observation that the confidence of off-the-shelf classifiers can effectively
identify hallucinated images during denoised smoothing. Based on this, we
develop a confidence-aware training objective to handle such hallucinated
images and improve the stability of fine-tuning from denoised images. In this
way, the classifier can be fine-tuned using only images that are beneficial for
adversarial robustness. We also find that such a fine-tuning can be done by
updating a small fraction of parameters of the classifier. Extensive
experiments demonstrate that FT-CADIS has established the state-of-the-art
certified robustness among denoised smoothing methods across all
$\ell_2$-adversary radius in various benchmarks.


http://arxiv.org/abs/2411.08460
Trap-MID: Trapdoor-based Defense against Model Inversion Attacks. (81%)
Zhen-Ting Liu; Shang-Tse Chen
  Model Inversion (MI) attacks pose a significant threat to the privacy of Deep
Neural Networks by recovering training data distribution from well-trained
models. While existing defenses often rely on regularization techniques to
reduce information leakage, they remain vulnerable to recent attacks. In this
paper, we propose the Trapdoor-based Model Inversion Defense (Trap-MID) to
mislead MI attacks. A trapdoor is integrated into the model to predict a
specific label when the input is injected with the corresponding trigger.
Consequently, this trapdoor information serves as the "shortcut" for MI
attacks, leading them to extract trapdoor triggers rather than private data. We
provide theoretical insights into the impacts of trapdoor's effectiveness and
naturalness on deceiving MI attacks. In addition, empirical experiments
demonstrate the state-of-the-art defense performance of Trap-MID against
various MI attacks without the requirements for extra data or large
computational overhead. Our source code is publicly available at
https://github.com/ntuaislab/Trap-MID.


http://arxiv.org/abs/2411.08618
Robust Optimal Power Flow Against Adversarial Attacks: A Tri-Level Optimization Approach. (81%)
Saman Mazaheri Khamaneh; Tong Wu
  In power systems, unpredictable events like extreme weather, equipment
failures, and cyberattacks present significant challenges to ensuring safety
and reliability. Ensuring resilience in the face of these uncertainties is
crucial for reliable and efficient operations. This paper presents a tri-level
optimization approach for robust power system operations that effectively
address worst-case attacks. The first stage focuses on optimizing economic
dispatch under normal operating conditions, aiming to minimize generation costs
while maintaining the supply-demand balance. The second stage introduces an
adversarial attack model, identifying worst-case scenarios that maximize the
system's vulnerability by targeting distributed generation (DG). In the third
stage, mitigation strategies are developed using fast-response energy storage
systems (ESS) to minimize disruptions caused by these attacks. By integrating
economic dispatch, vulnerability assessment, and mitigation into a unified
framework, this approach provides a robust solution for enhancing power system
resilience and safety against evolving adversarial threats. The approach is
validated using the IEEE-33 node distribution system to demonstrate its
effectiveness in achieving both cost efficiency and system resilience.


http://arxiv.org/abs/2411.08410
The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defense. (74%)
Yangyang Guo; Fangkai Jiao; Liqiang Nie; Mohan Kankanhalli
  The vulnerability of Vision Large Language Models (VLLMs) to jailbreak
attacks appears as no surprise. However, recent defense mechanisms against
these attacks have reached near-saturation performance on benchmarks, often
with minimal effort. This simultaneous high performance in both attack and
defense presents a perplexing paradox. Resolving it is critical for advancing
the development of trustworthy models. To address this research gap, we first
investigate why VLLMs are prone to these attacks. We then make a key
observation: existing defense mechanisms suffer from an \textbf{over-prudence}
problem, resulting in unexpected abstention even in the presence of benign
inputs. Additionally, we find that the two representative evaluation methods
for jailbreak often exhibit chance agreement. This limitation makes it
potentially misleading when evaluating attack strategies or defense mechanisms.
Beyond these empirical observations, our another contribution in this work is
to repurpose the guardrails of LLMs on the shelf, as an effective alternative
detector prior to VLLM response. We believe these findings offer useful
insights to rethink the foundational development of VLLM safety with respect to
benchmark datasets, evaluation methods, and defense strategies.


http://arxiv.org/abs/2411.08862
LLMStinger: Jailbreaking LLMs using RL fine-tuned LLMs. (8%)
Piyush Jha; Arnav Arora; Vijay Ganesh
  We introduce LLMStinger, a novel approach that leverages Large Language
Models (LLMs) to automatically generate adversarial suffixes for jailbreak
attacks. Unlike traditional methods, which require complex prompt engineering
or white-box access, LLMStinger uses a reinforcement learning (RL) loop to
fine-tune an attacker LLM, generating new suffixes based on existing attacks
for harmful questions from the HarmBench benchmark. Our method significantly
outperforms existing red-teaming approaches (we compared against 15 of the
latest methods), achieving a +57.2% improvement in Attack Success Rate (ASR) on
LLaMA2-7B-chat and a +50.3% ASR increase on Claude 2, both models known for
their extensive safety measures. Additionally, we achieved a 94.97% ASR on
GPT-3.5 and 99.4% on Gemma-2B-it, demonstrating the robustness and adaptability
of LLMStinger across open and closed-source models.


http://arxiv.org/abs/2411.07843
Chain Association-based Attacking and Shielding Natural Language Processing Systems. (99%)
Jiacheng Huang; Long Chen
  Association as a gift enables people do not have to mention something in
completely straightforward words and allows others to understand what they
intend to refer to. In this paper, we propose a chain association-based
adversarial attack against natural language processing systems, utilizing the
comprehension gap between humans and machines. We first generate a chain
association graph for Chinese characters based on the association paradigm for
building search space of potential adversarial examples. Then, we introduce an
discrete particle swarm optimization algorithm to search for the optimal
adversarial examples. We conduct comprehensive experiments and show that
advanced natural language processing models and applications, including large
language models, are vulnerable to our attack, while humans appear good at
understanding the perturbed text. We also explore two methods, including
adversarial training and associative graph-based recovery, to shield systems
from chain association-based attack. Since a few examples that use some
derogatory terms, this paper contains materials that may be offensive or
upsetting to some people.


http://arxiv.org/abs/2411.07850
IAE: Irony-based Adversarial Examples for Sentiment Analysis Systems. (99%)
Xiaoyin Yi; Jiacheng Huang
  Adversarial examples, which are inputs deliberately perturbed with
imperceptible changes to induce model errors, have raised serious concerns for
the reliability and security of deep neural networks (DNNs). While adversarial
attacks have been extensively studied in continuous data domains such as
images, the discrete nature of text presents unique challenges. In this paper,
we propose Irony-based Adversarial Examples (IAE), a method that transforms
straightforward sentences into ironic ones to create adversarial text. This
approach exploits the rhetorical device of irony, where the intended meaning is
opposite to the literal interpretation, requiring a deeper understanding of
context to detect. The IAE method is particularly challenging due to the need
to accurately locate evaluation words, substitute them with appropriate
collocations, and expand the text with suitable ironic elements while
maintaining semantic coherence. Our research makes the following key
contributions: (1) We introduce IAE, a strategy for generating textual
adversarial examples using irony. This method does not rely on pre-existing
irony corpora, making it a versatile tool for creating adversarial text in
various NLP tasks. (2) We demonstrate that the performance of several
state-of-the-art deep learning models on sentiment analysis tasks significantly
deteriorates when subjected to IAE attacks. This finding underscores the
susceptibility of current NLP systems to adversarial manipulation through
irony. (3) We compare the impact of IAE on human judgment versus NLP systems,
revealing that humans are less susceptible to the effects of irony in text.


http://arxiv.org/abs/2411.07559
Zer0-Jack: A Memory-efficient Gradient-based Jailbreaking Method for Black-box Multi-modal Large Language Models. (78%)
Tiejin Chen; Kaishen Wang; Hua Wei
  Jailbreaking methods, which induce Multi-modal Large Language Models (MLLMs)
to output harmful responses, raise significant safety concerns. Among these
methods, gradient-based approaches, which use gradients to generate malicious
prompts, have been widely studied due to their high success rates in white-box
settings, where full access to the model is available. However, these methods
have notable limitations: they require white-box access, which is not always
feasible, and involve high memory usage. To address scenarios where white-box
access is unavailable, attackers often resort to transfer attacks. In transfer
attacks, malicious inputs generated using white-box models are applied to
black-box models, but this typically results in reduced attack performance. To
overcome these challenges, we propose Zer0-Jack, a method that bypasses the
need for white-box access by leveraging zeroth-order optimization. We propose
patch coordinate descent to efficiently generate malicious image inputs to
directly attack black-box MLLMs, which significantly reduces memory usage
further. Through extensive experiments, Zer0-Jack achieves a high attack
success rate across various models, surpassing previous transfer-based methods
and performing comparably with existing white-box jailbreak techniques.
Notably, Zer0-Jack achieves a 95\% attack success rate on MiniGPT-4 with the
Harmful Behaviors Multi-modal Dataset on a black-box setting, demonstrating its
effectiveness. Additionally, we show that Zer0-Jack can directly attack
commercial MLLMs such as GPT-4o. Codes are provided in the supplement.


http://arxiv.org/abs/2411.08248
Deceiving Question-Answering Models: A Hybrid Word-Level Adversarial Approach. (67%)
Jiyao Li; Mingze Ni; Yongshun Gong; Wei Liu
  Deep learning underpins most of the currently advanced natural language
processing (NLP) tasks such as textual classification, neural machine
translation (NMT), abstractive summarization and question-answering (QA).
However, the robustness of the models, particularly QA models, against
adversarial attacks is a critical concern that remains insufficiently explored.
This paper introduces QA-Attack (Question Answering Attack), a novel word-level
adversarial strategy that fools QA models. Our attention-based attack exploits
the customized attention mechanism and deletion ranking strategy to identify
and target specific words within contextual passages. It creates deceptive
inputs by carefully choosing and substituting synonyms, preserving grammatical
integrity while misleading the model to produce incorrect responses. Our
approach demonstrates versatility across various question types, particularly
when dealing with extensive long textual inputs. Extensive experiments on
multiple benchmark datasets demonstrate that QA-Attack successfully deceives
baseline QA models and surpasses existing adversarial techniques regarding
success rate, semantics changes, BLEU score, fluency and grammar error rate.


http://arxiv.org/abs/2411.07597
A Survey on Adversarial Machine Learning for Code Data: Realistic Threats, Countermeasures, and Interpretations. (64%)
Yulong Yang; Haoran Fan; Chenhao Lin; Qian Li; Zhengyu Zhao; Chao Shen; Xiaohong Guan
  Code Language Models (CLMs) have achieved tremendous progress in source code
understanding and generation, leading to a significant increase in research
interests focused on applying CLMs to real-world software engineering tasks in
recent years. However, in realistic scenarios, CLMs are exposed to potential
malicious adversaries, bringing risks to the confidentiality, integrity, and
availability of CLM systems. Despite these risks, a comprehensive analysis of
the security vulnerabilities of CLMs in the extremely adversarial environment
has been lacking. To close this research gap, we categorize existing attack
techniques into three types based on the CIA triad: poisoning attacks
(integrity \& availability infringement), evasion attacks (integrity
infringement), and privacy attacks (confidentiality infringement). We have
collected so far the most comprehensive (79) papers related to adversarial
machine learning for CLM from the research fields of artificial intelligence,
computer security, and software engineering. Our analysis covers each type of
risk, examining threat model categorization, attack techniques, and
countermeasures, while also introducing novel perspectives on eXplainable AI
(XAI) and exploring the interconnections between different risks. Finally, we
identify current challenges and future research opportunities. This study aims
to provide a comprehensive roadmap for both researchers and practitioners and
pave the way towards more reliable CLMs for practical applications.


http://arxiv.org/abs/2411.07691
New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook. (13%)
Meng Yang; Tianqing Zhu; Chi Liu; WanLei Zhou; Shui Yu; Philip S. Yu
  Thanks to the explosive growth of data and the development of computational
resources, it is possible to build pre-trained models that can achieve
outstanding performance on various tasks, such as neural language processing,
computer vision, and more. Despite their powerful capabilities, pre-trained
models have also sparked attention to the emerging security challenges
associated with their real-world applications. Security and privacy issues,
such as leaking privacy information and generating harmful responses, have
seriously undermined users' confidence in these powerful models. Concerns are
growing as model performance improves dramatically. Researchers are eager to
explore the unique security and privacy issues that have emerged, their
distinguishing factors, and how to defend against them. However, the current
literature lacks a clear taxonomy of emerging attacks and defenses for
pre-trained models, which hinders a high-level and comprehensive understanding
of these questions. To fill the gap, we conduct a systematical survey on the
security risks of pre-trained models, proposing a taxonomy of attack and
defense methods based on the accessibility of pre-trained models' input and
weights in various security test scenarios. This taxonomy categorizes attacks
and defenses into No-Change, Input-Change, and Model-Change approaches. With
the taxonomy analysis, we capture the unique security and privacy issues of
pre-trained models, categorizing and summarizing existing security issues based
on their characteristics. In addition, we offer a timely and comprehensive
review of each category's strengths and limitations. Our survey concludes by
highlighting potential new research opportunities in the security and privacy
of pre-trained models.


http://arxiv.org/abs/2411.08923
Aligning Visual Contrastive learning models via Preference Optimization. (1%)
Amirabbas Afzali; Borna Khodabandeh; Ali Rasekh; Mahyar JafariNodeh; Sepehr kazemi; Simon Gottschalk
  Contrastive learning models have demonstrated impressive abilities to capture
semantic similarities by aligning representations in the embedding space.
However, their performance can be limited by the quality of the training data
and its inherent biases. While Preference Optimization (PO) methods such as
Reinforcement Learning from Human Feedback (RLHF) and Direct Preference
Optimization (DPO) have been applied to align generative models with human
preferences, their use in contrastive learning has yet to be explored. This
paper introduces a novel method for training contrastive learning models using
different PO methods to break down complex concepts. Our method systematically
aligns model behavior with desired preferences, enhancing performance on the
targeted task. In particular, we focus on enhancing model robustness against
typographic attacks and inductive biases, commonly seen in contrastive
vision-language models like CLIP. Our experiments demonstrate that models
trained using PO outperform standard contrastive learning techniques while
retaining their ability to handle adversarial challenges and maintain accuracy
on other downstream tasks. This makes our method well-suited for tasks
requiring fairness, robustness, and alignment with specific preferences. We
evaluate our method for tackling typographic attacks on images and explore its
ability to disentangle gender concepts and mitigate gender bias, showcasing the
versatility of our approach.


http://arxiv.org/abs/2411.08148
Adaptive Meta-Learning for Robust Deepfake Detection: A Multi-Agent Framework to Data Drift and Model Generalization. (1%)
Dinesh Srivasthav P; Badri Narayan Subudhi
  Pioneering advancements in artificial intelligence, especially in genAI, have
enabled significant possibilities for content creation, but also led to
widespread misinformation and false content. The growing sophistication and
realism of deepfakes is raising concerns about privacy invasion, identity
theft, and has societal, business impacts, including reputational damage and
financial loss. Many deepfake detectors have been developed to tackle this
problem. Nevertheless, as for every AI model, the deepfake detectors face the
wrath of lack of considerable generalization to unseen scenarios and
cross-domain deepfakes. Besides, adversarial robustness is another critical
challenge, as detectors drastically underperform to the slightest imperceptible
change. Most state-of-the-art detectors are trained on static datasets and lack
the ability to adapt to emerging deepfake attack trends. These three crucial
challenges though hold paramount importance for reliability in practise,
particularly in the deepfake domain, are also the problems with any other AI
application. This paper proposes an adversarial meta-learning algorithm using
task-specific adaptive sample synthesis and consistency regularization, in a
refinement phase. By focussing on the classifier's strengths and weaknesses, it
boosts both robustness and generalization of the model. Additionally, the paper
introduces a hierarchical multi-agent retrieval-augmented generation workflow
with a sample synthesis module to dynamically adapt the model to new data
trends by generating custom deepfake samples. The paper further presents a
framework integrating the meta-learning algorithm with the hierarchical
multi-agent workflow, offering a holistic solution for enhancing
generalization, robustness, and adaptability. Experimental results demonstrate
the model's consistent performance across various datasets, outperforming the
models in comparison.


http://arxiv.org/abs/2411.06784
Boosting the Targeted Transferability of Adversarial Examples via Salient Region & Weighted Feature Drop. (99%)
Shanjun Xu; Linghui Li; Kaiguo Yuan; Bingyu Li
  Deep neural networks can be vulnerable to adversarially crafted examples,
presenting significant risks to practical applications. A prevalent approach
for adversarial attacks relies on the transferability of adversarial examples,
which are generated from a substitute model and leveraged to attack unknown
black-box models. Despite various proposals aimed at improving transferability,
the success of these attacks in targeted black-box scenarios is often hindered
by the tendency for adversarial examples to overfit to the surrogate models. In
this paper, we introduce a novel framework based on Salient region & Weighted
Feature Drop (SWFD) designed to enhance the targeted transferability of
adversarial examples. Drawing from the observation that examples with higher
transferability exhibit smoother distributions in the deep-layer outputs, we
propose the weighted feature drop mechanism to modulate activation values
according to weights scaled by norm distribution, effectively addressing the
overfitting issue when generating adversarial examples. Additionally, by
leveraging salient region within the image to construct auxiliary images, our
method enables the adversarial example's features to be transferred to the
target category in a model-agnostic manner, thereby enhancing the
transferability. Comprehensive experiments confirm that our approach
outperforms state-of-the-art methods across diverse configurations. On average,
the proposed SWFD raises the attack success rate for normally trained models
and robust models by 16.31% and 7.06% respectively.


http://arxiv.org/abs/2411.06863
Computable Model-Independent Bounds for Adversarial Quantum Machine Learning. (69%)
Bacui Li; Tansu Alpcan; Chandra Thapa; Udaya Parampalli
  By leveraging the principles of quantum mechanics, QML opens doors to novel
approaches in machine learning and offers potential speedup. However, machine
learning models are well-documented to be vulnerable to malicious
manipulations, and this susceptibility extends to the models of QML. This
situation necessitates a thorough understanding of QML's resilience against
adversarial attacks, particularly in an era where quantum computing
capabilities are expanding. In this regard, this paper examines
model-independent bounds on adversarial performance for QML. To the best of our
knowledge, we introduce the first computation of an approximate lower bound for
adversarial error when evaluating model resilience against sophisticated
quantum-based adversarial attacks. Experimental results are compared to the
computed bound, demonstrating the potential of QML models to achieve high
robustness. In the best case, the experimental error is only 10% above the
estimated bound, offering evidence of the inherent robustness of quantum
models. This work not only advances our theoretical understanding of quantum
model resilience but also provides a precise reference bound for the future
development of robust QML algorithms.


http://arxiv.org/abs/2411.07023
The Inherent Adversarial Robustness of Analog In-Memory Computing. (61%)
Corey Lammie; Julian Büchel; Athanasios Vasilopoulos; Manuel Le Gallo; Abu Sebastian
  A key challenge for Deep Neural Network (DNN) algorithms is their
vulnerability to adversarial attacks. Inherently non-deterministic compute
substrates, such as those based on Analog In-Memory Computing (AIMC), have been
speculated to provide significant adversarial robustness when performing DNN
inference. In this paper, we experimentally validate this conjecture for the
first time on an AIMC chip based on Phase Change Memory (PCM) devices. We
demonstrate higher adversarial robustness against different types of
adversarial attacks when implementing an image classification network.
Additional robustness is also observed when performing hardware-in-the-loop
attacks, for which the attacker is assumed to have full access to the hardware.
A careful study of the various noise sources indicate that a combination of
stochastic noise sources (both recurrent and non-recurrent) are responsible for
the adversarial robustness and that their type and magnitude disproportionately
effects this property. Finally, it is demonstrated, via simulations, that when
a much larger transformer network is used to implement a Natural Language
Processing (NLP) task, additional robustness is still observed.


http://arxiv.org/abs/2411.07494
Rapid Response: Mitigating LLM Jailbreaks with a Few Examples. (54%)
Alwin Peng; Julian Michael; Henry Sleight; Ethan Perez; Mrinank Sharma
  As large language models (LLMs) grow more powerful, ensuring their safety
against misuse becomes crucial. While researchers have focused on developing
robust defenses, no method has yet achieved complete invulnerability to
attacks. We propose an alternative approach: instead of seeking perfect
adversarial robustness, we develop rapid response techniques to look to block
whole classes of jailbreaks after observing only a handful of attacks. To study
this setting, we develop RapidResponseBench, a benchmark that measures a
defense's robustness against various jailbreak strategies after adapting to a
few observed examples. We evaluate five rapid response methods, all of which
use jailbreak proliferation, where we automatically generate additional
jailbreaks similar to the examples observed. Our strongest method, which
fine-tunes an input classifier to block proliferated jailbreaks, reduces attack
success rate by a factor greater than 240 on an in-distribution set of
jailbreaks and a factor greater than 15 on an out-of-distribution set, having
observed just one example of each jailbreaking strategy. Moreover, further
studies suggest that the quality of proliferation model and number of
proliferated examples play an key role in the effectiveness of this defense.
Overall, our results highlight the potential of responding rapidly to novel
jailbreaks to limit LLM misuse.


http://arxiv.org/abs/2411.07472
Semi-Truths: A Large-Scale Dataset of AI-Augmented Images for Evaluating Robustness of AI-Generated Image detectors. (1%)
Anisha Pal; Julia Kruk; Mansi Phute; Manognya Bhattaram; Diyi Yang; Duen Horng Chau; Judy Hoffman
  Text-to-image diffusion models have impactful applications in art, design,
and entertainment, yet these technologies also pose significant risks by
enabling the creation and dissemination of misinformation. Although recent
advancements have produced AI-generated image detectors that claim robustness
against various augmentations, their true effectiveness remains uncertain. Do
these detectors reliably identify images with different levels of augmentation?
Are they biased toward specific scenes or data distributions? To investigate,
we introduce SEMI-TRUTHS, featuring 27,600 real images, 223,400 masks, and
1,472,700 AI-augmented images that feature targeted and localized perturbations
produced using diverse augmentation techniques, diffusion models, and data
distributions. Each augmented image is accompanied by metadata for standardized
and targeted evaluation of detector robustness. Our findings suggest that
state-of-the-art detectors exhibit varying sensitivities to the types and
degrees of perturbations, data distributions, and augmentation methods used,
offering new insights into their performance and limitations. The code for the
augmentation and evaluation pipeline is available at
https://github.com/J-Kruk/SemiTruths.


http://arxiv.org/abs/2411.06666
Adversarial Detection with a Dynamically Stable System. (99%)
Xiaowei Long; Jie Lin; Xiangyuan Yang
  Adversarial detection is designed to identify and reject maliciously crafted
adversarial examples(AEs) which are generated to disrupt the classification of
target models.
  Presently, various input transformation-based methods have been developed on
adversarial example detection, which typically rely on empirical experience and
lead to unreliability against new attacks.
  To address this issue, we propose and conduct a Dynamically Stable System
(DSS), which can effectively detect the adversarial examples from normal
examples according to the stability of input examples.
  Particularly, in our paper, the generation of adversarial examples is
considered as the perturbation process of a Lyapunov dynamic system, and we
propose an example stability mechanism, in which a novel control term is added
in adversarial example generation to ensure that the normal examples can
achieve dynamic stability while the adversarial examples cannot achieve the
stability.
  Then, based on the proposed example stability mechanism, a Dynamically Stable
System (DSS) is proposed, which can utilize the disruption and restoration
actions to determine the stability of input examples and detect the adversarial
examples through changes in the stability of the input examples.
  In comparison with existing methods in three benchmark datasets(MNIST,
CIFAR10, and CIFAR100), our evaluation results show that our proposed DSS can
achieve ROC-AUC values of 99.83%, 97.81% and 94.47%, surpassing the
state-of-the-art(SOTA) values of 97.35%, 91.10% and 93.49% in the other 7
methods.


http://arxiv.org/abs/2411.14449
Deferred Backdoor Functionality Attacks on Deep Learning Models. (82%)
Jeongjin Shin; Sangdon Park
  Deep learning models are vulnerable to backdoor attacks, where adversaries
inject malicious functionality during training that activates on trigger inputs
at inference time. Extensive research has focused on developing stealthy
backdoor attacks to evade detection and defense mechanisms. However, these
approaches still have limitations that leave the door open for detection and
mitigation due to their inherent design to cause malicious behavior in the
presence of a trigger. To address this limitation, we introduce Deferred
Backdoor Functionality Activation (DBFA), a new paradigm in backdoor attacks.
Unlike conventional attacks, DBFA initially conceals its backdoor, producing
benign outputs even when triggered. This stealthy behavior allows DBFA to
bypass multiple detection and defense methods, remaining undetected during
initial inspections. The backdoor functionality is strategically activated only
after the model undergoes subsequent updates, such as retraining on benign
data. DBFA attacks exploit the common practice in the life cycle of machine
learning models to perform model updates and fine-tuning after initial
deployment. To implement DBFA attacks, we approach the problem by making the
unlearning of the backdoor fragile, allowing it to be easily cancelled and
subsequently reactivate the backdoor functionality. To achieve this, we propose
a novel two-stage training scheme, called DeferBad. Our extensive experiments
across various fine-tuning scenarios, backdoor attack types, datasets, and
model architectures demonstrate the effectiveness and stealthiness of DeferBad.


http://arxiv.org/abs/2411.06426
SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains. (70%)
Bijoy Ahmed Saiem; MD Sadik Hossain Shanto; Rakib Ahsan; Md Rafi ur Rashid
  As the integration of the Large Language Models (LLMs) into various
applications increases, so does their susceptibility to misuse, raising
significant security concerns. Numerous jailbreak attacks have been proposed to
assess the security defense of LLMs. Current jailbreak attacks mainly rely on
scenario camouflage, prompt obfuscation, prompt optimization, and prompt
iterative optimization to conceal malicious prompts. In particular, sequential
prompt chains in a single query can lead LLMs to focus on certain prompts while
ignoring others, facilitating context manipulation. This paper introduces
SequentialBreak, a novel jailbreak attack that exploits this vulnerability. We
discuss several scenarios, not limited to examples like Question Bank, Dialog
Completion, and Game Environment, where the harmful prompt is embedded within
benign ones that can fool LLMs into generating harmful responses. The distinct
narrative structures of these scenarios show that SequentialBreak is flexible
enough to adapt to various prompt formats beyond those discussed. Extensive
experiments demonstrate that SequentialBreak uses only a single query to
achieve a substantial gain of attack success rate over existing baselines
against both open-source and closed-source models. Through our research, we
highlight the urgent need for more robust and resilient safeguards to enhance
LLM security and prevent potential misuse. All the result files and website
associated with this research are available in this GitHub repository:
https://anonymous.4open.science/r/JailBreakAttack-4F3B/.


http://arxiv.org/abs/2411.07268
Target-driven Attack for Large Language Models. (73%)
Chong Zhang; Mingyu Jin; Dong Shu; Taowen Wang; Dongfang Liu; Xiaobo Jin
  Current large language models (LLM) provide a strong foundation for
large-scale user-oriented natural language tasks. Many users can easily inject
adversarial text or instructions through the user interface, thus causing LLM
model security challenges like the language model not giving the correct
answer. Although there is currently a large amount of research on black-box
attacks, most of these black-box attacks use random and heuristic strategies.
It is unclear how these strategies relate to the success rate of attacks and
thus effectively improve model robustness. To solve this problem, we propose
our target-driven black-box attack method to maximize the KL divergence between
the conditional probabilities of the clean text and the attack text to redefine
the attack's goal. We transform the distance maximization problem into two
convex optimization problems based on the attack goal to solve the attack text
and estimate the covariance. Furthermore, the projected gradient descent
algorithm solves the vector corresponding to the attack text. Our target-driven
black-box attack approach includes two attack strategies: token manipulation
and misinformation attack. Experimental results on multiple Large Language
Models and datasets demonstrate the effectiveness of our attack method.


http://arxiv.org/abs/2411.06146
AI-Compass: A Comprehensive and Effective Multi-module Testing Tool for AI Systems. (33%)
Zhiyu Zhu; Zhibo Jin; Hongsheng Hu; Minhui Xue; Ruoxi Sun; Seyit Camtepe; Praveen Gauravaram; Huaming Chen
  AI systems, in particular with deep learning techniques, have demonstrated
superior performance for various real-world applications. Given the need for
tailored optimization in specific scenarios, as well as the concerns related to
the exploits of subsurface vulnerabilities, a more comprehensive and in-depth
testing AI system becomes a pivotal topic. We have seen the emergence of
testing tools in real-world applications that aim to expand testing
capabilities. However, they often concentrate on ad-hoc tasks, rendering them
unsuitable for simultaneously testing multiple aspects or components.
Furthermore, trustworthiness issues arising from adversarial attacks and the
challenge of interpreting deep learning models pose new challenges for
developing more comprehensive and in-depth AI system testing tools. In this
study, we design and implement a testing tool, \tool, to comprehensively and
effectively evaluate AI systems. The tool extensively assesses multiple
measurements towards adversarial robustness, model interpretability, and
performs neuron analysis. The feasibility of the proposed testing tool is
thoroughly validated across various modalities, including image classification,
object detection, and text classification. Extensive experiments demonstrate
that \tool is the state-of-the-art tool for a comprehensive assessment of the
robustness and trustworthiness of AI systems. Our research sheds light on a
general solution for AI systems testing landscape.


http://arxiv.org/abs/2411.05399
Post-Hoc Robustness Enhancement in Graph Neural Networks with Conditional Random Fields. (41%)
Yassine Abbahaddou; Sofiane Ennadir; Johannes F. Lutzeyer; Fragkiskos D. Malliaros; Michalis Vazirgiannis
  Graph Neural Networks (GNNs), which are nowadays the benchmark approach in
graph representation learning, have been shown to be vulnerable to adversarial
attacks, raising concerns about their real-world applicability. While existing
defense techniques primarily concentrate on the training phase of GNNs,
involving adjustments to message passing architectures or pre-processing
methods, there is a noticeable gap in methods focusing on increasing robustness
during inference. In this context, this study introduces RobustCRF, a post-hoc
approach aiming to enhance the robustness of GNNs at the inference stage. Our
proposed method, founded on statistical relational learning using a Conditional
Random Field, is model-agnostic and does not require prior knowledge about the
underlying model architecture. We validate the efficacy of this approach across
various models, leveraging benchmark node classification datasets.


http://arxiv.org/abs/2411.05345
Reasoning Robustness of LLMs to Adversarial Typographical Errors. (13%)
Esther Gan; Yiran Zhao; Liying Cheng; Yancan Mao; Anirudh Goyal; Kenji Kawaguchi; Min-Yen Kan; Michael Shieh
  Large Language Models (LLMs) have demonstrated impressive capabilities in
reasoning using Chain-of-Thought (CoT) prompting. However, CoT can be biased by
users' instruction. In this work, we study the reasoning robustness of LLMs to
typographical errors, which can naturally occur in users' queries. We design an
Adversarial Typo Attack ($\texttt{ATA}$) algorithm that iteratively samples
typos for words that are important to the query and selects the edit that is
most likely to succeed in attacking. It shows that LLMs are sensitive to
minimal adversarial typographical changes. Notably, with 1 character edit,
Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K, while with 8
character edits the performance further drops to 19.2%. To extend our
evaluation to larger and closed-source LLMs, we develop the $\texttt{R$^2$ATA}$
benchmark, which assesses models' $\underline{R}$easoning
$\underline{R}$obustness to $\underline{\texttt{ATA}}$. It includes adversarial
typographical questions derived from three widely used reasoning
datasets-GSM8K, BBH, and MMLU-by applying $\texttt{ATA}$ to open-source LLMs.
$\texttt{R$^2$ATA}$ demonstrates remarkable transferability and causes notable
performance drops across multiple super large and closed-source LLMs.


http://arxiv.org/abs/2411.05658
Towards a Re-evaluation of Data Forging Attacks in Practice. (2%)
Mohamed Suliman; Anisa Halimi; Swanand Kadhe; Nathalie Baracaldo; Douglas Leith
  Data forging attacks provide counterfactual proof that a model was trained on
a given dataset, when in fact, it was trained on another. These attacks work by
forging (replacing) mini-batches with ones containing distinct training
examples that produce nearly identical gradients. Data forging appears to break
any potential avenues for data governance, as adversarial model owners may
forge their training set from a dataset that is not compliant to one that is.
Given these serious implications on data auditing and compliance, we critically
analyse data forging from both a practical and theoretical point of view,
finding that a key practical limitation of current attack methods makes them
easily detectable by a verifier; namely that they cannot produce sufficiently
identical gradients. Theoretically, we analyse the question of whether two
distinct mini-batches can produce the same gradient. Generally, we find that
while there may exist an infinite number of distinct mini-batches with
real-valued training examples and labels that produce the same gradient,
finding those that are within the allowed domain e.g. pixel values between
0-255 and one hot labels is a non trivial task. Our results call for the
reevaluation of the strength of existing attacks, and for additional research
into successful data forging, given the serious consequences it may have on
machine learning and privacy.


http://arxiv.org/abs/2411.04533
Neural Fingerprints for Adversarial Attack Detection. (99%)
Haim Fisher; Moni Shahar; Yehezkel S. Resheff
  Deep learning models for image classification have become standard tools in
recent years. A well known vulnerability of these models is their
susceptibility to adversarial examples. These are generated by slightly
altering an image of a certain class in a way that is imperceptible to humans
but causes the model to classify it wrongly as another class. Many algorithms
have been proposed to address this problem, falling generally into one of two
categories: (i) building robust classifiers (ii) directly detecting attacked
images. Despite the good performance of these detectors, we argue that in a
white-box setting, where the attacker knows the configuration and weights of
the network and the detector, they can overcome the detector by running many
examples on a local copy, and sending only those that were not detected to the
actual model. This problem is common in security applications where even a very
good model is not sufficient to ensure safety. In this paper we propose to
overcome this inherent limitation of any static defence with randomization. To
do so, one must generate a very large family of detectors with consistent
performance, and select one or more of them randomly for each input. For the
individual detectors, we suggest the method of neural fingerprints. In the
training phase, for each class we repeatedly sample a tiny random subset of
neurons from certain layers of the network, and if their average is
sufficiently different between clean and attacked images of the focal class
they are considered a fingerprint and added to the detector bank. During test
time, we sample fingerprints from the bank associated with the label predicted
by the model, and detect attacks using a likelihood ratio test. We evaluate our
detectors on ImageNet with different attack methods and model architectures,
and show near-perfect detection with low rates of false detection.


http://arxiv.org/abs/2411.05056
Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models. (97%)
Pete Janowczyk; Linda Laurier; Ave Giulietta; Arlo Octavia; Meade Cleti
  Multi-Modal Language Models (MLLMs) have transformed artificial intelligence
by combining visual and text data, making applications like image captioning,
visual question answering, and multi-modal content creation possible. This
ability to understand and work with complex information has made MLLMs useful
in areas such as healthcare, autonomous systems, and digital content. However,
integrating multiple types of data also creates security risks. Attackers can
manipulate either the visual or text inputs, or both, to make the model produce
unintended or even harmful responses. This paper reviews how visual inputs in
MLLMs can be exploited by various attack strategies. We break down these
attacks into categories: simple visual tweaks and cross-modal manipulations, as
well as advanced strategies like VLATTACK, HADES, and Collaborative Multimodal
Adversarial Attack (Co-Attack). These attacks can mislead even the most robust
models while looking nearly identical to the original visuals, making them hard
to detect. We also discuss the broader security risks, including threats to
privacy and safety in important applications. To counter these risks, we review
current defense methods like the SmoothVLM framework, pixel-wise randomization,
and MirrorCheck, looking at their strengths and limitations. We also discuss
new methods to make MLLMs more secure, including adaptive defenses, better
evaluation tools, and security approaches that protect both visual and text
data. By bringing together recent developments and identifying key areas for
improvement, this review aims to support the creation of more secure and
reliable multi-modal AI systems for real-world use.


http://arxiv.org/abs/2411.04772
Attention Masks Help Adversarial Attacks to Bypass Safety Detectors. (97%)
Yunfan Shi
  Despite recent research advancements in adversarial attack methods, current
approaches against XAI monitors are still discoverable and slower. In this
paper, we present an adaptive framework for attention mask generation to enable
stealthy, explainable and efficient PGD image classification adversarial attack
under XAI monitors. Specifically, we utilize mutation XAI mixture and multitask
self-supervised X-UNet for attention mask generation to guide PGD attack.
Experiments on MNIST (MLP), CIFAR-10 (AlexNet) have shown that our system can
outperform benchmark PGD, Sparsefool and SOTA SINIFGSM in balancing among
stealth, efficiency and explainability which is crucial for effectively fooling
SOTA defense protected classifiers.


http://arxiv.org/abs/2411.04811
Defending Deep Regression Models against Backdoor Attacks. (78%)
Lingyu Du; Yupei Liu; Jinyuan Jia; Guohao Lan
  Deep regression models are used in a wide variety of safety-critical
applications, but are vulnerable to backdoor attacks. Although many defenses
have been proposed for classification models, they are ineffective as they do
not consider the uniqueness of regression models. First, the outputs of
regression models are continuous values instead of discretized labels. Thus,
the potential infected target of a backdoored regression model has infinite
possibilities, which makes it impossible to be determined by existing defenses.
Second, the backdoor behavior of backdoored deep regression models is triggered
by the activation values of all the neurons in the feature space, which makes
it difficult to be detected and mitigated using existing defenses. To resolve
these problems, we propose DRMGuard, the first defense to identify if a deep
regression model in the image domain is backdoored or not. DRMGuard formulates
the optimization problem for reverse engineering based on the unique
output-space and feature-space characteristics of backdoored deep regression
models. We conduct extensive evaluations on two regression tasks and four
datasets. The results show that DRMGuard can consistently defend against
various backdoor attacks. We also generalize four state-of-the-art defenses
designed for classifiers to regression models, and compare DRMGuard with them.
The results show that DRMGuard significantly outperforms all those defenses.


http://arxiv.org/abs/2411.05197
Hardware and Software Platform Inference. (9%)
Cheng Zhang; Hanna Foerster; Robert D. Mullins; Yiren Zhao; Ilia Shumailov
  It is now a common business practice to buy access to large language model
(LLM) inference rather than self-host, because of significant upfront hardware
infrastructure and energy costs. However, as a buyer, there is no mechanism to
verify the authenticity of the advertised service including the serving
hardware platform, e.g. that it is actually being served using an NVIDIA H100.
Furthermore, there are reports suggesting that model providers may deliver
models that differ slightly from the advertised ones, often to make them run on
less expensive hardware. That way, a client pays premium for a capable model
access on more expensive hardware, yet ends up being served by a (potentially
less capable) cheaper model on cheaper hardware. In this paper we introduce
\textit{\textbf{hardware and software platform inference (HSPI)}} -- a method
for identifying the underlying \GPU{} architecture and software stack of a
(black-box) machine learning model solely based on its input-output behavior.
Our method leverages the inherent differences of various \GPU{} architectures
and compilers to distinguish between different \GPU{} types and software
stacks. By analyzing the numerical patterns in the model's outputs, we propose
a classification framework capable of accurately identifying the \GPU{} used
for model inference as well as the underlying software configuration. Our
findings demonstrate the feasibility of inferring \GPU{} type from black-box
models. We evaluate HSPI against models served on different real hardware and
find that in a white-box setting we can distinguish between different \GPU{}s
with between $83.9\%$ and $100\%$ accuracy. Even in a black-box setting we are
able to achieve results that are up to three times higher than random guess
accuracy.


http://arxiv.org/abs/2411.05858
Saliency Assisted Quantization for Neural Networks. (1%)
Elmira Mousa Rezabeyk; Salar Beigzad; Yasin Hamzavi; Mohsen Bagheritabar; Seyedeh Sogol Mirikhoozani
  Deep learning methods have established a significant place in image
classification. While prior research has focused on enhancing final outcomes,
the opaque nature of the decision-making process in these models remains a
concern for experts. Additionally, the deployment of these methods can be
problematic in resource-limited environments. This paper tackles the inherent
black-box nature of these models by providing real-time explanations during the
training phase, compelling the model to concentrate on the most distinctive and
crucial aspects of the input. Furthermore, we employ established quantization
techniques to address resource constraints. To assess the effectiveness of our
approach, we explore how quantization influences the interpretability and
accuracy of Convolutional Neural Networks through a comparative analysis of
saliency maps from standard and quantized models. Quantization is implemented
during the training phase using the Parameterized Clipping Activation method,
with a focus on the MNIST and FashionMNIST benchmark datasets. We evaluated
three bit-width configurations (2-bit, 4-bit, and mixed 4/2-bit) to explore the
trade-off between efficiency and interpretability, with each configuration
designed to highlight varying impacts on saliency map clarity and model
accuracy. The results indicate that while quantization is crucial for
implementing models on resource-limited devices, it necessitates a trade-off
between accuracy and interpretability. Lower bit-widths result in more
pronounced reductions in both metrics, highlighting the necessity of meticulous
quantization parameter selection in applications where model transparency is
paramount. The study underscores the importance of achieving a balance between
efficiency and interpretability in the deployment of neural networks.


http://arxiv.org/abs/2411.04731
MISGUIDE: Security-Aware Attack Analytics for Smart Grid Load Frequency Control. (1%)
Nur Imtiazul Haque; Prabin Mali; Mohammad Zakaria Haider; Mohammad Ashiqur Rahman; Sumit Paudyal
  Incorporating advanced information and communication technologies into smart
grids (SGs) offers substantial operational benefits while increasing
vulnerability to cyber threats like false data injection (FDI) attacks. Current
SG attack analysis tools predominantly employ formal methods or adversarial
machine learning (ML) techniques with rule-based bad data detectors to analyze
the attack space. However, these attack analytics either generate simplistic
attack vectors detectable by the ML-based anomaly detection models (ADMs) or
fail to identify critical attack vectors from complex controller dynamics in a
feasible time. This paper introduces MISGUIDE, a novel defense-aware attack
analytics designed to extract verifiable multi-time slot-based FDI attack
vectors from complex SG load frequency control dynamics and ADMs, utilizing the
Gurobi optimizer. MISGUIDE can identify optimal (maliciously triggering
under/over frequency relays in minimal time) and stealthy attack vectors. Using
real-world load data, we validate the MISGUIDE-identified attack vectors
through real-time hardware-in-the-loop (OPALRT) simulations of the IEEE 39-bus
system.


http://arxiv.org/abs/2411.04376
Game-Theoretic Defenses for Robust Conformal Prediction Against Adversarial Attacks in Medical Imaging. (95%)
Rui Luo; Jie Bao; Zhixin Zhou; Chuangyin Dang
  Adversarial attacks pose significant threats to the reliability and safety of
deep learning models, especially in critical domains such as medical imaging.
This paper introduces a novel framework that integrates conformal prediction
with game-theoretic defensive strategies to enhance model robustness against
both known and unknown adversarial perturbations. We address three primary
research questions: constructing valid and efficient conformal prediction sets
under known attacks (RQ1), ensuring coverage under unknown attacks through
conservative thresholding (RQ2), and determining optimal defensive strategies
within a zero-sum game framework (RQ3). Our methodology involves training
specialized defensive models against specific attack types and employing
maximum and minimum classifiers to aggregate defenses effectively. Extensive
experiments conducted on the MedMNIST datasets, including PathMNIST,
OrganAMNIST, and TissueMNIST, demonstrate that our approach maintains high
coverage guarantees while minimizing prediction set sizes. The game-theoretic
analysis reveals that the optimal defensive strategy often converges to a
singular robust model, outperforming uniform and simple strategies across all
evaluated datasets. This work advances the state-of-the-art in uncertainty
quantification and adversarial robustness, providing a reliable mechanism for
deploying deep learning models in adversarial environments.


http://arxiv.org/abs/2411.03752
Deferred Poisoning: Making the Model More Vulnerable via Hessian Singularization. (86%)
Yuhao He; Jinyu Tian; Xianwei Zheng; Li Dong; Yuanman Li; Leo Yu Zhang; Jiantao Zhou
  Recent studies have shown that deep learning models are very vulnerable to
poisoning attacks. Many defense methods have been proposed to address this
issue. However, traditional poisoning attacks are not as threatening as
commonly believed. This is because they often cause differences in how the
model performs on the training set compared to the validation set. Such
inconsistency can alert defenders that their data has been poisoned, allowing
them to take the necessary defensive actions. In this paper, we introduce a
more threatening type of poisoning attack called the Deferred Poisoning Attack.
This new attack allows the model to function normally during the training and
validation phases but makes it very sensitive to evasion attacks or even
natural noise. We achieve this by ensuring the poisoned model's loss function
has a similar value as a normally trained model at each input sample but with a
large local curvature. A similar model loss ensures that there is no obvious
inconsistency between the training and validation accuracy, demonstrating high
stealthiness. On the other hand, the large curvature implies that a small
perturbation may cause a significant increase in model loss, leading to
substantial performance degradation, which reflects a worse robustness. We
fulfill this purpose by making the model have singular Hessian information at
the optimal point via our proposed Singularization Regularization term. We have
conducted both theoretical and empirical analyses of the proposed method and
validated its effectiveness through experiments on image classification tasks.
Furthermore, we have confirmed the hazards of this form of poisoning attack
under more general scenarios using natural noise, offering a new perspective
for research in the field of security.


http://arxiv.org/abs/2411.03861
FedSECA: Sign Election and Coordinate-wise Aggregation of Gradients for Byzantine Tolerant Federated Learning. (41%)
Joseph Geo Benjamin; Mothilal Asokan; Mohammad Yaqub; Karthik Nandakumar
  One of the most common defense strategies against Byzantine clients in
federated learning (FL) is to employ a robust aggregator mechanism that makes
the training more resilient. While many existing Byzantine robust aggregators
provide theoretical convergence guarantees and are empirically effective
against certain categories of attacks, we observe that certain high-strength
attacks can subvert the robust aggregator and collapse the training. To
overcome this limitation, we propose a method called FedSECA for robust Sign
Election and Coordinate-wise Aggregation of gradients in FL that is less
susceptible to malicious updates by an omniscient attacker. The proposed method
has two main components. The Concordance Ratio Induced Sign Election(CRISE)
module determines the consensus direction (elected sign) for each individual
parameter gradient through a weighted voting strategy. The client weights are
assigned based on a novel metric called concordance ratio, which quantifies the
degree of sign agreement between the client gradient updates. Based on the
elected sign, a Robust Coordinate-wise Aggregation(RoCA) strategy is employed,
where variance-reduced sparse gradients are aggregated only if they are in
alignment with the corresponding elected sign. We compare our proposed FedSECA
method against 10 robust aggregators under 7 Byzantine attacks on 3 datasets
and architectures. The results show that existing robust aggregators fail for
at least some attacks, while FedSECA exhibits better robustness. Code -
https://github.com/JosephGeoBenjamin/FedSECA-ByzantineTolerance


http://arxiv.org/abs/2411.03814
MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue. (11%)
Fengxiang Wang; Ranjie Duan; Peng Xiao; Xiaojun Jia; Shiji Zhao; Cheng Wei; YueFeng Chen; Chongwen Wang; Jialing Tao; Hang Su; Jun Zhu; Hui Xue
  Large Language Models (LLMs) demonstrate outstanding performance in their
reservoir of knowledge and understanding capabilities, but they have also been
shown to be prone to illegal or unethical reactions when subjected to jailbreak
attacks. To ensure their responsible deployment in critical applications, it is
crucial to understand the safety capabilities and vulnerabilities of LLMs.
Previous works mainly focus on jailbreak in single-round dialogue, overlooking
the potential jailbreak risks in multi-round dialogues, which are a vital way
humans interact with and extract information from LLMs. Some studies have
increasingly concentrated on the risks associated with jailbreak in multi-round
dialogues. These efforts typically involve the use of manually crafted
templates or prompt engineering techniques. However, due to the inherent
complexity of multi-round dialogues, their jailbreak performance is limited. To
solve this problem, we propose a novel multi-round dialogue jailbreaking agent,
emphasizing the importance of stealthiness in identifying and mitigating
potential threats to human values posed by LLMs. We propose a risk
decomposition strategy that distributes risks across multiple rounds of queries
and utilizes psychological strategies to enhance attack strength. Extensive
experiments show that our proposed method surpasses other attack methods and
achieves state-of-the-art attack success rate. We will make the corresponding
code and dataset available for future research. The code will be released soon.


http://arxiv.org/abs/2411.04365
Towards Secured Smart Grid 2.0: Exploring Security Threats, Protection Models, and Challenges. (4%)
Lan-Huong Nguyen; Van-Linh Nguyen; Ren-Hung Hwang; Jian-Jhih Kuo; Yu-Wen Chen; Chien-Chung Huang; Ping-I Pan
  Many nations are promoting the green transition in the energy sector to
attain neutral carbon emissions by 2050. Smart Grid 2.0 (SG2) is expected to
explore data-driven analytics and enhance communication technologies to improve
the efficiency and sustainability of distributed renewable energy systems.
These features are beyond smart metering and electric surplus distribution in
conventional smart grids. Given the high dependence on communication networks
to connect distributed microgrids in SG2, potential cascading failures of
connectivity can cause disruption to data synchronization to the remote control
systems. This paper reviews security threats and defense tactics for three
stakeholders: power grid operators, communication network providers, and
consumers. Through the survey, we found that SG2's stakeholders are
particularly vulnerable to substation attacks/vandalism, malware/ransomware
threats, blockchain vulnerabilities and supply chain breakdowns. Furthermore,
incorporating artificial intelligence (AI) into autonomous energy management in
distributed energy resources of SG2 creates new challenges. Accordingly,
adversarial samples and false data injection on electricity reading and
measurement sensors at power plants can fool AI-powered control functions and
cause messy error-checking operations in energy storage, wrong energy
estimation in electric vehicle charging, and even fraudulent transactions in
peer-to-peer energy trading models. Scalable blockchain-based models, physical
unclonable function, interoperable security protocols, and trustworthy AI
models designed for managing distributed microgrids in SG2 are typical
promising protection models for future research.


http://arxiv.org/abs/2411.05034
Mitigating Privacy Risks in LLM Embeddings from Embedding Inversion. (1%)
Tiantian Liu; Hongwei Yao; Tong Wu; Zhan Qin; Feng Lin; Kui Ren; Chun Chen
  Embeddings have become a cornerstone in the functionality of large language
models (LLMs) due to their ability to transform text data into rich, dense
numerical representations that capture semantic and syntactic properties. These
embedding vector databases serve as the long-term memory of LLMs, enabling
efficient handling of a wide range of natural language processing tasks.
However, the surge in popularity of embedding vector databases in LLMs has been
accompanied by significant concerns about privacy leakage. Embedding vector
databases are particularly vulnerable to embedding inversion attacks, where
adversaries can exploit the embeddings to reverse-engineer and extract
sensitive information from the original text data. Existing defense mechanisms
have shown limitations, often struggling to balance security with the
performance of downstream tasks. To address these challenges, we introduce
Eguard, a novel defense mechanism designed to mitigate embedding inversion
attacks. Eguard employs a transformer-based projection network and text mutual
information optimization to safeguard embeddings while preserving the utility
of LLMs. Our approach significantly reduces privacy risks, protecting over 95%
of tokens from inversion while maintaining high performance across downstream
tasks consistent with original embeddings.


http://arxiv.org/abs/2411.02871
Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Training. (99%)
Junhao Dong; Xinghua Qu; Z. Jane Wang; Yew-Soon Ong
  Despite remarkable achievements in deep learning across various domains, its
inherent vulnerability to adversarial examples still remains a critical concern
for practical deployment. Adversarial training has emerged as one of the most
effective defensive techniques for improving model robustness against such
malicious inputs. However, existing adversarial training schemes often lead to
limited generalization ability against underlying adversaries with diversity
due to their overreliance on a point-by-point augmentation strategy by mapping
each clean example to its adversarial counterpart during training. In addition,
adversarial examples can induce significant disruptions in the statistical
information w.r.t. the target model, thereby introducing substantial
uncertainty and challenges to modeling the distribution of adversarial
examples. To circumvent these issues, in this paper, we propose a novel
uncertainty-aware distributional adversarial training method, which enforces
adversary modeling by leveraging both the statistical information of
adversarial examples and its corresponding uncertainty estimation, with the
goal of augmenting the diversity of adversaries. Considering the potentially
negative impact induced by aligning adversaries to misclassified clean
examples, we also refine the alignment reference based on the statistical
proximity to clean examples during adversarial training, thereby reframing
adversarial training within a distribution-to-distribution matching framework
interacted between the clean and adversarial domains. Furthermore, we design an
introspective gradient alignment approach via matching input gradients between
these domains without introducing external models. Extensive experiments across
four benchmark datasets and various network architectures demonstrate that our
approach achieves state-of-the-art adversarial robustness and maintains natural
performance.


http://arxiv.org/abs/2411.02974
Region-Guided Attack on the Segment Anything Model (SAM). (99%)
Xiaoliang Liu; Furao Shen; Jian Zhao
  The Segment Anything Model (SAM) is a cornerstone of image segmentation,
demonstrating exceptional performance across various applications, particularly
in autonomous driving and medical imaging, where precise segmentation is
crucial. However, SAM is vulnerable to adversarial attacks that can
significantly impair its functionality through minor input perturbations.
Traditional techniques, such as FGSM and PGD, are often ineffective in
segmentation tasks due to their reliance on global perturbations that overlook
spatial nuances. Recent methods like Attack-SAM-K and UAD have begun to address
these challenges, but they frequently depend on external cues and do not fully
leverage the structural interdependencies within segmentation processes. This
limitation underscores the need for a novel adversarial strategy that exploits
the unique characteristics of segmentation tasks. In response, we introduce the
Region-Guided Attack (RGA), designed specifically for SAM. RGA utilizes a
Region-Guided Map (RGM) to manipulate segmented regions, enabling targeted
perturbations that fragment large segments and expand smaller ones, resulting
in erroneous outputs from SAM. Our experiments demonstrate that RGA achieves
high success rates in both white-box and black-box scenarios, emphasizing the
need for robust defenses against such sophisticated attacks. RGA not only
reveals SAM's vulnerabilities but also lays the groundwork for developing more
resilient defenses against adversarial threats in image segmentation.


http://arxiv.org/abs/2411.02866
Double Whammy: Stealthy Data Manipulation aided Reconstruction Attack on Graph Federated Learning. (91%)
Jinyin Chen; Minying Ma; Haibin Zheng; Qi Xuan
  Recent research has constructed successful graph reconstruction attack (GRA)
on GFL. But these attacks are still challenged in aspects of effectiveness and
stealth. To address the issues, we propose the first Data Manipulation aided
Reconstruction attack on GFL, dubbed as DMan4Rec. The malicious client is born
to manipulate its locally collected data to enhance graph stealing privacy from
benign ones, so as to construct double whammy on GFL. It differs from previous
work in three terms: (1) effectiveness - to fully utilize the sparsity and
feature smoothness of the graph, novel penalty terms are designed adaptive to
diverse similarity functions for connected and unconnected node pairs, as well
as incorporation label smoothing on top of the original cross-entropy loss. (2)
scalability - DMan4Rec is capable of both white-box and black-box attacks via
training a supervised model to infer the posterior probabilities obtained from
limited queries (3) stealthiness - by manipulating the malicious client's node
features, it can maintain the overall graph structure's invariance and conceal
the attack. Comprehensive experiments on four real datasets and three GNN
models demonstrate that DMan4Rec achieves the state-of-the-art (SOTA) attack
performance, e.g., the attack AUC and precision improved by 9.2% and 10.5%
respectively compared with the SOTA baselines. Particularly, DMan4Rec achieves
an AUC score and a precision score of up to 99.59% and 99.56%, respectively in
black-box setting. Nevertheless, the complete overlap of the distribution
graphs supports the stealthiness of the attack. Besides, DMan4Rec still beats
the defensive GFL, which alarms a new threat to GFL.


http://arxiv.org/abs/2411.03022
Flashy Backdoor: Real-world Environment Backdoor Attack on SNNs with DVS Cameras. (75%)
Roberto Riaño; Gorka Abad; Stjepan Picek; Aitor Urbieta
  While security vulnerabilities in traditional Deep Neural Networks (DNNs)
have been extensively studied, the susceptibility of Spiking Neural Networks
(SNNs) to adversarial attacks remains mostly underexplored. Until now, the
mechanisms to inject backdoors into SNN models have been limited to digital
scenarios; thus, we present the first evaluation of backdoor attacks in
real-world environments.
  We begin by assessing the applicability of existing digital backdoor attacks
and identifying their limitations for deployment in physical environments. To
address each of the found limitations, we present three novel backdoor attack
methods on SNNs, i.e., Framed, Strobing, and Flashy Backdoor. We also assess
the effectiveness of traditional backdoor procedures and defenses adapted for
SNNs, such as pruning, fine-tuning, and fine-pruning. The results show that
while these procedures and defenses can mitigate some attacks, they often fail
against stronger methods like Flashy Backdoor or sacrifice too much clean
accuracy, rendering the models unusable.
  Overall, all our methods can achieve up to a 100% Attack Success Rate while
maintaining high clean accuracy in every tested dataset. Additionally, we
evaluate the stealthiness of the triggers with commonly used metrics, finding
them highly stealthy. Thus, we propose new alternatives more suited for
identifying poisoned samples in these scenarios. Our results show that further
research is needed to ensure the security of SNN-based systems against backdoor
attacks and their safe application in real-world scenarios. The code,
experiments, and results are available in our repository.


http://arxiv.org/abs/2411.03231
Formal Logic-guided Robust Federated Learning against Poisoning Attacks. (68%)
Dung Thuy Nguyen; Ziyan An; Taylor T. Johnson; Meiyi Ma; Kevin Leach
  Federated Learning (FL) offers a promising solution to the privacy concerns
associated with centralized Machine Learning (ML) by enabling decentralized,
collaborative learning. However, FL is vulnerable to various security threats,
including poisoning attacks, where adversarial clients manipulate the training
data or model updates to degrade overall model performance. Recognizing this
threat, researchers have focused on developing defense mechanisms to counteract
poisoning attacks in FL systems. However, existing robust FL methods
predominantly focus on computer vision tasks, leaving a gap in addressing the
unique challenges of FL with time series data. In this paper, we present
FLORAL, a defense mechanism designed to mitigate poisoning attacks in federated
learning for time-series tasks, even in scenarios with heterogeneous client
data and a large number of adversarial participants. Unlike traditional
model-centric defenses, FLORAL leverages logical reasoning to evaluate client
trustworthiness by aligning their predictions with global time-series patterns,
rather than relying solely on the similarity of client updates. Our approach
extracts logical reasoning properties from clients, then hierarchically infers
global properties, and uses these to verify client updates. Through formal
logic verification, we assess the robustness of each client contribution,
identifying deviations indicative of adversarial behavior. Experimental results
on two datasets demonstrate the superior performance of our approach compared
to existing baseline methods, highlighting its potential to enhance the
robustness of FL to time series applications. Notably, FLORAL reduced the
prediction error by 93.27\% in the best-case scenario compared to the
second-best baseline. Our code is available at
\url{https://anonymous.4open.science/r/FLORAL-Robust-FTS}.


http://arxiv.org/abs/2411.03279
Oblivious Defense in ML Models: Backdoor Removal without Detection. (15%)
Shafi Goldwasser; Jonathan Shafer; Neekon Vafa; Vinod Vaikuntanathan
  As society grows more reliant on machine learning, ensuring the security of
machine learning systems against sophisticated attacks becomes a pressing
concern. A recent result of Goldwasser, Kim, Vaikuntanathan, and Zamir (2022)
shows that an adversary can plant undetectable backdoors in machine learning
models, allowing the adversary to covertly control the model's behavior.
Backdoors can be planted in such a way that the backdoored machine learning
model is computationally indistinguishable from an honest model without
backdoors.
  In this paper, we present strategies for defending against backdoors in ML
models, even if they are undetectable. The key observation is that it is
sometimes possible to provably mitigate or even remove backdoors without
needing to detect them, using techniques inspired by the notion of random
self-reducibility. This depends on properties of the ground-truth labels
(chosen by nature), and not of the proposed ML model (which may be chosen by an
attacker).
  We give formal definitions for secure backdoor mitigation, and proceed to
show two types of results. First, we show a "global mitigation" technique,
which removes all backdoors from a machine learning model under the assumption
that the ground-truth labels are close to a Fourier-heavy function. Second, we
consider distributions where the ground-truth labels are close to a linear or
polynomial function in $\mathbb{R}^n$. Here, we show "local mitigation"
techniques, which remove backdoors with high probability for every inputs of
interest, and are computationally cheaper than global mitigation. All of our
constructions are black-box, so our techniques work without needing access to
the model's representation (i.e., its code or parameters). Along the way we
prove a simple result for robust mean estimation.


http://arxiv.org/abs/2411.03364
DM4Steal: Diffusion Model For Link Stealing Attack On Graph Neural Networks. (13%)
Jinyin Chen; Haonan Ma; Haibin Zheng
  Graph has become increasingly integral to the advancement of recommendation
systems, particularly with the fast development of graph neural network(GNN).
By exploring the virtue of rich node features and link information, GNN is
designed to provide personalized and accurate suggestions. Meanwhile, the
privacy leakage of GNN in such contexts has also captured special attention.
Prior work has revealed that a malicious user can utilize auxiliary knowledge
to extract sensitive link data of the target graph, integral to recommendation
systems, via the decision made by the target GNN model. This poses a
significant risk to the integrity and confidentiality of data used in
recommendation system. Though important, previous works on GNN's privacy
leakage are still challenged in three aspects, i.e., limited stealing attack
scenarios, sub-optimal attack performance, and adaptation against defense. To
address these issues, we propose a diffusion model based link stealing attack,
named DM4Steal. It differs previous work from three critical aspects. (i)
Generality: aiming at six attack scenarios with limited auxiliary knowledge, we
propose a novel training strategy for diffusion models so that DM4Steal is
transferable to diverse attack scenarios. (ii) Effectiveness: benefiting from
the retention of semantic structure in the diffusion model during the training
process, DM4Steal is capable to learn the precise topology of the target graph
through the GNN decision process. (iii) Adaptation: when GNN is defensive
(e.g., DP, Dropout), DM4Steal relies on the stability that comes from sampling
the score model multiple times to keep performance degradation to a minimum,
thus DM4Steal implements successful adaptive attack on defensive GNN.


http://arxiv.org/abs/2411.03019
FEDLAD: Federated Evaluation of Deep Leakage Attacks and Defenses. (9%)
Isaac Baglin; Xiatian Zhu; Simon Hadfield
  Federated Learning is a privacy preserving decentralized machine learning
paradigm designed to collaboratively train models across multiple clients by
exchanging gradients to the server and keeping private data local.
Nevertheless, recent research has revealed that the security of Federated
Learning is compromised, as private ground truth data can be recovered through
a gradient inversion technique known as Deep Leakage. While these attacks are
crafted with a focus on applications in Federated Learning, they generally are
not evaluated in realistic scenarios. This paper introduces the FEDLAD
Framework (Federated Evaluation of Deep Leakage Attacks and Defenses), a
comprehensive benchmark for evaluating Deep Leakage attacks and defenses within
a realistic Federated context. By implementing a unified benchmark that
encompasses multiple state-of-the-art Deep Leakage techniques and various
defense strategies, our framework facilitates the evaluation and comparison of
the efficacy of these methods across different datasets and training states.
This work highlights a crucial trade-off between privacy and model accuracy in
Federated Learning and aims to advance the understanding of security challenges
in decentralized machine learning systems, stimulate future research, and
enhance reproducibility in evaluating Deep Leakage attacks and defenses.


http://arxiv.org/abs/2411.02833
Lost in Context: The Influence of Context on Feature Attribution Methods for Object Recognition. (3%)
Sayanta Adhikari; Rishav Kumar; Konda Reddy Mopuri; Rajalakshmi Pachamuthu
  Contextual information plays a critical role in object recognition models
within computer vision, where changes in context can significantly affect
accuracy, underscoring models' dependence on contextual cues. This study
investigates how context manipulation influences both model accuracy and
feature attribution, providing insights into the reliance of object recognition
models on contextual information as understood through the lens of feature
attribution methods.
  We employ a range of feature attribution techniques to decipher the reliance
of deep neural networks on context in object recognition tasks. Using the
ImageNet-9 and our curated ImageNet-CS datasets, we conduct experiments to
evaluate the impact of contextual variations, analyzed through feature
attribution methods. Our findings reveal several key insights: (a) Correctly
classified images predominantly emphasize object volume attribution over
context volume attribution. (b) The dependence on context remains relatively
stable across different context modifications, irrespective of classification
accuracy. (c) Context change exerts a more pronounced effect on model
performance than Context perturbations. (d) Surprisingly, context attribution
in `no-information' scenarios is non-trivial. Our research moves beyond
traditional methods by assessing the implications of broad-level modifications
on object recognition, either in the object or its context.


http://arxiv.org/abs/2411.03554
Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset. (1%)
Yingzi Ma; Jiongxiao Wang; Fei Wang; Siyuan Ma; Jiazhao Li; Jinsheng Pan; Xiujun Li; Furong Huang; Lichao Sun; Bo Li; Yejin Choi; Muhao Chen; Chaowei Xiao
  Machine unlearning has emerged as an effective strategy for forgetting
specific information in the training data. However, with the increasing
integration of visual data, privacy concerns in Vision Language Models (VLMs)
remain underexplored. To address this, we introduce Facial Identity Unlearning
Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly
evaluate the effectiveness of unlearning algorithms under the Right to be
Forgotten setting. Specifically, we formulate the VLM unlearning task via
constructing the Fictitious Facial Identity VQA dataset and apply a two-stage
evaluation pipeline that is designed to precisely control the sources of
information and their exposure levels. In terms of evaluation, since VLM
supports various forms of ways to ask questions with the same semantic meaning,
we also provide robust evaluation metrics including membership inference
attacks and carefully designed adversarial privacy attacks to evaluate the
performance of algorithms. Through the evaluation of four baseline VLM
unlearning algorithms within FIUBench, we find that all methods remain limited
in their unlearning performance, with significant trade-offs between model
utility and forget quality. Furthermore, our findings also highlight the
importance of privacy attacks for robust evaluations. We hope FIUBench will
drive progress in developing more effective VLM unlearning algorithms.


http://arxiv.org/abs/2411.03312
Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters. (1%)
Kevin Y. Li; Sachin Goyal; Joao D. Semedo; J. Zico Kolter
  Vision Language Models (VLMs) have demonstrated strong capabilities across
various visual understanding and reasoning tasks, driven by incorporating image
representations into the token inputs of Large Language Models (LLMs). However,
their real-world deployment is often constrained by high latency during
inference due to the substantial compute required by the LLM to process the
large number of input tokens, predominantly arising from the image. To reduce
inference costs, one can either downsize the LLM or reduce the number of input
tokens needed to represent the image, the latter of which has been the focus of
many recent efforts around token compression. However, it is unclear what the
optimal trade-off is given a fixed inference budget. We first characterize this
optimal trade-off between the number of visual tokens and LLM parameters by
establishing scaling laws that capture variations in performance with these two
factors. Our results reveal a surprising trend: for visual reasoning tasks, the
inference-optimal behavior in VLMs is achieved by using the largest LLM that
fits within the inference budget while minimizing visual token count - often to
a single token. While the token reduction literature has mainly focused on
maintaining base model performance by modestly reducing the token count (e.g.,
$5-10\times$), our results indicate that the compute-optimal inference regime
requires operating under even higher token compression ratios. Based on these
insights, we take the first steps toward designing token compression algorithms
tailored for high-compression settings, utilizing prompt-based compression of
tokens. Our work underscores the performance and efficiency benefits of
operating in low visual token regimes and the importance of developing tailored
token reduction algorithms for such conditions. Code is available at
https://github.com/locuslab/llava-token-compression.


http://arxiv.org/abs/2411.02809
Query-Efficient Adversarial Attack Against Vertical Federated Graph Learning. (99%)
Jinyin Chen; Wenbo Mu; Luxin Zhang; Guohan Huang; Haibin Zheng; Yao Cheng
  Graph neural network (GNN) has captured wide attention due to its capability
of graph representation learning for graph-structured data. However, the
distributed data silos limit the performance of GNN. Vertical federated
learning (VFL), an emerging technique to process distributed data, successfully
makes GNN possible to handle the distributed graph-structured data. Despite the
prosperous development of vertical federated graph learning (VFGL), the
robustness of VFGL against the adversarial attack has not been explored yet.
Although numerous adversarial attacks against centralized GNNs are proposed,
their attack performance is challenged in the VFGL scenario. To the best of our
knowledge, this is the first work to explore the adversarial attack against
VFGL. A query-efficient hybrid adversarial attack framework is proposed to
significantly improve the centralized adversarial attacks against VFGL, denoted
as NA2, short for Neuron-based Adversarial Attack. Specifically, a malicious
client manipulates its local training data to improve its contribution in a
stealthy fashion. Then a shadow model is established based on the manipulated
data to simulate the behavior of the server model in VFGL. As a result, the
shadow model can improve the attack success rate of various centralized attacks
with a few queries. Extensive experiments on five real-world benchmarks
demonstrate that NA2 improves the performance of the centralized adversarial
attacks against VFGL, achieving state-of-the-art performance even under
potential adaptive defense where the defender knows the attack method.
Additionally, we provide interpretable experiments of the effectiveness of NA2
via sensitive neurons identification and visualization of t-SNE.


http://arxiv.org/abs/2411.02669
Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack. (99%)
Xiaojun Jia; Sensen Gao; Qing Guo; Ke Ma; Yihao Huang; Simeng Qin; Yang Liu; Ivor Tsang Fellow; Xiaochun Cao
  Vision-language pre-training (VLP) models excel at interpreting both images
and text but remain vulnerable to multimodal adversarial examples (AEs).
Advancing the generation of transferable AEs, which succeed across unseen
models, is key to developing more robust and practical VLP models. Previous
approaches augment image-text pairs to enhance diversity within the adversarial
example generation process, aiming to improve transferability by expanding the
contrast space of image-text features. However, these methods focus solely on
diversity around the current AEs, yielding limited gains in transferability. To
address this issue, we propose to increase the diversity of AEs by leveraging
the intersection regions along the adversarial trajectory during optimization.
Specifically, we propose sampling from adversarial evolution triangles composed
of clean, historical, and current adversarial examples to enhance adversarial
diversity. We provide a theoretical analysis to demonstrate the effectiveness
of the proposed adversarial evolution triangle. Moreover, we find that
redundant inactive dimensions can dominate similarity calculations, distorting
feature matching and making AEs model-dependent with reduced transferability.
Hence, we propose to generate AEs in the semantic image-text feature contrast
space, which can project the original feature space into a semantic corpus
subspace. The proposed semantic-aligned subspace can reduce the image feature
redundancy, thereby improving adversarial transferability. Extensive
experiments across different datasets and models demonstrate that the proposed
method can effectively improve adversarial transferability and outperform
state-of-the-art adversarial attack methods. The code is released at
https://github.com/jiaxiaojunQAQ/SA-AET.


http://arxiv.org/abs/2411.01889
LiDAttack: Robust Black-box Attack on LiDAR-based Object Detection. (99%)
Jinyin Chen; Danxin Liao; Sheng Xiang; Haibin Zheng
  Since DNN is vulnerable to carefully crafted adversarial examples,
adversarial attack on LiDAR sensors have been extensively studied. We introduce
a robust black-box attack dubbed LiDAttack. It utilizes a genetic algorithm
with a simulated annealing strategy to strictly limit the location and number
of perturbation points, achieving a stealthy and effective attack. And it
simulates scanning deviations, allowing it to adapt to dynamic changes in real
world scenario variations. Extensive experiments are conducted on 3 datasets
(i.e., KITTI, nuScenes, and self-constructed data) with 3 dominant object
detection models (i.e., PointRCNN, PointPillar, and PV-RCNN++). The results
reveal the efficiency of the LiDAttack when targeting a wide range of object
detection models, with an attack success rate (ASR) up to 90%.


http://arxiv.org/abs/2411.02094
Alignment-Based Adversarial Training (ABAT) for Improving the Robustness and Accuracy of EEG-Based BCIs. (91%)
Xiaoqing Chen; Ziwei Wang; Dongrui Wu
  Machine learning has achieved great success in electroencephalogram (EEG)
based brain-computer interfaces (BCIs). Most existing BCI studies focused on
improving the decoding accuracy, with only a few considering the adversarial
security. Although many adversarial defense approaches have been proposed in
other application domains such as computer vision, previous research showed
that their direct extensions to BCIs degrade the classification accuracy on
benign samples. This phenomenon greatly affects the applicability of
adversarial defense approaches to EEG-based BCIs. To mitigate this problem, we
propose alignment-based adversarial training (ABAT), which performs EEG data
alignment before adversarial training. Data alignment aligns EEG trials from
different domains to reduce their distribution discrepancies, and adversarial
training further robustifies the classification boundary. The integration of
data alignment and adversarial training can make the trained EEG classifiers
simultaneously more accurate and more robust. Experiments on five EEG datasets
from two different BCI paradigms (motor imagery classification, and event
related potential recognition), three convolutional neural network classifiers
(EEGNet, ShallowCNN and DeepCNN) and three different experimental settings
(offline within-subject cross-block/-session classification, online
cross-session classification, and pre-trained classifiers) demonstrated its
effectiveness. It is very intriguing that adversarial attacks, which are
usually used to damage BCI systems, can be used in ABAT to simultaneously
improve the model accuracy and robustness.


http://arxiv.org/abs/2411.02391
Attacking Vision-Language Computer Agents via Pop-ups. (9%)
Yanzhe Zhang; Tao Yu; Diyi Yang
  Autonomous agents powered by large vision and language models (VLM) have
demonstrated significant potential in completing daily computer tasks, such as
browsing the web to book travel and operating desktop software, which requires
agents to understand these interfaces. Despite such visual inputs becoming more
integrated into agentic applications, what types of risks and attacks exist
around them still remain unclear. In this work, we demonstrate that VLM agents
can be easily attacked by a set of carefully designed adversarial pop-ups,
which human users would typically recognize and ignore. This distraction leads
agents to click these pop-ups instead of performing their tasks as usual.
Integrating these pop-ups into existing agent testing environments like OSWorld
and VisualWebArena leads to an attack success rate (the frequency of the agent
clicking the pop-ups) of 86% on average and decreases the task success rate by
47%. Basic defense techniques, such as asking the agent to ignore pop-ups or
including an advertisement notice, are ineffective against the attack.


http://arxiv.org/abs/2411.02785
Stochastic Monkeys at Play: Random Augmentations Cheaply Break LLM Safety Alignment. (2%)
Jason Vega; Junsheng Huang; Gaokai Zhang; Hangoo Kang; Minjia Zhang; Gagandeep Singh
  Safety alignment of Large Language Models (LLMs) has recently become a
critical objective of model developers. In response, a growing body of work has
been investigating how safety alignment can be bypassed through various
jailbreaking methods, such as adversarial attacks. However, these jailbreak
methods can be rather costly or involve a non-trivial amount of creativity and
effort, introducing the assumption that malicious users are high-resource or
sophisticated. In this paper, we study how simple random augmentations to the
input prompt affect safety alignment effectiveness in state-of-the-art LLMs,
such as Llama 3 and Qwen 2. We perform an in-depth evaluation of 17 different
models and investigate the intersection of safety under random augmentations
with multiple dimensions: augmentation type, model size, quantization,
fine-tuning-based defenses, and decoding strategies (e.g., sampling
temperature). We show that low-resource and unsophisticated attackers, i.e.
$\textit{stochastic monkeys}$, can significantly improve their chances of
bypassing alignment with just 25 random augmentations per prompt.


http://arxiv.org/abs/2411.02603
FactTest: Factuality Testing in Large Language Models with Statistical Guarantees. (1%)
Fan Nie; Xiaotian Hou; Shuhang Lin; James Zou; Huaxiu Yao; Linjun Zhang
  The propensity of Large Language Models (LLMs) to generate hallucinations and
non-factual content undermines their reliability in high-stakes domains, where
rigorous control over Type I errors (the conditional probability of incorrectly
classifying hallucinations as truthful content) is essential. Despite its
importance, formal verification of LLM factuality with such guarantees remains
largely unexplored. In this paper, we introduce FactTest, a novel framework
that statistically assesses whether an LLM can confidently provide correct
answers to given questions with high-probability correctness guarantees. We
formulate factuality testing as hypothesis testing problem to enforce an upper
bound of Type I errors at user-specified significance levels. Notably, we prove
that our framework also ensures strong Type II error control under mild
conditions and can be extended to maintain its effectiveness when covariate
shifts exist. %These analyses are amenable to the principled NP framework. Our
approach is distribution-free and works for any number of human-annotated
samples. It is model-agnostic and applies to any black-box or white-box LM.
Extensive experiments on question-answering (QA) and multiple-choice benchmarks
demonstrate that \approach effectively detects hallucinations and improves the
model's ability to abstain from answering unknown questions, leading to an over
40% accuracy improvement.


http://arxiv.org/abs/2411.02099
Differentially Private Integrated Decision Gradients (IDG-DP) for Radar-based Human Activity Recognition. (1%)
Idris Zakariyya; Linda Tran; Kaushik Bhargav Sivangi; Paul Henderson; Fani Deligianni
  Human motion analysis offers significant potential for healthcare monitoring
and early detection of diseases. The advent of radar-based sensing systems has
captured the spotlight for they are able to operate without physical contact
and they can integrate with pre-existing Wi-Fi networks. They are also seen as
less privacy-invasive compared to camera-based systems. However, recent
research has shown high accuracy in recognizing subjects or gender from radar
gait patterns, raising privacy concerns. This study addresses these issues by
investigating privacy vulnerabilities in radar-based Human Activity Recognition
(HAR) systems and proposing a novel method for privacy preservation using
Differential Privacy (DP) driven by attributions derived with Integrated
Decision Gradient (IDG) algorithm. We investigate Black-box Membership
Inference Attack (MIA) Models in HAR settings across various levels of
attacker-accessible information. We extensively evaluated the effectiveness of
the proposed IDG-DP method by designing a CNN-based HAR model and rigorously
assessing its resilience against MIAs. Experimental results demonstrate the
potential of IDG-DP in mitigating privacy attacks while maintaining utility
across all settings, particularly excelling against label-only and shadow model
black-box MIA attacks. This work represents a crucial step towards balancing
the need for effective radar-based HAR with robust privacy protection in
healthcare environments.


http://arxiv.org/abs/2411.03348
Undermining Image and Text Classification Algorithms Using Adversarial Attacks. (98%)
Langalibalele Lunga; Suhas Sreehari
  Machine learning models are prone to adversarial attacks, where inputs can be
manipulated in order to cause misclassifications. While previous research has
focused on techniques like Generative Adversarial Networks (GANs), there's
limited exploration of GANs and Synthetic Minority Oversampling Technique
(SMOTE) in text and image classification models to perform adversarial attacks.
Our study addresses this gap by training various machine learning models and
using GANs and SMOTE to generate additional data points aimed at attacking text
classification models. Furthermore, we extend our investigation to face
recognition models, training a Convolutional Neural Network(CNN) and subjecting
it to adversarial attacks with fast gradient sign perturbations on key features
identified by GradCAM, a technique used to highlight key image characteristics
CNNs use in classification. Our experiments reveal a significant vulnerability
in classification models. Specifically, we observe a 20 % decrease in accuracy
for the top-performing text classification models post-attack, along with a 30
% decrease in facial recognition accuracy. This highlights the susceptibility
of these models to manipulation of input data. Adversarial attacks not only
compromise the security but also undermine the reliability of machine learning
systems. By showcasing the impact of adversarial attacks on both text
classification and face recognition models, our study underscores the urgent
need for develop robust defenses against such vulnerabilities.


http://arxiv.org/abs/2411.01565
SQL Injection Jailbreak: a structural disaster of large language models. (78%)
Jiawei Zhao; Kejiang Chen; Weiming Zhang; Nenghai Yu
  In recent years, the rapid development of large language models (LLMs) has
brought new vitality to the various domains and generated substantial social
and economic benefits. However, the swift advancement of LLMs has introduced
new security vulnerabilities. Jailbreak, a form of attack that induces LLMs to
output harmful content through carefully crafted prompts, poses a challenge to
the safe and trustworthy development of LLMs. Previous jailbreak attack methods
primarily exploited the internal capabilities of the model. Among them, one
category leverages the model's implicit capabilities for jailbreak attacks,
where the attacker is unaware of the exact reasons for the attack's success.
The other category utilizes the model's explicit capabilities for jailbreak
attacks, where the attacker understands the reasons for the attack's success.
For example, these attacks exploit the model's abilities in coding, contextual
learning, or understanding ASCII characters. However, these earlier jailbreak
attacks have certain limitations, as they only exploit the inherent
capabilities of the model. In this paper, we propose a novel jailbreak method,
SQL Injection Jailbreak (SIJ), which utilizes the construction of input prompts
by LLMs to inject jailbreak information into user prompts, enabling successful
jailbreak of the LLMs. Our SIJ method achieves nearly 100\% attack success
rates on five well-known open-source LLMs in the context of AdvBench, while
incurring lower time costs compared to previous methods. More importantly, SIJ
reveals a new vulnerability in LLMs that urgently needs to be addressed. To
this end, we propose a defense method called Self-Reminder-Key and demonstrate
its effectiveness through experiments. Our code is available at
\href{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}{https://github.com/weiyezhimeng/SQL-Injection-Jailbreak}.


http://arxiv.org/abs/2411.01703
UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models. (4%)
Sejoon Oh; Yiqiao Jin; Megha Sharma; Donghyun Kim; Eric Ma; Gaurav Verma; Srijan Kumar
  Multimodal large language models (MLLMs) have revolutionized vision-language
understanding but remain vulnerable to multimodal jailbreak attacks, where
adversarial inputs are meticulously crafted to elicit harmful or inappropriate
responses. We propose UniGuard, a novel multimodal safety guardrail that
jointly considers the unimodal and cross-modal harmful signals. UniGuard trains
a multimodal guardrail to minimize the likelihood of generating harmful
responses in a toxic corpus. The guardrail can be seamlessly applied to any
input prompt during inference with minimal computational costs. Extensive
experiments demonstrate the generalizability of UniGuard across multiple
modalities, attack strategies, and multiple state-of-the-art MLLMs, including
LLaVA, Gemini Pro, GPT-4o, MiniGPT-4, and InstructBLIP. Notably, this robust
defense mechanism maintains the models' overall vision-language understanding
capabilities.


http://arxiv.org/abs/2411.01748
Rotation Perturbation Robustness in Point Cloud Analysis: A Perspective of Manifold Distillation. (2%)
Xinyu Xu; Huazhen Liu; Feiming Wei; Huilin Xiong; Wenxian Yu; Tao Zhang
  Point cloud is often regarded as a discrete sampling of Riemannian manifold
and plays a pivotal role in the 3D image interpretation. Particularly, rotation
perturbation, an unexpected small change in rotation caused by various factors
(like equipment offset, system instability, measurement errors and so on), can
easily lead to the inferior results in point cloud learning tasks. However,
classical point cloud learning methods are sensitive to rotation perturbation,
and the existing networks with rotation robustness also have much room for
improvements in terms of performance and noise tolerance. Given these, this
paper remodels the point cloud from the perspective of manifold as well as
designs a manifold distillation method to achieve the robustness of rotation
perturbation without any coordinate transformation. In brief, during the
training phase, we introduce a teacher network to learn the rotation robustness
information and transfer this information to the student network through online
distillation. In the inference phase, the student network directly utilizes the
original 3D coordinate information to achieve the robustness of rotation
perturbation. Experiments carried out on four different datasets verify the
effectiveness of our method. Averagely, on the Modelnet40 and ScanobjectNN
classification datasets with random rotation perturbations, our classification
accuracy has respectively improved by 4.92% and 4.41%, compared to popular
rotation-robust networks; on the ShapeNet and S3DIS segmentation datasets,
compared to the rotation-robust networks, the improvements of mIoU are 7.36%
and 4.82%, respectively. Besides, from the experimental results, the proposed
algorithm also shows excellent performance in resisting noise and outliers.


http://arxiv.org/abs/2412.06788
Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant. (1%)
Ivan A. Fernandez; Subash Neupane; Sudip Mittal; Shahram Rahimi
  Recent research has shown that large language models (LLMs) are particularly
vulnerable to adversarial attacks. Since the release of ChatGPT, various
industries are adopting LLM-based chatbots and virtual assistants in their data
workflows. The rapid development pace of AI-based systems is being driven by
the potential of Generative AI (GenAI) to assist humans in decision making. The
immense optimism behind GenAI often overshadows the adversarial risks
associated with these technologies. A threat actor can use security gaps, poor
safeguards, and limited data governance to carry out attacks that grant
unauthorized access to the system and its data. As a proof-of-concept, we
assess the performance of BarkPlug, the Mississippi State University chatbot,
against data poison attacks from a red team perspective.


http://arxiv.org/abs/2411.01779
TabSec: A Collaborative Framework for Novel Insider Threat Detection. (1%)
Zilin Huang; Xiangyan Tang; Hongyu Li; Xinyi Cao; Jieren Cheng
  In the era of the Internet of Things (IoT) and data sharing, users frequently
upload their personal information to enterprise databases to enjoy enhanced
service experiences provided by various online services. However, the
widespread presence of system vulnerabilities, remote network intrusions, and
insider threats significantly increases the exposure of private enterprise data
on the internet. If such data is stolen or leaked by attackers, it can result
in severe asset losses and business operation disruptions. To address these
challenges, this paper proposes a novel threat detection framework, TabITD.
This framework integrates Intrusion Detection Systems (IDS) with User and
Entity Behavior Analytics (UEBA) strategies to form a collaborative detection
system that bridges the gaps in existing systems' capabilities. It effectively
addresses the blurred boundaries between external and insider threats caused by
the diversification of attack methods, thereby enhancing the model's learning
ability and overall detection performance. Moreover, the proposed method
leverages the TabNet architecture, which employs a sparse attention feature
selection mechanism that allows TabNet to select the most relevant features at
each decision step, thereby improving the detection of rare-class attacks. We
evaluated our proposed solution on two different datasets, achieving average
accuracies of 96.71% and 97.25%, respectively. The results demonstrate that
this approach can effectively detect malicious behaviors such as masquerade
attacks and external threats, significantly enhancing network security defenses
and the efficiency of network attack detection.


http://arxiv.org/abs/2411.01777
Learning predictable and robust neural representations by straightening image sequences. (1%)
Xueyan Niu; Cristina Savin; Eero P. Simoncelli
  Prediction is a fundamental capability of all living organisms, and has been
proposed as an objective for learning sensory representations. Recent work
demonstrates that in primate visual systems, prediction is facilitated by
neural representations that follow straighter temporal trajectories than their
initial photoreceptor encoding, which allows for prediction by linear
extrapolation. Inspired by these experimental findings, we develop a
self-supervised learning (SSL) objective that explicitly quantifies and
promotes straightening. We demonstrate the power of this objective in training
deep feedforward neural networks on smoothly-rendered synthetic image sequences
that mimic commonly-occurring properties of natural videos. The learned model
contains neural embeddings that are predictive, but also factorize the
geometric, photometric, and semantic attributes of objects. The representations
also prove more robust to noise and adversarial attacks compared to previous
SSL methods that optimize for invariance to random augmentations. Moreover,
these beneficial properties can be transferred to other training procedures by
using the straightening objective as a regularizer, suggesting a broader
utility for straightening as a principle for robust unsupervised learning.


http://arxiv.org/abs/2411.01252
Quantum Token Obfuscation via Superposition. (78%)
S. M. Yousuf Iqbal Tomal; Abdullah Al Shafin
  As quantum computing advances, traditional cryptographic security measures,
including token obfuscation, face increasing vulnerability to quantum attacks.
This paper presents a quantum-enhanced approach to token obfuscation that
leverages quantum superposition and multi-basis verification to establish a
resilient defense against such threats. In this method, tokens are encoded in
superposition states, ensuring probabilistic concealment until measured,
significantly increasing the computational difficulty of unauthorized
reconstruction. A quantum decay protocol and a refresh mechanism are introduced
to dynamically manage the token lifecycle, mitigating risks associated with
prolonged token exposure and replay attacks. Experimental validation confirms
the robustness of this approach, demonstrating an entropy quality score of
0.9996, a 0% success rate across five adversarial attack models, and an
estimated false positive rate of 67%, highlighting the system's strict security
constraints. These results validate the feasibility of quantum-based token
obfuscation as a post-quantum cryptographic framework. By integrating dynamic
token evolution, entropy-driven state transformations, and multi-basis
verification, this approach provides a foundation for future quantum-secure
authentication and access control mechanisms.


http://arxiv.org/abs/2411.01222
$B^4$: A Black-Box Scrubbing Attack on LLM Watermarks. (75%)
Baizhou Huang; Xiao Pu; Xiaojun Wan
  Watermarking has emerged as a prominent technique for LLM-generated content
detection by embedding imperceptible patterns. Despite supreme performance, its
robustness against adversarial attacks remains underexplored. Previous work
typically considers a grey-box attack setting, where the specific type of
watermark is already known. Some even necessitates knowledge about
hyperparameters of the watermarking method. Such prerequisites are unattainable
in real-world scenarios. Targeting at a more realistic black-box threat model
with fewer assumptions, we here propose $\mathcal{B}^4$, a black-box scrubbing
attack on watermarks. Specifically, we formulate the watermark scrubbing attack
as a constrained optimization problem by capturing its objectives with two
distributions, a Watermark Distribution and a Fidelity Distribution. This
optimization problem can be approximately solved using two proxy distributions.
Experimental results across 12 different settings demonstrate the superior
performance of $\mathcal{B}^4$ compared with other baselines.


http://arxiv.org/abs/2411.03343
What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks. (1%)
Nathalie Maria Kirch; Severin Field; Stephen Casper
  While `jailbreaks' have been central to research on the safety and
reliability of LLMs (large language models), the underlying mechanisms behind
these attacks are not well understood. Some prior works have used linear
methods to analyze jailbreak prompts or model refusal. Here, however, we
compare linear and nonlinear methods to study the features in prompts that
contribute to successful jailbreaks. We do this by probing for jailbreak
success based only on the portions of the latent representations corresponding
to prompt tokens. First, we introduce a dataset of 10,800 jailbreak attempts
from 35 attack methods. We then show that different jailbreaking methods work
via different nonlinear features in prompts. Specifically, we find that while
probes can distinguish between successful and unsuccessful jailbreaking prompts
with a high degree of accuracy, they often transfer poorly to held-out attack
methods. We also show that nonlinear probes can be used to mechanistically
jailbreak the LLM by guiding the design of adversarial latent perturbations.
These mechanistic jailbreaks are able to jailbreak Gemma-7B-IT more reliably
than 34 of the 35 techniques that it was trained on. Ultimately, our results
suggest that jailbreaks cannot be thoroughly understood in terms of universal
or linear prompt features alone.


http://arxiv.org/abs/2411.00898
Replace-then-Perturb: Targeted Adversarial Attacks With Visual Reasoning for Vision-Language Models. (99%)
Jonggyu Jang; Hyeonsu Lyu; Jungyeon Koh; Hyun Jong Yang
  The conventional targeted adversarial attacks add a small perturbation to an
image to make neural network models estimate the image as a predefined target
class, even if it is not the correct target class. Recently, for
visual-language models (VLMs), the focus of targeted adversarial attacks is to
generate a perturbation that makes VLMs answer intended target text outputs.
For example, they aim to make a small perturbation on an image to make VLMs'
answers change from "there is an apple" to "there is a baseball." However,
answering just intended text outputs is insufficient for tricky questions like
"if there is a baseball, tell me what is below it." This is because the target
of the adversarial attacks does not consider the overall integrity of the
original image, thereby leading to a lack of visual reasoning. In this work, we
focus on generating targeted adversarial examples with visual reasoning against
VLMs. To this end, we propose 1) a novel adversarial attack procedure --
namely, Replace-then-Perturb and 2) a contrastive learning-based adversarial
loss -- namely, Contrastive-Adv. In Replace-then-Perturb, we first leverage a
text-guided segmentation model to find the target object in the image. Then, we
get rid of the target object and inpaint the empty space with the desired
prompt. By doing this, we can generate a target image corresponding to the
desired prompt, while maintaining the overall integrity of the original image.
Furthermore, in Contrastive-Adv, we design a novel loss function to obtain
better adversarial examples. Our extensive benchmark results demonstrate that
Replace-then-Perturb and Contrastive-Adv outperform the baseline adversarial
attack algorithms. We note that the source code to reproduce the results will
be available.


http://arxiv.org/abs/2411.00459
Defense Against Prompt Injection Attack by Leveraging Attack Techniques. (81%)
Yulin Chen; Haoran Li; Zihao Zheng; Yangqiu Song; Dekai Wu; Bryan Hooi
  With the advancement of technology, large language models (LLMs) have
achieved remarkable performance across various natural language processing
(NLP) tasks, powering LLM-integrated applications like Microsoft Copilot.
However, as LLMs continue to evolve, new vulnerabilities, especially prompt
injection attacks arise. These attacks trick LLMs into deviating from the
original input instructions and executing the attacker's instructions injected
in data content, such as retrieved results. Recent attack methods leverage
LLMs' instruction-following abilities and their inabilities to distinguish
instructions injected in the data content, and achieve a high attack success
rate (ASR). When comparing the attack and defense methods, we interestingly
find that they share similar design goals, of inducing the model to ignore
unwanted instructions and instead to execute wanted instructions. Therefore, we
raise an intuitive question: Could these attack techniques be utilized for
defensive purposes? In this paper, we invert the intention of prompt injection
methods to develop novel defense methods based on previous training-free attack
methods, by repeating the attack process but with the original input
instruction rather than the injected instruction. Our comprehensive experiments
demonstrate that our defense techniques outperform existing training-free
defense approaches, achieving state-of-the-art results.


http://arxiv.org/abs/2411.00899
Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing. (68%)
Weizhi Gao; Zhichao Hou; Han Xu; Xiaorui Liu
  Implicit models such as Deep Equilibrium Models (DEQs) have emerged as
promising alternative approaches for building deep neural networks. Their
certified robustness has gained increasing research attention due to security
concerns. Existing certified defenses for DEQs employing deterministic
certification methods such as interval bound propagation and Lipschitz-bounds
can not certify on large-scale datasets. Besides, they are also restricted to
specific forms of DEQs. In this paper, we provide the first randomized
smoothing certified defense for DEQs to solve these limitations. Our study
reveals that simply applying randomized smoothing to certify DEQs provides
certified robustness generalized to large-scale datasets but incurs extremely
expensive computation costs. To reduce computational redundancy, we propose a
novel Serialized Randomized Smoothing (SRS) approach that leverages historical
information. Additionally, we derive a new certified radius estimation for SRS
to theoretically ensure the correctness of our algorithm. Extensive experiments
and ablation studies on image recognition demonstrate that our algorithm can
significantly accelerate the certification of DEQs by up to 7x almost without
sacrificing the certified accuracy. Our code is available at
https://github.com/WeizhiGao/Serialized-Randomized-Smoothing.


http://arxiv.org/abs/2411.00348
Attention Tracker: Detecting Prompt Injection Attacks in LLMs. (26%)
Kuo-Han Hung; Ching-Yun Ko; Ambrish Rawat; I-Hsin Chung; Winston H. Hsu; Pin-Yu Chen
  Large Language Models (LLMs) have revolutionized various domains but remain
vulnerable to prompt injection attacks, where malicious inputs manipulate the
model into ignoring original instructions and executing designated action. In
this paper, we investigate the underlying mechanisms of these attacks by
analyzing the attention patterns within LLMs. We introduce the concept of the
distraction effect, where specific attention heads, termed important heads,
shift focus from the original instruction to the injected instruction. Building
on this discovery, we propose Attention Tracker, a training-free detection
method that tracks attention patterns on instruction to detect prompt injection
attacks without the need for additional LLM inference. Our method generalizes
effectively across diverse models, datasets, and attack types, showing an AUROC
improvement of up to 10.0% over existing methods, and performs well even on
small LLMs. We demonstrate the robustness of our approach through extensive
evaluations and provide insights into safeguarding LLM-integrated systems from
prompt injection vulnerabilities.


http://arxiv.org/abs/2411.00519
Outlier-Oriented Poisoning Attack: A Grey-box Approach to Disturb Decision Boundaries by Perturbing Outliers in Multiclass Learning. (13%)
Anum Paracha; Junaid Arshad; Mohamed Ben Farah; Khalid Ismail
  Poisoning attacks are a primary threat to machine learning models, aiming to
compromise their performance and reliability by manipulating training datasets.
This paper introduces a novel attack - Outlier-Oriented Poisoning (OOP) attack,
which manipulates labels of most distanced samples from the decision
boundaries. The paper also investigates the adverse impact of such attacks on
different machine learning algorithms within a multiclass classification
scenario, analyzing their variance and correlation between different poisoning
levels and performance degradation. To ascertain the severity of the OOP attack
for different degrees (5% - 25%) of poisoning, we analyzed variance, accuracy,
precision, recall, f1-score, and false positive rate for chosen ML
models.Benchmarking our OOP attack, we have analyzed key characteristics of
multiclass machine learning algorithms and their sensitivity to poisoning
attacks. Our experimentation used three publicly available datasets: IRIS,
MNIST, and ISIC. Our analysis shows that KNN and GNB are the most affected
algorithms with a decrease in accuracy of 22.81% and 56.07% while increasing
false positive rate to 17.14% and 40.45% for IRIS dataset with 15% poisoning.
Further, Decision Trees and Random Forest are the most resilient algorithms
with the least accuracy disruption of 12.28% and 17.52% with 15% poisoning of
the IRIS dataset. We have also analyzed the correlation between number of
dataset classes and the performance degradation of models. Our analysis
highlighted that number of classes are inversely proportional to the
performance degradation, specifically the decrease in accuracy of the models,
which is normalized with increasing number of classes. Further, our analysis
identified that imbalanced dataset distribution can aggravate the impact of
poisoning for machine learning models


http://arxiv.org/abs/2411.01040
Identify Backdoored Model in Federated Learning via Individual Unlearning. (5%)
Jiahao Xu; Zikai Zhang; Rui Hu
  Backdoor attacks present a significant threat to the robustness of Federated
Learning (FL) due to their stealth and effectiveness. They maintain both the
main task of the FL system and the backdoor task simultaneously, causing
malicious models to appear statistically similar to benign ones, which enables
them to evade detection by existing defense methods. We find that malicious
parameters in backdoored models are inactive on the main task, resulting in a
significantly large empirical loss during the machine unlearning process on
clean inputs. Inspired by this, we propose MASA, a method that utilizes
individual unlearning on local models to identify malicious models in FL. To
improve the performance of MASA in challenging non-independent and identically
distributed (non-IID) settings, we design pre-unlearning model fusion that
integrates local models with knowledge learned from other datasets to mitigate
the divergence in their unlearning behaviors caused by the non-IID data
distributions of clients. Additionally, we propose a new anomaly detection
metric with minimal hyperparameters to filter out malicious models efficiently.
Extensive experiments on IID and non-IID datasets across six different attacks
validate the effectiveness of MASA. To the best of our knowledge, this is the
first work to leverage machine unlearning to identify malicious models in FL.
Code is available at \url{https://github.com/JiiahaoXU/MASA}.


http://arxiv.org/abs/2411.01084
Plentiful Jailbreaks with String Compositions. (2%)
Brian R. Y. Huang
  Large language models (LLMs) remain vulnerable to a slew of adversarial
attacks and jailbreaking methods. One common approach employed by white-hat
attackers, or \textit{red-teamers}, is to process model inputs and outputs
using string-level obfuscations, which can include leetspeak, rotary ciphers,
Base64, ASCII, and more. Our work extends these encoding-based attacks by
unifying them in a framework of invertible string transformations. With
invertibility, we can devise arbitrary \textit{string compositions}, defined as
sequences of transformations, that we can encode and decode end-to-end
programmatically. We devise a automated best-of-n attack that samples from a
combinatorially large number of string compositions. Our jailbreaks obtain
competitive attack success rates on several leading frontier models when
evaluated on HarmBench, highlighting that encoding-based attacks remain a
persistent vulnerability even in advanced LLMs.


http://arxiv.org/abs/2411.00465
Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions. (2%)
Rui Yang; Jie Wang; Guoping Wu; Bin Li
  Real-world offline datasets are often subject to data corruptions (such as
noise or adversarial attacks) due to sensor failures or malicious attacks.
Despite advances in robust offline reinforcement learning (RL), existing
methods struggle to learn robust agents under high uncertainty caused by the
diverse corrupted data (i.e., corrupted states, actions, rewards, and
dynamics), leading to performance degradation in clean environments. To tackle
this problem, we propose a novel robust variational Bayesian inference for
offline RL (TRACER). It introduces Bayesian inference for the first time to
capture the uncertainty via offline data for robustness against all types of
data corruptions. Specifically, TRACER first models all corruptions as the
uncertainty in the action-value function. Then, to capture such uncertainty, it
uses all offline data as the observations to approximate the posterior
distribution of the action-value function under a Bayesian inference framework.
An appealing feature of TRACER is that it can distinguish corrupted data from
clean data using an entropy-based uncertainty measure, since corrupted data
often induces higher uncertainty and entropy. Based on the aforementioned
measure, TRACER can regulate the loss associated with corrupted data to reduce
its influence, thereby enhancing robustness and performance in clean
environments. Experiments demonstrate that TRACER significantly outperforms
several state-of-the-art approaches across both individual and simultaneous
data corruptions.


http://arxiv.org/abs/2411.00349
Examining Attacks on Consensus and Incentive Systems in Proof-of-Work Blockchains: A Systematic Literature Review. (1%)
Dinitha Wijewardhana; Sugandima Vidanagamachchi; Nalin Arachchilage
  Cryptocurrencies have gained popularity due to their transparency, security,
and accessibility compared to traditional financial systems, with Bitcoin,
introduced in 2009, leading the market. Bitcoin's security relies on blockchain
technology - a decentralized ledger consisting of a consensus and an incentive
mechanism. The consensus mechanism, Proof of Work (PoW), requires miners to
solve difficult cryptographic puzzles to add new blocks, while the incentive
mechanism rewards them with newly minted bitcoins. However, as Bitcoin's
acceptance grows, it faces increasing threats from attacks targeting these
mechanisms, such as selfish mining, double-spending, and block withholding.
These attacks compromise security, efficiency, and reward distribution. Recent
research shows that these attacks can be combined with each other or with
either malicious strategies, such as network-layer attacks, or non-malicious
strategies, like honest mining. These combinations lead to more sophisticated
attacks, increasing the attacker's success rates and profitability. Therefore,
understanding and evaluating these attacks is essential for developing
effective countermeasures and ensuring long-term security. This paper begins by
examining individual attacks executed in isolation and their profitability. It
then explores how combining these attacks with each other or with other
malicious and non-malicious strategies can enhance their overall effectiveness
and profitability. The analysis further explores how the deployment of attacks
such as selfish mining and block withholding by multiple competing mining pools
against each other impacts their economic returns. Lastly, a set of design
guidelines is provided, outlining areas future work should focus on to prevent
or mitigate the identified threats.


http://arxiv.org/abs/2411.00715
B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable. (1%)
Shreyash Arya; Sukrut Rao; Moritz Böhle; Bernt Schiele
  B-cos Networks have been shown to be effective for obtaining highly human
interpretable explanations of model decisions by architecturally enforcing
stronger alignment between inputs and weight. B-cos variants of convolutional
networks (CNNs) and vision transformers (ViTs), which primarily replace linear
layers with B-cos transformations, perform competitively to their respective
standard variants while also yielding explanations that are faithful by design.
However, it has so far been necessary to train these models from scratch, which
is increasingly infeasible in the era of large, pre-trained foundation models.
In this work, inspired by the architectural similarities in standard DNNs and
B-cos networks, we propose 'B-cosification', a novel approach to transform
existing pre-trained models to become inherently interpretable. We perform a
thorough study of design choices to perform this conversion, both for
convolutional neural networks and vision transformers. We find that
B-cosification can yield models that are on par with B-cos models trained from
scratch in terms of interpretability, while often outperforming them in terms
of classification performance at a fraction of the training cost. Subsequently,
we apply B-cosification to a pretrained CLIP model, and show that, even with
limited data and compute cost, we obtain a B-cosified version that is highly
interpretable and competitive on zero shot performance across a variety of
datasets. We release our code and pre-trained model weights at
https://github.com/shrebox/B-cosification.


http://arxiv.org/abs/2411.00403
Towards Building Secure UAV Navigation with FHE-aware Knowledge Distillation. (1%)
Arjun Ramesh Kaushik; Charanjit Jutla; Nalini Ratha
  In safeguarding mission-critical systems, such as Unmanned Aerial Vehicles
(UAVs), preserving the privacy of path trajectories during navigation is
paramount. While the combination of Reinforcement Learning (RL) and Fully
Homomorphic Encryption (FHE) holds promise, the computational overhead of FHE
presents a significant challenge. This paper proposes an innovative approach
that leverages Knowledge Distillation to enhance the practicality of secure UAV
navigation. By integrating RL and FHE, our framework addresses vulnerabilities
to adversarial attacks while enabling real-time processing of encrypted UAV
camera feeds, ensuring data security. To mitigate FHE's latency, Knowledge
Distillation is employed to compress the network, resulting in an impressive
18x speedup without compromising performance, as evidenced by an R-squared
score of 0.9499 compared to the original model's score of 0.9631. Our
methodology underscores the feasibility of processing encrypted data for UAV
navigation tasks, emphasizing security alongside performance efficiency and
timely processing. These findings pave the way for deploying autonomous UAVs in
sensitive environments, bolstering their resilience against potential security
threats.


http://arxiv.org/abs/2410.23870
Noise as a Double-Edged Sword: Reinforcement Learning Exploits Randomized Defenses in Neural Networks. (99%)
Steve Bakos; Pooria Madani; Heidar Davoudi
  This study investigates a counterintuitive phenomenon in adversarial machine
learning: the potential for noise-based defenses to inadvertently aid evasion
attacks in certain scenarios. While randomness is often employed as a defensive
strategy against adversarial examples, our research reveals that this approach
can sometimes backfire, particularly when facing adaptive attackers using
reinforcement learning (RL). Our findings show that in specific cases,
especially with visually noisy classes, the introduction of noise in the
classifier's confidence values can be exploited by the RL attacker, leading to
a significant increase in evasion success rates. In some instances, the
noise-based defense scenario outperformed other strategies by up to 20\% on a
subset of classes. However, this effect was not consistent across all
classifiers tested, highlighting the complexity of the interaction between
noise-based defenses and different models. These results suggest that in some
cases, noise-based defenses can inadvertently create an adversarial training
loop beneficial to the RL attacker. Our study emphasizes the need for a more
nuanced approach to defensive strategies in adversarial machine learning,
particularly in safety-critical applications. It challenges the assumption that
randomness universally enhances defense against evasion attacks and highlights
the importance of considering adaptive, RL-based attackers when designing
robust defense mechanisms.


http://arxiv.org/abs/2411.00222
Protecting Feed-Forward Networks from Adversarial Attacks Using Predictive Coding. (99%)
Ehsan Ganjidoost; Jeff Orchard
  An adversarial example is a modified input image designed to cause a Machine
Learning (ML) model to make a mistake; these perturbations are often invisible
or subtle to human observers and highlight vulnerabilities in a model's ability
to generalize from its training data. Several adversarial attacks can create
such examples, each with a different perspective, effectiveness, and
perceptibility of changes. Conversely, defending against such adversarial
attacks improves the robustness of ML models in image processing and other
domains of deep learning. Most defence mechanisms require either a level of
model awareness, changes to the model, or access to a comprehensive set of
adversarial examples during training, which is impractical. Another option is
to use an auxiliary model in a preprocessing manner without changing the
primary model. This study presents a practical and effective solution -- using
predictive coding networks (PCnets) as an auxiliary step for adversarial
defence. By seamlessly integrating PCnets into feed-forward networks as a
preprocessing step, we substantially bolster resilience to adversarial
perturbations. Our experiments on MNIST and CIFAR10 demonstrate the remarkable
effectiveness of PCnets in mitigating adversarial examples with about 82% and
65% improvements in robustness, respectively. The PCnet, trained on a small
subset of the dataset, leverages its generative nature to effectively counter
adversarial efforts, reverting perturbed images closer to their original forms.
This innovative approach holds promise for enhancing the security and
reliability of neural network classifiers in the face of the escalating threat
of adversarial attacks.


http://arxiv.org/abs/2410.23677
Wide Two-Layer Networks can Learn from Adversarial Perturbations. (98%)
Soichiro Kumano; Hiroshi Kera; Toshihiko Yamasaki
  Adversarial examples have raised several open questions, such as why they can
deceive classifiers and transfer between different models. A prevailing
hypothesis to explain these phenomena suggests that adversarial perturbations
appear as random noise but contain class-specific features. This hypothesis is
supported by the success of perturbation learning, where classifiers trained
solely on adversarial examples and the corresponding incorrect labels
generalize well to correctly labeled test data. Although this hypothesis and
perturbation learning are effective in explaining intriguing properties of
adversarial examples, their solid theoretical foundation is limited. In this
study, we theoretically explain the counterintuitive success of perturbation
learning. We assume wide two-layer networks and the results hold for any data
distribution. We prove that adversarial perturbations contain sufficient
class-specific features for networks to generalize from them. Moreover, the
predictions of classifiers trained on mislabeled adversarial examples coincide
with those of classifiers trained on correctly labeled clean samples. The code
is available at https://github.com/s-kumano/perturbation-learning.


http://arxiv.org/abs/2410.24006
DiffPAD: Denoising Diffusion-based Adversarial Patch Decontamination. (93%)
Jia Fu; Xiao Zhang; Sepideh Pashami; Fatemeh Rahimian; Anders Holst
  In the ever-evolving adversarial machine learning landscape, developing
effective defenses against patch attacks has become a critical challenge,
necessitating reliable solutions to safeguard real-world AI systems. Although
diffusion models have shown remarkable capacity in image synthesis and have
been recently utilized to counter $\ell_p$-norm bounded attacks, their
potential in mitigating localized patch attacks remains largely underexplored.
In this work, we propose DiffPAD, a novel framework that harnesses the power of
diffusion models for adversarial patch decontamination. DiffPAD first performs
super-resolution restoration on downsampled input images, then adopts
binarization, dynamic thresholding scheme and sliding window for effective
localization of adversarial patches. Such a design is inspired by the
theoretically derived correlation between patch size and diffusion restoration
error that is generalized across diverse patch attack scenarios. Finally,
DiffPAD applies inpainting techniques to the original input images with the
estimated patch region being masked. By integrating closed-form solutions for
super-resolution restoration and image inpainting into the conditional reverse
sampling process of a pre-trained diffusion model, DiffPAD obviates the need
for text guidance or fine-tuning. Through comprehensive experiments, we
demonstrate that DiffPAD not only achieves state-of-the-art adversarial
robustness against patch attacks but also excels in recovering naturalistic
images without patch remnants. The source code is available at
https://github.com/JasonFu1998/DiffPAD.


http://arxiv.org/abs/2411.00121
I Can Hear You: Selective Robust Training for Deepfake Audio Detection. (86%)
Zirui Zhang; Wei Hao; Aroon Sankoh; William Lin; Emanuel Mendiola-Ortiz; Junfeng Yang; Chengzhi Mao
  Recent advances in AI-generated voices have intensified the challenge of
detecting deepfake audio, posing risks for scams and the spread of
disinformation. To tackle this issue, we establish the largest public voice
dataset to date, named DeepFakeVox-HQ, comprising 1.3 million samples,
including 270,000 high-quality deepfake samples from 14 diverse sources.
Despite previously reported high accuracy, existing deepfake voice detectors
struggle with our diversely collected dataset, and their detection success
rates drop even further under realistic corruptions and adversarial attacks. We
conduct a holistic investigation into factors that enhance model robustness and
show that incorporating a diversified set of voice augmentations is beneficial.
Moreover, we find that the best detection models often rely on high-frequency
features, which are imperceptible to humans and can be easily manipulated by an
attacker. To address this, we propose the F-SAT: Frequency-Selective
Adversarial Training method focusing on high-frequency components. Empirical
results demonstrate that using our training dataset boosts baseline model
performance (without robust training) by 33%, and our robust training further
improves accuracy by 7.7% on clean samples and by 29.3% on corrupted and
attacked samples, over the state-of-the-art RawNet3 model.


http://arxiv.org/abs/2410.23678
Pseudo-Conversation Injection for LLM Goal Hijacking. (75%)
Zheng Chen; Buhui Yao
  Goal hijacking is a type of adversarial attack on Large Language Models
(LLMs) where the objective is to manipulate the model into producing a
specific, predetermined output, regardless of the user's original input. In
goal hijacking, an attacker typically appends a carefully crafted malicious
suffix to the user's prompt, which coerces the model into ignoring the user's
original input and generating the target response. In this paper, we introduce
a novel goal hijacking attack method called Pseudo-Conversation Injection,
which leverages the weaknesses of LLMs in role identification within
conversation contexts. Specifically, we construct the suffix by fabricating
responses from the LLM to the user's initial prompt, followed by a prompt for a
malicious new task. This leads the model to perceive the initial prompt and
fabricated response as a completed conversation, thereby executing the new,
falsified prompt. Following this approach, we propose three Pseudo-Conversation
construction strategies: Targeted Pseudo-Conversation, Universal
Pseudo-Conversation, and Robust Pseudo-Conversation. These strategies are
designed to achieve effective goal hijacking across various scenarios. Our
experiments, conducted on two mainstream LLM platforms including ChatGPT and
Qwen, demonstrate that our proposed method significantly outperforms existing
approaches in terms of attack effectiveness.


http://arxiv.org/abs/2410.24214
ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs. (67%)
Yuchen Yang; Shubham Ugare; Yifan Zhao; Gagandeep Singh; Sasa Misailovic
  Mixed precision quantization has become an important technique for enabling
the execution of deep neural networks (DNNs) on limited resource computing
platforms. Traditional quantization methods have primarily concentrated on
maintaining neural network accuracy, either ignoring the impact of quantization
on the robustness of the network, or using only empirical techniques for
improving robustness. In contrast, techniques for robustness certification,
which can provide strong guarantees about the robustness of DNNs have not been
used during quantization due to their high computation cost.
  This paper introduces ARQ, an innovative mixed-precision quantization method
that not only preserves the clean accuracy of the smoothed classifiers but also
maintains their certified robustness. ARQ uses reinforcement learning to find
accurate and robust DNN quantization, while efficiently leveraging randomized
smoothing, a popular class of statistical DNN verification algorithms, to guide
the search process.
  We compare ARQ with multiple state-of-the-art quantization techniques on
several DNN architectures commonly used in quantization studies: ResNet-20 on
CIFAR-10, ResNet-50 on ImageNet, and MobileNetV2 on ImageNet. We demonstrate
that ARQ consistently performs better than these baselines across all the
benchmarks and the input perturbation levels. In many cases, the performance of
ARQ quantized networks can reach that of the original DNN with floating-point
weights, but with only 1.5% instructions.


http://arxiv.org/abs/2411.00192
Optical Lens Attack on Monocular Depth Estimation for Autonomous Driving. (5%)
Ce Michigan State University Zhou; Qiben Michigan State University Yan; Daniel Michigan State University Kent; Guangjing University of South Florida Wang; Weikang Michigan State University Ding; Ziqi Peking University Zhang; Hayder Michigan State University Radha
  Monocular Depth Estimation (MDE) is a pivotal component of vision-based
Autonomous Driving (AD) systems, enabling vehicles to estimate the depth of
surrounding objects using a single camera image. This estimation guides
essential driving decisions, such as braking before an obstacle or changing
lanes to avoid collisions. In this paper, we explore vulnerabilities of MDE
algorithms in AD systems, presenting LensAttack, a novel physical attack that
strategically places optical lenses on the camera of an autonomous vehicle to
manipulate the perceived object depths. LensAttack encompasses two attack
formats: concave lens attack and convex lens attack, each utilizing different
optical lenses to induce false depth perception. We first develop a
mathematical model that outlines the parameters of the attack, followed by
simulations and real-world evaluations to assess its efficacy on
state-of-the-art MDE models. Additionally, we adopt an attack optimization
method to further enhance the attack success rate by optimizing the attack
focal length. To better evaluate the implications of LensAttack on AD, we
conduct comprehensive end-to-end system simulations using the CARLA platform.
The results reveal that LensAttack can significantly disrupt the depth
estimation processes in AD systems, posing a serious threat to their
reliability and safety. Finally, we discuss some potential defense methods to
mitigate the effects of the proposed attack.


http://arxiv.org/abs/2410.23687
Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey. (2%)
Chiyu Zhang; Xiaogang Xu; Jiafei Wu; Zhe Liu; Lu Zhou
  Adversarial attacks, which manipulate input data to undermine model
availability and integrity, pose significant security threats during machine
learning inference. With the advent of Large Vision-Language Models (LVLMs),
new attack vectors, such as cognitive bias, prompt injection, and jailbreak
techniques, have emerged. Understanding these attacks is crucial for developing
more robust systems and demystifying the inner workings of neural networks.
However, existing reviews often focus on attack classifications and lack
comprehensive, in-depth analysis. The research community currently needs: 1)
unified insights into adversariality, transferability, and generalization; 2)
detailed evaluations of existing methods; 3) motivation-driven attack
categorizations; and 4) an integrated perspective on both traditional and LVLM
attacks. This article addresses these gaps by offering a thorough summary of
traditional and LVLM adversarial attacks, emphasizing their connections and
distinctions, and providing actionable insights for future research.


http://arxiv.org/abs/2410.23142
FAIR-TAT: Improving Model Fairness Using Targeted Adversarial Training. (99%)
Tejaswini Medi; Steffen Jung; Margret Keuper
  Deep neural networks are susceptible to adversarial attacks and common
corruptions, which undermine their robustness. In order to enhance model
resilience against such challenges, Adversarial Training (AT) has emerged as a
prominent solution. Nevertheless, adversarial robustness is often attained at
the expense of model fairness during AT, i.e., disparity in class-wise
robustness of the model. While distinctive classes become more robust towards
such adversaries, hard to detect classes suffer. Recently, research has focused
on improving model fairness specifically for perturbed images, overlooking the
accuracy of the most likely non-perturbed data. Additionally, despite their
robustness against the adversaries encountered during model training,
state-of-the-art adversarial trained models have difficulty maintaining
robustness and fairness when confronted with diverse adversarial threats or
common corruptions. In this work, we address the above concerns by introducing
a novel approach called Fair Targeted Adversarial Training (FAIR-TAT). We show
that using targeted adversarial attacks for adversarial training (instead of
untargeted attacks) can allow for more favorable trade-offs with respect to
adversarial fairness. Empirical results validate the efficacy of our approach.


http://arxiv.org/abs/2410.23483
Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System. (99%)
Julian Collado; Kevin Stangl
  Recent approaches in machine learning often solve a task using a composition
of multiple models or agentic architectures. When targeting a composed system
with adversarial attacks, it might not be computationally or informationally
feasible to train an end-to-end proxy model or a proxy model for every
component of the system. We introduce a method to craft an adversarial attack
against the overall multi-model system when we only have a proxy model for the
final black-box model, and when the transformation applied by the initial
models can make the adversarial perturbations ineffective. Current methods
handle this by applying many copies of the first model/transformation to an
input and then re-use a standard adversarial attack by averaging gradients, or
learning a proxy model for both stages. To our knowledge, this is the first
attack specifically designed for this threat model and our method has a
substantially higher attack success rate (80% vs 25%) and contains 9.4% smaller
perturbations (MSE) compared to prior state-of-the-art methods. Our experiments
focus on a supervised image pipeline, but we are confident the attack will
generalize to other multi-model settings [e.g. a mix of open/closed source
foundation models], or agentic systems


http://arxiv.org/abs/2410.23091
CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense. (99%)
Mingkun Zhang; Keping Bi; Wei Chen; Quanrun Chen; Jiafeng Guo; Xueqi Cheng
  Despite ongoing efforts to defend neural classifiers from adversarial
attacks, they remain vulnerable, especially to unseen attacks. In contrast,
humans are difficult to be cheated by subtle manipulations, since we make
judgments only based on essential factors. Inspired by this observation, we
attempt to model label generation with essential label-causative factors and
incorporate label-non-causative factors to assist data generation. For an
adversarial example, we aim to discriminate the perturbations as non-causative
factors and make predictions only based on the label-causative factors.
Concretely, we propose a casual diffusion model (CausalDiff) that adapts
diffusion models for conditional data generation and disentangles the two types
of casual factors by learning towards a novel casual information bottleneck
objective. Empirically, CausalDiff has significantly outperformed
state-of-the-art defense methods on various unseen attacks, achieving an
average robustness of 86.39% (+4.01%) on CIFAR-10, 56.25% (+3.13%) on
CIFAR-100, and 82.62% (+4.93%) on GTSRB (German Traffic Sign Recognition
Benchmark). The code is available at
\href{https://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff}{https://github.com/CAS-AISafetyBasicResearchGroup/CausalDiff}


http://arxiv.org/abs/2410.22725
One Prompt to Verify Your Models: Black-Box Text-to-Image Models Verification via Non-Transferable Adversarial Attacks. (98%)
Ji Guo; Wenbo Jiang; Rui Zhang; Guoming Lu; Hongwei Li
  Recently, the success of Text-to-Image (T2I) models has led to the rise of
numerous third-party platforms, which claim to provide cheaper API services and
more flexibility in model options. However, this also raises a new security
concern: Are these third-party services truly offering the models they claim?
To address this problem, we propose the first T2I model verification method
named Text-to-Image Model Verification via Non-Transferable Adversarial Attacks
(TVN). The non-transferability of adversarial examples means that these
examples are only effective on a target model and ineffective on other models,
thereby allowing for the verification of the target model. TVN utilizes the
Non-dominated Sorting Genetic Algorithm II (NSGA-II) to optimize the cosine
similarity of a prompt's text encoding, generating non-transferable adversarial
prompts. By calculating the CLIP-text scores between the non-transferable
adversarial prompts without perturbations and the images, we can verify if the
model matches the claimed target model, based on a 3-sigma threshold. The
experiments showed that TVN performed well in both closed-set and open-set
scenarios, achieving a verification accuracy of over 90\%. Moreover, the
adversarial prompts generated by TVN significantly reduced the CLIP-text scores
of the target model, while having little effect on other models.


http://arxiv.org/abs/2410.22888
Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector. (87%)
Youcheng Huang; Fengbin Zhu; Jingkun Tang; Pan Zhou; Wenqiang Lei; Jiancheng Lv; Tat-Seng Chua
  Visual Language Models (VLMs) are vulnerable to adversarial attacks,
especially those from adversarial images, which is however under-explored in
literature. To facilitate research on this critical safety problem, we first
construct a new laRge-scale Adervsarial images dataset with Diverse hArmful
Responses (RADAR), given that existing datasets are either small-scale or only
contain limited types of harmful responses. With the new RADAR dataset, we
further develop a novel and effective iN-time Embedding-based AdveRSarial Image
DEtection (NEARSIDE) method, which exploits a single vector that distilled from
the hidden states of VLMs, which we call the attacking direction, to achieve
the detection of adversarial images against benign ones in the input. Extensive
experiments with two victim VLMs, LLaVA and MiniGPT-4, well demonstrate the
effectiveness, efficiency, and cross-model transferrability of our proposed
method. Our code is available at https://github.com/mob-scu/RADAR-NEARSIDE


http://arxiv.org/abs/2410.22832
HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models. (82%)
Yucheng Zhang; Qinfeng Li; Tianyu Du; Xuhong Zhang; Xinkui Zhao; Zhengwen Feng; Jianwei Yin
  Retrieval-Augmented Generation (RAG) systems enhance large language models
(LLMs) by integrating external knowledge, making them adaptable and
cost-effective for various applications. However, the growing reliance on these
systems also introduces potential security risks. In this work, we reveal a
novel vulnerability, the retrieval prompt hijack attack (HijackRAG), which
enables attackers to manipulate the retrieval mechanisms of RAG systems by
injecting malicious texts into the knowledge database. When the RAG system
encounters target questions, it generates the attacker's pre-determined answers
instead of the correct ones, undermining the integrity and trustworthiness of
the system. We formalize HijackRAG as an optimization problem and propose both
black-box and white-box attack strategies tailored to different levels of the
attacker's knowledge. Extensive experiments on multiple benchmark datasets show
that HijackRAG consistently achieves high attack success rates, outperforming
existing baseline attacks. Furthermore, we demonstrate that the attack is
transferable across different retriever models, underscoring the widespread
risk it poses to RAG systems. Lastly, our exploration of various defense
mechanisms reveals that they are insufficient to counter HijackRAG, emphasizing
the urgent need for more robust security measures to protect RAG systems in
real-world deployments.


http://arxiv.org/abs/2410.22844
Understanding and Improving Adversarial Collaborative Filtering for Robust Recommendation. (67%)
Kaike Zhang; Qi Cao; Yunfan Wu; Fei Sun; Huawei Shen; Xueqi Cheng
  Adversarial Collaborative Filtering (ACF), which typically applies
adversarial perturbations at user and item embeddings through adversarial
training, is widely recognized as an effective strategy for enhancing the
robustness of Collaborative Filtering (CF) recommender systems against
poisoning attacks. Besides, numerous studies have empirically shown that ACF
can also improve recommendation performance compared to traditional CF. Despite
these empirical successes, the theoretical understanding of ACF's effectiveness
in terms of both performance and robustness remains unclear. To bridge this
gap, in this paper, we first theoretically show that ACF can achieve a lower
recommendation error compared to traditional CF with the same training epochs
in both clean and poisoned data contexts. Furthermore, by establishing bounds
for reductions in recommendation error during ACF's optimization process, we
find that applying personalized magnitudes of perturbation for different users
based on their embedding scales can further improve ACF's effectiveness.
Building on these theoretical understandings, we propose Personalized Magnitude
Adversarial Collaborative Filtering (PamaCF). Extensive experiments demonstrate
that PamaCF effectively defends against various types of poisoning attacks
while significantly enhancing recommendation performance.


http://arxiv.org/abs/2410.23118
Teaching a Language Model to Distinguish Between Similar Details using a Small Adversarial Training Set. (64%)
Chris Achard
  Language models can achieve high accuracy on natural language tasks such as
NLI, but performance suffers on manually created adversarial examples. We
investigate the performance of a language model trained on the Stanford Natural
Language Inference (SNLI) corpus on a manually created adversarial test set. We
then improve the model's performance by fine tuning the model on a small,
manually created adversarial training set, designed to help the language model
to learn to differentiate between similar words and phrases in the data. We
show an increase in accuracy on the adversarial test set (+ 13%) while still
maintaining good performance on the original NLI task. We also show an increase
in accuracy from 91.2% to 92.9% on the most similar contradictions in the SNLI
test set (as judged by cosine similarity).


http://arxiv.org/abs/2410.22678
Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion. (62%)
Ji Guo; Hongwei Li; Wenbo Jiang; Guoming Lu
  Vision Transformers (ViTs) have outperformed traditional Convolutional Neural
Networks (CNN) across various computer vision tasks. However, akin to CNN, ViTs
are vulnerable to backdoor attacks, where the adversary embeds the backdoor
into the victim model, causing it to make wrong predictions about testing
samples containing a specific trigger. Existing backdoor attacks against ViTs
have the limitation of failing to strike an optimal balance between attack
stealthiness and attack effectiveness.
  In this work, we propose an Attention Gradient-based Erosion Backdoor (AGEB)
targeted at ViTs. Considering the attention mechanism of ViTs, AGEB selectively
erodes pixels in areas of maximal attention gradient, embedding a covert
backdoor trigger. Unlike previous backdoor attacks against ViTs, AGEB achieves
an optimal balance between attack stealthiness and attack effectiveness,
ensuring the trigger remains invisible to human detection while preserving the
model's accuracy on clean samples. Extensive experimental evaluations across
various ViT architectures and datasets confirm the effectiveness of AGEB,
achieving a remarkable Attack Success Rate (ASR) without diminishing Clean Data
Accuracy (CDA). Furthermore, the stealthiness of AGEB is rigorously validated,
demonstrating minimal visual discrepancies between the clean and the triggered
images.


http://arxiv.org/abs/2410.22705
Geometry Cloak: Preventing TGS-based 3D Reconstruction from Copyrighted Images. (2%)
Qi Song; Ziyuan Luo; Ka Chun Cheung; Simon See; Renjie Wan
  Single-view 3D reconstruction methods like Triplane Gaussian Splatting (TGS)
have enabled high-quality 3D model generation from just a single image input
within seconds. However, this capability raises concerns about potential
misuse, where malicious users could exploit TGS to create unauthorized 3D
models from copyrighted images. To prevent such infringement, we propose a
novel image protection approach that embeds invisible geometry perturbations,
termed "geometry cloaks", into images before supplying them to TGS. These
carefully crafted perturbations encode a customized message that is revealed
when TGS attempts 3D reconstructions of the cloaked image. Unlike conventional
adversarial attacks that simply degrade output quality, our method forces TGS
to fail the 3D reconstruction in a specific way - by generating an identifiable
customized pattern that acts as a watermark. This watermark allows copyright
holders to assert ownership over any attempted 3D reconstructions made from
their protected images. Extensive experiments have verified the effectiveness
of our geometry cloak. Our project is available at
https://qsong2001.github.io/geometry_cloak.


http://arxiv.org/abs/2410.23123
On Memorization of Large Language Models in Logical Reasoning. (1%)
Chulin Xie; Yangsibo Huang; Chiyuan Zhang; Da Yu; Xinyun Chen; Bill Yuchen Lin; Bo Li; Badih Ghazi; Ravi Kumar
  Large language models (LLMs) achieve good performance on challenging
reasoning benchmarks, yet could also make basic reasoning mistakes. This
contrasting behavior is puzzling when it comes to understanding the mechanisms
behind LLMs' reasoning capabilities. One hypothesis is that the increasingly
high and nearly saturated performance on common reasoning benchmarks could be
due to the memorization of similar problems. In this paper, we systematically
investigate this hypothesis with a quantitative measurement of memorization in
reasoning tasks, using a dynamically generated logical reasoning benchmark
based on Knights and Knaves (K&K) puzzles. We find that LLMs could interpolate
and memorize the training puzzles (achieving near-perfect accuracy) after
fine-tuning, yet they struggle with slight variations of these puzzles. On the
other hand, we show that while fine-tuning leads to heavy memorization, it also
consistently improves generalization performance. Through in-depth analyses
with perturbation tests, cross difficulty-level transferability, probing model
internals, and fine-tuning with wrong answers, we establish that LLMs develop
reasoning skills on K&K puzzles alongside memorization. Finally, our analysis
based on a per-sample memorization score sheds light on how LLMs switch between
reasoning and memorization when solving logical puzzles. Our code and data are
available at https://memkklogic.github.io.


http://arxiv.org/abs/2410.22680
Byzantine-Robust Federated Learning: An Overview With Focus on Developing Sybil-based Attacks to Backdoor Augmented Secure Aggregation Protocols. (1%)
Atharv Deshmukh
  Federated Learning (FL) paradigms enable large numbers of clients to
collaboratively train Machine Learning models on private data. However, due to
their multi-party nature, traditional FL schemes are left vulnerable to
Byzantine attacks that attempt to hurt model performance by injecting malicious
backdoors. A wide variety of prevention methods have been proposed to protect
frameworks from such attacks. This paper provides a exhaustive and updated
taxonomy of existing methods and frameworks, before zooming in and conducting
an in-depth analysis of the strengths and weaknesses of the Robustness of
Federated Learning (RoFL) protocol. From there, we propose two novel
Sybil-based attacks that take advantage of vulnerabilities in RoFL. Finally, we
conclude with comprehensive proposals for future testing, describe and detail
implementation of the proposed attacks, and offer direction for improvements in
the RoFL protocol as well as Byzantine-robust frameworks as a whole.


http://arxiv.org/abs/2410.23182
ProTransformer: Robustify Transformers via Plug-and-Play Paradigm. (1%)
Zhichao Hou; Weizhi Gao; Yuchen Shen; Feiyi Wang; Xiaorui Liu
  Transformer-based architectures have dominated various areas of machine
learning in recent years. In this paper, we introduce a novel robust attention
mechanism designed to enhance the resilience of transformer-based
architectures. Crucially, this technique can be integrated into existing
transformers as a plug-and-play layer, improving their robustness without the
need for additional training or fine-tuning. Through comprehensive experiments
and ablation studies, we demonstrate that our ProTransformer significantly
enhances the robustness of transformer models across a variety of prediction
tasks, attack mechanisms, backbone architectures, and data domains. Notably,
without further fine-tuning, the ProTransformer consistently improves the
performance of vanilla transformers by 19.5%, 28.3%, 16.1%, and 11.4% for BERT,
ALBERT, DistilBERT, and RoBERTa, respectively, under the classical TextFooler
attack. Furthermore, ProTransformer shows promising resilience in large
language models (LLMs) against prompting-based attacks, improving the
performance of T5 and LLaMA by 24.8% and 17.8%, respectively, and enhancing
Vicuna by an average of 10.4% against the Jailbreaking attack. Beyond the
language domain, ProTransformer also demonstrates outstanding robustness in
both vision and graph domains.


http://arxiv.org/abs/2410.23232
Attribute-to-Delete: Machine Unlearning via Datamodel Matching. (1%)
Kristian Georgiev; Roy Rinberg; Sung Min Park; Shivam Garg; Andrew Ilyas; Aleksander Madry; Seth Neel
  Machine unlearning -- efficiently removing the effect of a small "forget set"
of training data on a pre-trained machine learning model -- has recently
attracted significant research interest. Despite this interest, however, recent
work shows that existing machine unlearning techniques do not hold up to
thorough evaluation in non-convex settings. In this work, we introduce a new
machine unlearning technique that exhibits strong empirical performance even in
such challenging settings. Our starting point is the perspective that the goal
of unlearning is to produce a model whose outputs are statistically
indistinguishable from those of a model re-trained on all but the forget set.
This perspective naturally suggests a reduction from the unlearning problem to
that of data attribution, where the goal is to predict the effect of changing
the training set on a model's outputs. Thus motivated, we propose the following
meta-algorithm, which we call Datamodel Matching (DMM): given a trained model,
we (a) use data attribution to predict the output of the model if it were
re-trained on all but the forget set points; then (b) fine-tune the pre-trained
model to match these predicted outputs. In a simple convex setting, we show how
this approach provably outperforms a variety of iterative unlearning
algorithms. Empirically, we use a combination of existing evaluations and a new
metric based on the KL-divergence to show that even in non-convex settings, DMM
achieves strong unlearning performance relative to existing algorithms. An
added benefit of DMM is that it is a meta-algorithm, in the sense that future
advances in data attribution translate directly into better unlearning
algorithms, pointing to a clear direction for future progress in unlearning.


http://arxiv.org/abs/2410.22884
Stealing User Prompts from Mixture of Experts. (1%)
Itay Yona; Ilia Shumailov; Jamie Hayes; Nicholas Carlini
  Mixture-of-Experts (MoE) models improve the efficiency and scalability of
dense language models by routing each token to a small number of experts in
each layer. In this paper, we show how an adversary that can arrange for their
queries to appear in the same batch of examples as a victim's queries can
exploit Expert-Choice-Routing to fully disclose a victim's prompt. We
successfully demonstrate the effectiveness of this attack on a two-layer
Mixtral model, exploiting the tie-handling behavior of the torch.topk CUDA
implementation. Our results show that we can extract the entire prompt using
$O({VM}^2)$ queries (with vocabulary size $V$ and prompt length $M$) or 100
queries on average per token in the setting we consider. This is the first
attack to exploit architectural flaws for the purpose of extracting user
prompts, introducing a new class of LLM vulnerabilities.


http://arxiv.org/abs/2410.22770
InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models. (1%)
Hao Li; Xiaogeng Liu; Chaowei Xiao
  Prompt injection attacks pose a critical threat to large language models
(LLMs), enabling goal hijacking and data leakage. Prompt guard models, though
effective in defense, suffer from over-defense -- falsely flagging benign
inputs as malicious due to trigger word bias. To address this issue, we
introduce NotInject, an evaluation dataset that systematically measures
over-defense across various prompt guard models. NotInject contains 339 benign
samples enriched with trigger words common in prompt injection attacks,
enabling fine-grained evaluation. Our results show that state-of-the-art models
suffer from over-defense issues, with accuracy dropping close to random
guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt
guard model that incorporates a new training strategy, Mitigating Over-defense
for Free (MOF), which significantly reduces the bias on trigger words.
InjecGuard demonstrates state-of-the-art performance on diverse benchmarks
including NotInject, surpassing the existing best model by 30.8%, offering a
robust and open-source solution for detecting prompt injection attacks. The
code and datasets are released at https://github.com/SaFoLab-WISC/InjecGuard.


http://arxiv.org/abs/2410.21952
On the Robustness of Adversarial Training Against Uncertainty Attacks. (99%)
Emanuele Ledda; Giovanni Scodeller; Daniele Angioni; Giorgio Piras; Antonio Emanuele Cinà; Giorgio Fumera; Battista Biggio; Fabio Roli
  In learning problems, the noise inherent to the task at hand hinders the
possibility to infer without a certain degree of uncertainty. Quantifying this
uncertainty, regardless of its wide use, assumes high relevance for
security-sensitive applications. Within these scenarios, it becomes fundamental
to guarantee good (i.e., trustworthy) uncertainty measures, which downstream
modules can securely employ to drive the final decision-making process.
However, an attacker may be interested in forcing the system to produce either
(i) highly uncertain outputs jeopardizing the system's availability or (ii) low
uncertainty estimates, making the system accept uncertain samples that would
instead require a careful inspection (e.g., human intervention). Therefore, it
becomes fundamental to understand how to obtain robust uncertainty estimates
against these kinds of attacks. In this work, we reveal both empirically and
theoretically that defending against adversarial examples, i.e., carefully
perturbed samples that cause misclassification, additionally guarantees a more
secure, trustworthy uncertainty estimate under common attack scenarios without
the need for an ad-hoc defense strategy. To support our claims, we evaluate
multiple adversarial-robust models from the publicly available benchmark
RobustBench on the CIFAR-10 and ImageNet datasets.


http://arxiv.org/abs/2411.00839
CausAdv: A Causal-based Framework for Detecting Adversarial Examples. (99%)
Hichem Debbi
  Deep learning has led to tremendous success in many real-world applications
of computer vision, thanks to sophisticated architectures such as Convolutional
neural networks (CNNs). However, CNNs have been shown to be vulnerable to
crafted adversarial perturbations in inputs. These inputs appear almost
indistinguishable from natural images, yet they are incorrectly classified by
CNN architectures. This vulnerability of adversarial examples has led
researchers to focus on enhancing the robustness of deep learning models in
general, and CNNs in particular, by creating defense and detection methods to
distinguish adversarials inputs from natural ones. In this paper, we address
the adversarial robustness of CNNs through causal reasoning.
  We propose CausAdv: a causal framework for detecting adversarial examples
based on counterfactual reasoning. CausAdv learns causal and non-causal
features of every input, and quantifies the counterfactual information (CI) of
every filter of the last convolutional layer. Then we perform statistical
analysis on the filters CI of every sample, whether clan or adversarials, to
demonstrate how adversarial examples indeed exhibit different CI distributions
compared to clean samples. Our results show that causal reasoning enhances the
process of adversarials detection without the need to train a separate
detector. In addition, we illustrate the efficiency of causal explanations as a
helpful detection technique through visualizing the causal features. The
results can be reproduced using the code available in the repository:
https://github.com/HichemDebbi/CausAdv.


http://arxiv.org/abs/2410.21802
Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models. (98%)
Lu Yu; Haiyang Zhang; Changsheng Xu
  Due to the impressive zero-shot capabilities, pre-trained vision-language
models (e.g. CLIP), have attracted widespread attention and adoption across
various domains. Nonetheless, CLIP has been observed to be susceptible to
adversarial examples. Through experimental analysis, we have observed a
phenomenon wherein adversarial perturbations induce shifts in text-guided
attention. Building upon this observation, we propose a simple yet effective
strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This
framework incorporates two components: the Attention Refinement module and the
Attention-based Model Constraint module. Our goal is to maintain the
generalization of the CLIP model and enhance its adversarial robustness: The
Attention Refinement module aligns the text-guided attention obtained from the
target model via adversarial examples with the text-guided attention acquired
from the original model via clean examples. This alignment enhances the model's
robustness. Additionally, the Attention-based Model Constraint module acquires
text-guided attention from both the target and original models using clean
examples. Its objective is to maintain model performance on clean samples while
enhancing overall robustness. The experiments validate that our method yields a
9.58% enhancement in zero-shot robust accuracy over the current
state-of-the-art techniques across 16 datasets. Our code is available at
https://github.com/zhyblue424/TGA-ZSR.


http://arxiv.org/abs/2410.22663
Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers. (83%)
Lam Nguyen Tung; Steven Cho; Xiaoning Du; Neelofar Neelofar; Valerio Terragni; Stefano Ruberto; Aldeida Aleti
  Machine learning (ML) for text classification has been widely used in various
domains. These applications can significantly impact ethics, economics, and
human behavior, raising serious concerns about trusting ML decisions. Studies
indicate that conventional metrics are insufficient to build human trust in ML
models. These models often learn spurious correlations and predict based on
them. In the real world, their performance can deteriorate significantly. To
avoid this, a common practice is to test whether predictions are reasonable
based on valid patterns in the data. Along with this, a challenge known as the
trustworthiness oracle problem has been introduced. Due to the lack of
automated trustworthiness oracles, the assessment requires manual validation of
the decision process disclosed by explanation methods. However, this is
time-consuming, error-prone, and unscalable.
  We propose TOKI, the first automated trustworthiness oracle generation method
for text classifiers. TOKI automatically checks whether the words contributing
the most to a prediction are semantically related to the predicted class.
Specifically, we leverage ML explanations to extract the decision-contributing
words and measure their semantic relatedness with the class based on word
embeddings. We also introduce a novel adversarial attack method that targets
trustworthiness vulnerabilities identified by TOKI. To evaluate their alignment
with human judgement, experiments are conducted. We compare TOKI with a naive
baseline based solely on model confidence and TOKI-guided adversarial attack
method with A2T, a SOTA adversarial attack method. Results show that relying on
prediction uncertainty cannot effectively distinguish between trustworthy and
untrustworthy predictions, TOKI achieves 142% higher accuracy than the naive
baseline, and TOKI-guided attack method is more effective with fewer
perturbations than A2T.


http://arxiv.org/abs/2411.00827
IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves. (83%)
Ruofan Wang; Juncheng Li; Yixu Wang; Bo Wang; Xiaosen Wang; Yan Teng; Yingchun Wang; Xingjun Ma; Yu-Gang Jiang
  As large Vision-Language Models (VLMs) gain prominence, ensuring their safe
deployment has become critical. Recent studies have explored VLM robustness
against jailbreak attacks-techniques that exploit model vulnerabilities to
elicit harmful outputs. However, the limited availability of diverse multimodal
data has constrained current approaches to rely heavily on adversarial or
manually crafted images derived from harmful text datasets, which often lack
effectiveness and diversity across different contexts. In this paper, we
propose IDEATOR, a novel jailbreak method that autonomously generates malicious
image-text pairs for black-box jailbreak attacks. IDEATOR is grounded in the
insight that VLMs themselves could serve as powerful red team models for
generating multimodal jailbreak prompts. Specifically, IDEATOR leverages a VLM
to create targeted jailbreak texts and pairs them with jailbreak images
generated by a state-of-the-art diffusion model. Extensive experiments
demonstrate IDEATOR's high effectiveness and transferability, achieving a 94%
attack success rate (ASR) in jailbreaking MiniGPT-4 with an average of only
5.34 queries, and high ASRs of 82%, 88%, and 75% when transferred to LLaVA,
InstructBLIP, and Chameleon, respectively. Building on IDEATOR's strong
transferability and automated process, we introduce the VLBreakBench, a safety
benchmark comprising 3,654 multimodal jailbreak samples. Our benchmark results
on 11 recently released VLMs reveal significant gaps in safety alignment. For
instance, our challenge set achieves ASRs of 46.31% on GPT-4o and 19.65% on
Claude-3.5-Sonnet, underscoring the urgent need for stronger defenses.


http://arxiv.org/abs/2411.00837
Longitudinal Mammogram Exam-based Breast Cancer Diagnosis Models: Vulnerability to Adversarial Attacks. (81%)
Zhengbo Zhou; Degan Hao; Dooman Arefan; Margarita Zuley; Jules Sumkin; Shandong Wu
  In breast cancer detection and diagnosis, the longitudinal analysis of
mammogram images is crucial. Contemporary models excel in detecting temporal
imaging feature changes, thus enhancing the learning process over sequential
imaging exams. Yet, the resilience of these longitudinal models against
adversarial attacks remains underexplored. In this study, we proposed a novel
attack method that capitalizes on the feature-level relationship between two
sequential mammogram exams of a longitudinal model, guided by both
cross-entropy loss and distance metric learning, to achieve significant attack
efficacy, as implemented using attack transferring in a black-box attacking
manner. We performed experiments on a cohort of 590 breast cancer patients
(each has two sequential mammogram exams) in a case-control setting. Results
showed that our proposed method surpassed several state-of-the-art adversarial
attacks in fooling the diagnosis models to give opposite outputs. Our method
remained effective even if the model was trained with the common defending
method of adversarial training.


http://arxiv.org/abs/2410.22143
AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts. (78%)
Vishal Kumar; Zeyi Liao; Jaylen Jones; Huan Sun
  Although large language models (LLMs) are typically aligned, they remain
vulnerable to jailbreaking through either carefully crafted prompts in natural
language or, interestingly, gibberish adversarial suffixes. However, gibberish
tokens have received relatively less attention despite their success in
attacking aligned LLMs. Recent work, AmpleGCG~\citep{liao2024amplegcg},
demonstrates that a generative model can quickly produce numerous customizable
gibberish adversarial suffixes for any harmful query, exposing a range of
alignment gaps in out-of-distribution (OOD) language spaces. To bring more
attention to this area, we introduce AmpleGCG-Plus, an enhanced version that
achieves better performance in fewer attempts. Through a series of exploratory
experiments, we identify several training strategies to improve the learning of
gibberish suffixes. Our results, verified under a strict evaluation setting,
show that it outperforms AmpleGCG on both open-weight and closed-source models,
achieving increases in attack success rate (ASR) of up to 17\% in the white-box
setting against Llama-2-7B-chat, and more than tripling ASR in the black-box
setting against GPT-4. Notably, AmpleGCG-Plus jailbreaks the newer GPT-4o
series of models at similar rates to GPT-4, and, uncovers vulnerabilities
against the recently proposed circuit breakers defense. We publicly release
AmpleGCG-Plus along with our collected training datasets.


http://arxiv.org/abs/2410.22284
Embedding-based classifiers can detect prompt injection attacks. (64%)
Md. Ahsan Ayub; Subhabrata Majumdar
  Large Language Models (LLMs) are seeing significant adoption in every type of
organization due to their exceptional generative capabilities. However, LLMs
are found to be vulnerable to various adversarial attacks, particularly prompt
injection attacks, which trick them into producing harmful or inappropriate
content. Adversaries execute such attacks by crafting malicious prompts to
deceive the LLMs. In this paper, we propose a novel approach based on
embedding-based Machine Learning (ML) classifiers to protect LLM-based
applications against this severe threat. We leverage three commonly used
embedding models to generate embeddings of malicious and benign prompts and
utilize ML classifiers to predict whether an input prompt is malicious. Out of
several traditional ML methods, we achieve the best performance with
classifiers built using Random Forest and XGBoost. Our classifiers outperform
state-of-the-art prompt injection classifiers available in open-source
implementations, which use encoder-only neural networks.


http://arxiv.org/abs/2410.21791
Enhancing Adversarial Attacks through Chain of Thought. (54%)
Jingbo Su
  Large language models (LLMs) have demonstrated impressive performance across
various domains but remain susceptible to safety concerns. Prior research
indicates that gradient-based adversarial attacks are particularly effective
against aligned LLMs and the chain of thought (CoT) prompting can elicit
desired answers through step-by-step reasoning. This paper proposes enhancing
the robustness of adversarial attacks on aligned LLMs by integrating CoT
prompts with the greedy coordinate gradient (GCG) technique. Using CoT triggers
instead of affirmative targets stimulates the reasoning abilities of backend
LLMs, thereby improving the transferability and universality of adversarial
attacks. We conducted an ablation study comparing our CoT-GCG approach with
Amazon Web Services auto-cot. Results revealed our approach outperformed both
the baseline GCG attack and CoT prompting. Additionally, we used Llama Guard to
evaluate potentially harmful interactions, providing a more objective risk
assessment of entire conversations compared to matching outputs to rejection
phrases. The code of this paper is available at
https://github.com/sujingbo0217/CS222W24-LLM-Attack.


http://arxiv.org/abs/2410.22425
Power side-channel leakage localization through adversarial training of deep neural networks. (11%)
Jimmy Gammell; Anand Raghunathan; Kaushik Roy
  Supervised deep learning has emerged as an effective tool for carrying out
power side-channel attacks on cryptographic implementations. While
increasingly-powerful deep learning-based attacks are regularly published,
comparatively-little work has gone into using deep learning to defend against
these attacks. In this work we propose a technique for identifying which
timesteps in a power trace are responsible for leaking a cryptographic key,
through an adversarial game between a deep learning-based side-channel attacker
which seeks to classify a sensitive variable from the power traces recorded
during encryption, and a trainable noise generator which seeks to thwart this
attack by introducing a minimal amount of noise into the power traces. We
demonstrate on synthetic datasets that our method can outperform existing
techniques in the presence of common countermeasures such as Boolean masking
and trace desynchronization. Results on real datasets are weak because the
technique is highly sensitive to hyperparameters and early-stop point, and we
lack a holdout dataset with ground truth knowledge of leaking points for model
selection. Nonetheless, we believe our work represents an important first step
towards deep side-channel leakage localization without relying on strong
assumptions about the implementation or the nature of its leakage. An
open-source PyTorch implementation of our experiments is provided.


http://arxiv.org/abs/2410.21736
Enhancing Safety and Robustness of Vision-Based Controllers via Reachability Analysis. (1%)
Kaustav Chakraborty; Aryaman Gupta; Somil Bansal
  Autonomous systems, such as self-driving cars and drones, have made
significant strides in recent years by leveraging visual inputs and machine
learning for decision-making and control. Despite their impressive performance,
these vision-based controllers can make erroneous predictions when faced with
novel or out-of-distribution inputs. Such errors can cascade into catastrophic
system failures and compromise system safety. In this work, we compute Neural
Reachable Tubes, which act as parameterized approximations of Backward
Reachable Tubes to stress-test the vision-based controllers and mine their
failure modes. The identified failures are then used to enhance the system
safety through both offline and online methods. The online approach involves
training a classifier as a run-time failure monitor to detect closed-loop,
system-level failures, subsequently triggering a fallback controller that
robustly handles these detected failures to preserve system safety. For the
offline approach, we improve the original controller via incremental training
using a carefully augmented failure dataset, resulting in a more robust
controller that is resistant to the known failure modes. In either approach,
the system is safeguarded against shortcomings that transcend the vision-based
controller and pertain to the closed-loop safety of the overall system. We
validate the proposed approaches on an autonomous aircraft taxiing task that
involves using a vision-based controller to guide the aircraft towards the
centerline of the runway. Our results show the efficacy of the proposed
algorithms in identifying and handling system-level failures, outperforming
methods that rely on controller prediction error or uncertainty quantification
for identifying system failures.


http://arxiv.org/abs/2410.22307
SVIP: Towards Verifiable Inference of Open-source Large Language Models. (1%)
Yifan Sun; Yuhang Li; Yue Zhang; Yuchen Jin; Huan Zhang
  The ever-increasing size of open-source Large Language Models (LLMs) renders
local deployment impractical for individual users. Decentralized computing has
emerged as a cost-effective solution, allowing individuals and small companies
to perform LLM inference for users using surplus computational power. However,
a computing provider may stealthily substitute the requested LLM with a
smaller, less capable model without consent from users, thereby benefiting from
cost savings. We introduce SVIP, a secret-based verifiable LLM inference
protocol. Unlike existing solutions based on cryptographic or game-theoretic
techniques, our method is computationally effective and does not rest on strong
assumptions. Our protocol requires the computing provider to return both the
generated text and processed hidden representations from LLMs. We then train a
proxy task on these representations, effectively transforming them into a
unique model identifier. With our protocol, users can reliably verify whether
the computing provider is acting honestly. A carefully integrated secret
mechanism further strengthens its security. We thoroughly analyze our protocol
under multiple strong and adaptive adversarial scenarios. Our extensive
experiments demonstrate that SVIP is accurate, generalizable, computationally
efficient, and resistant to various attacks. Notably, SVIP achieves false
negative rates below 5% and false positive rates below 3%, while requiring less
than 0.01 seconds per prompt query for verification.


http://arxiv.org/abs/2411.00836
DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. (1%)
Chengke Zou; Xingang Guo; Rui Yang; Junyu Zhang; Bin Hu; Huan Zhang
  The rapid advancements in Vision-Language Models (VLMs) have shown great
potential in tackling mathematical reasoning tasks that involve visual context.
Unlike humans who can reliably apply solution steps to similar problems with
minor modifications, we found that SOTA VLMs like GPT-4o can consistently fail
in these scenarios, revealing limitations in their mathematical reasoning
capabilities. In this paper, we investigate the mathematical reasoning
robustness in VLMs and evaluate how well these models perform under different
variants of the same question, such as changes in visual numerical values or
function graphs. While several vision-based math benchmarks have been developed
to assess VLMs' problem-solving capabilities, these benchmarks contain only
static sets of problems and cannot easily evaluate mathematical reasoning
robustness. To fill this gap, we introduce DynaMath, a dynamic visual math
benchmark designed for in-depth assessment of VLMs. DynaMath includes 501
high-quality, multi-topic seed questions, each represented as a Python program.
Those programs are carefully designed and annotated to enable the automatic
generation of a much larger set of concrete questions, including many different
types of visual and textual variations. DynaMath allows us to evaluate the
generalization ability of VLMs, by assessing their performance under varying
input conditions of a seed question. We evaluated 14 SOTA VLMs with 5,010
generated concrete questions. Our results show that the worst-case model
accuracy, defined as the percentage of correctly answered seed questions in all
10 variants, is significantly lower than the average-case accuracy. Our
analysis emphasizes the need to study the robustness of VLMs' reasoning
abilities, and DynaMath provides valuable insights to guide the development of
more reliable models for mathematical reasoning.


http://arxiv.org/abs/2410.20893
Evaluating the Robustness of LiDAR Point Cloud Tracking Against Adversarial Attack. (99%)
Shengjing Tian; Yinan Han; Xiantong Zhao; Bin Liu; Xiuping Liu
  In this study, we delve into the robustness of neural network-based LiDAR
point cloud tracking models under adversarial attacks, a critical aspect often
overlooked in favor of performance enhancement. These models, despite
incorporating advanced architectures like Transformer or Bird's Eye View (BEV),
tend to neglect robustness in the face of challenges such as adversarial
attacks, domain shifts, or data corruption. We instead focus on the robustness
of the tracking models under the threat of adversarial attacks. We begin by
establishing a unified framework for conducting adversarial attacks within the
context of 3D object tracking, which allows us to thoroughly investigate both
white-box and black-box attack strategies. For white-box attacks, we tailor
specific loss functions to accommodate various tracking paradigms and extend
existing methods such as FGSM, C\&W, and PGD to the point cloud domain. In
addressing black-box attack scenarios, we introduce a novel transfer-based
approach, the Target-aware Perturbation Generation (TAPG) algorithm, with the
dual objectives of achieving high attack performance and maintaining low
perceptibility. This method employs a heuristic strategy to enforce sparse
attack constraints and utilizes random sub-vector factorization to bolster
transferability. Our experimental findings reveal a significant vulnerability
in advanced tracking methods when subjected to both black-box and white-box
attacks, underscoring the necessity for incorporating robustness against
adversarial attacks into the design of LiDAR point cloud tracking models.
Notably, compared to existing methods, the TAPG also strikes an optimal balance
between the effectiveness of the attack and the concealment of the
perturbations.


http://arxiv.org/abs/2410.21471
AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models. (96%)
Yaopei Zeng; Yuanpu Cao; Bochuan Cao; Yurui Chang; Jinghui Chen; Lu Lin
  Recent advances in diffusion models have significantly enhanced the quality
of image synthesis, yet they have also introduced serious safety concerns,
particularly the generation of Not Safe for Work (NSFW) content. Previous
research has demonstrated that adversarial prompts can be used to generate NSFW
content. However, such adversarial text prompts are often easily detectable by
text-based filters, limiting their efficacy. In this paper, we expose a
previously overlooked vulnerability: adversarial image attacks targeting
Image-to-Image (I2I) diffusion models. We propose AdvI2I, a novel framework
that manipulates input images to induce diffusion models to generate NSFW
content. By optimizing a generator to craft adversarial images, AdvI2I
circumvents existing defense mechanisms, such as Safe Latent Diffusion (SLD),
without altering the text prompts. Furthermore, we introduce AdvI2I-Adaptive,
an enhanced version that adapts to potential countermeasures and minimizes the
resemblance between adversarial images and NSFW concept embeddings, making the
attack more resilient against defenses. Through extensive experiments, we
demonstrate that both AdvI2I and AdvI2I-Adaptive can effectively bypass current
safeguards, highlighting the urgent need for stronger security measures to
address the misuse of I2I diffusion models.


http://arxiv.org/abs/2410.20971
BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks. (93%)
Yunhan Zhao; Xiang Zheng; Lin Luo; Yige Li; Xingjun Ma; Yu-Gang Jiang
  In this paper, we focus on black-box defense for VLMs against jailbreak
attacks. Existing black-box defense methods are either unimodal or bimodal.
Unimodal methods enhance either the vision or language module of the VLM, while
bimodal methods robustify the model through text-image representation
realignment. However, these methods suffer from two limitations: 1) they fail
to fully exploit the cross-modal information, or 2) they degrade the model
performance on benign inputs. To address these limitations, we propose a novel
blue-team method BlueSuffix that defends target VLMs against jailbreak attacks
without compromising its performance under black-box setting. BlueSuffix
includes three key components: 1) a visual purifier against jailbreak images,
2) a textual purifier against jailbreak texts, and 3) a blue-team suffix
generator using reinforcement fine-tuning for enhancing cross-modal robustness.
We empirically show on four VLMs (LLaVA, MiniGPT-4, InstructionBLIP, and
Gemini) and four safety benchmarks (Harmful Instruction, AdvBench,
MM-SafetyBench, and RedTeam-2K) that BlueSuffix outperforms the baseline
defenses by a significant margin. Our BlueSuffix opens up a promising direction
for defending VLMs against jailbreak attacks. Code is available at
https://github.com/Vinsonzyh/BlueSuffix.


http://arxiv.org/abs/2410.21492
FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks. (91%)
Jiongxiao Wang; Fangzhou Wu; Wendi Li; Jinsheng Pan; Edward Suh; Z. Morley Mao; Muhao Chen; Chaowei Xiao
  Large language models (LLMs) have been widely deployed as the backbone with
additional tools and text information for real-world applications. However,
integrating external information into LLM-integrated applications raises
significant security concerns. Among these, prompt injection attacks are
particularly threatening, where malicious instructions injected in the external
text information can exploit LLMs to generate answers as the attackers desire.
While both training-time and test-time defense methods have been developed to
mitigate such attacks, the unaffordable training costs associated with
training-time methods and the limited effectiveness of existing test-time
methods make them impractical. This paper introduces a novel test-time defense
strategy, named Formatting AuThentication with Hash-based tags (FATH). Unlike
existing approaches that prevent LLMs from answering additional instructions in
external text, our method implements an authentication system, requiring LLMs
to answer all received instructions with a security policy and selectively
filter out responses to user instructions as the final output. To achieve this,
we utilize hash-based authentication tags to label each response, facilitating
accurate identification of responses according to the user's instructions and
improving the robustness against adaptive attacks. Comprehensive experiments
demonstrate that our defense method can effectively defend against indirect
prompt injection attacks, achieving state-of-the-art performance under Llama3
and GPT3.5 models across various attack methods. Our code is released at:
https://github.com/Jayfeather1024/FATH


http://arxiv.org/abs/2410.21443
TACO: Adversarial Camouflage Optimization on Trucks to Fool Object Detectors. (88%)
Adonisz Dimitriu; Tamás Michaletzky; Viktor Remeli
  Adversarial attacks threaten the reliability of machine learning models in
critical applications like autonomous vehicles and defense systems. As object
detectors become more robust with models like YOLOv8, developing effective
adversarial methodologies is increasingly challenging. We present Truck
Adversarial Camouflage Optimization (TACO), a novel framework that generates
adversarial camouflage patterns on 3D vehicle models to deceive
state-of-the-art object detectors. Adopting Unreal Engine 5, TACO integrates
differentiable rendering with a Photorealistic Rendering Network to optimize
adversarial textures targeted at YOLOv8. To ensure the generated textures are
both effective in deceiving detectors and visually plausible, we introduce the
Convolutional Smooth Loss function, a generalized smooth loss function.
Experimental evaluations demonstrate that TACO significantly degrades YOLOv8's
detection performance, achieving an AP@0.5 of 0.0099 on unseen test data.
Furthermore, these adversarial patterns exhibit strong transferability to other
object detection models such as Faster R-CNN and earlier YOLO versions.


http://arxiv.org/abs/2410.20940
Attacking Misinformation Detection Using Adversarial Examples Generated by Language Models. (84%)
Piotr Przybyła
  We investigate the challenge of generating adversarial examples to test the
robustness of text classification algorithms detecting low-credibility content,
including propaganda, false claims, rumours and hyperpartisan news. We focus on
simulation of content moderation by setting realistic limits on the number of
queries an attacker is allowed to attempt. Within our solution (TREPAT),
initial rephrasings are generated by large language models with prompts
inspired by meaning-preserving NLP tasks, e.g. text simplification and style
transfer. Subsequently, these modifications are decomposed into small changes,
applied through beam search procedure until the victim classifier changes its
decision. The evaluation confirms the superiority of our approach in the
constrained scenario, especially in case of long input text (news articles),
where exhaustive search is not feasible.


http://arxiv.org/abs/2410.20911
Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks. (80%)
Dario Pasquini; Evgenios M. Kornaropoulos; Giuseppe Ateniese
  Large language models (LLMs) are increasingly being harnessed to automate
cyberattacks, making sophisticated exploits more accessible and scalable. In
response, we propose a new defense strategy tailored to counter LLM-driven
cyberattacks. We introduce Mantis, a defensive framework that exploits LLMs'
susceptibility to adversarial inputs to undermine malicious operations. Upon
detecting an automated cyberattack, Mantis plants carefully crafted inputs into
system responses, leading the attacker's LLM to disrupt their own operations
(passive defense) or even compromise the attacker's machine (active defense).
By deploying purposefully vulnerable decoy services to attract the attacker and
using dynamic prompt injections for the attacker's LLM, Mantis can autonomously
hack back the attacker. In our experiments, Mantis consistently achieved over
95% effectiveness against automated LLM-driven attacks. To foster further
research and collaboration, Mantis is available as an open-source tool:
https://github.com/pasquini-dario/project_mantis


http://arxiv.org/abs/2410.21083
Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring. (50%)
Honglin Mu; Han He; Yuxin Zhou; Yunlong Feng; Yang Xu; Libo Qin; Xiaoming Shi; Zeming Liu; Xudong Han; Qi Shi; Qingfu Zhu; Wanxiang Che
  Large language model (LLM) safety is a critical issue, with numerous studies
employing red team testing to enhance model security. Among these, jailbreak
methods explore potential vulnerabilities by crafting malicious prompts that
induce model outputs contrary to safety alignments. Existing black-box
jailbreak methods often rely on model feedback, repeatedly submitting queries
with detectable malicious instructions during the attack search process.
Although these approaches are effective, the attacks may be intercepted by
content moderators during the search process. We propose an improved transfer
attack method that guides malicious prompt construction by locally training a
mirror model of the target black-box model through benign data distillation.
This method offers enhanced stealth, as it does not involve submitting
identifiable malicious instructions to the target model during the search
phase. Our approach achieved a maximum attack success rate of 92%, or a
balanced value of 80% with an average of 1.5 detectable jailbreak queries per
sample against GPT-3.5 Turbo on a subset of AdvBench. These results underscore
the need for more robust defense mechanisms.


http://arxiv.org/abs/2410.20742
Mitigating Unauthorized Speech Synthesis for Voice Protection. (9%)
Zhisheng Zhang; Qianyi Yang; Derui Wang; Pengyang Huang; Yuxin Cao; Kai Ye; Jie Hao
  With just a few speech samples, it is possible to perfectly replicate a
speaker's voice in recent years, while malicious voice exploitation (e.g.,
telecom fraud for illegal financial gain) has brought huge hazards in our daily
lives. Therefore, it is crucial to protect publicly accessible speech data that
contains sensitive information, such as personal voiceprints. Most previous
defense methods have focused on spoofing speaker verification systems in timbre
similarity but the synthesized deepfake speech is still of high quality. In
response to the rising hazards, we devise an effective, transferable, and
robust proactive protection technology named Pivotal Objective Perturbation
(POP) that applies imperceptible error-minimizing noises on original speech
samples to prevent them from being effectively learned for text-to-speech (TTS)
synthesis models so that high-quality deepfake speeches cannot be generated. We
conduct extensive experiments on state-of-the-art (SOTA) TTS models utilizing
objective and subjective metrics to comprehensively evaluate our proposed
method. The experimental results demonstrate outstanding effectiveness and
transferability across various models. Compared to the speech unclarity score
of 21.94% from voice synthesizers trained on samples without protection,
POP-protected samples significantly increase it to 127.31%. Moreover, our
method shows robustness against noise reduction and data augmentation
techniques, thereby greatly reducing potential hazards.


http://arxiv.org/abs/2410.21637
Mitigating Paraphrase Attacks on Machine-Text Detectors via Paraphrase Inversion. (1%)
Rafael Rivera Soto; Barry Chen; Nicholas Andrews
  High-quality paraphrases are easy to produce using instruction-tuned language
models or specialized paraphrasing models. Although this capability has a
variety of benign applications, paraphrasing
attacks$\unicode{x2013}$paraphrases applied to machine-generated
texts$\unicode{x2013}$are known to significantly degrade the performance of
machine-text detectors. This motivates us to consider the novel problem of
paraphrase inversion, where, given paraphrased text, the objective is to
recover an approximation of the original text. The closer the approximation is
to the original text, the better machine-text detectors will perform. We
propose an approach which frames the problem as translation from paraphrased
text back to the original text, which requires examples of texts and
corresponding paraphrases to train the inversion model. Fortunately, such
training data can easily be generated, given a corpus of original texts and one
or more paraphrasing models. We find that language models such as GPT-4 and
Llama-3 exhibit biases when paraphrasing which an inversion model can learn
with a modest amount of data. Perhaps surprisingly, we also find that such
models generalize well, including to paraphrase models unseen at training time.
Finally, we show that when combined with a paraphrased-text detector, our
inversion models provide an effective defense against paraphrasing attacks, and
overall our approach yields an average improvement of +22% AUROC across seven
machine-text detectors and three different domains.


http://arxiv.org/abs/2410.21146
Palisade -- Prompt Injection Detection Framework. (1%)
Sahasra Kokkula; Somanathan R; Nandavardhan R; Aashishkumar; G Divya
  The advent of Large Language Models LLMs marks a milestone in Artificial
Intelligence, altering how machines comprehend and generate human language.
However, LLMs are vulnerable to malicious prompt injection attacks, where
crafted inputs manipulate the models behavior in unintended ways, compromising
system integrity and causing incorrect outcomes. Conventional detection methods
rely on static, rule-based approaches, which often fail against sophisticated
threats like abnormal token sequences and alias substitutions, leading to
limited adaptability and higher rates of false positives and false
negatives.This paper proposes a novel NLP based approach for prompt injection
detection, emphasizing accuracy and optimization through a layered input
screening process. In this framework, prompts are filtered through three
distinct layers rule-based, ML classifier, and companion LLM before reaching
the target model, thereby minimizing the risk of malicious interaction.Tests
show the ML classifier achieves the highest accuracy among individual layers,
yet the multi-layer framework enhances overall detection accuracy by reducing
false negatives. Although this increases false positives, it minimizes the risk
of overlooking genuine injected prompts, thus prioritizing security.This
multi-layered detection approach highlights LLM vulnerabilities and provides a
comprehensive framework for future research, promoting secure interactions
between humans and AI systems.


http://arxiv.org/abs/2410.20788
SCULPT: Systematic Tuning of Long Prompts. (1%)
Shanu Kumar; Akhila Yesantarao Venkata; Shubhanshu Khandelwal; Bishal Santra; Parag Agrawal; Manish Gupta
  Prompt optimization is essential for effective utilization of large language
models (LLMs) across diverse tasks. While existing optimization methods are
effective in optimizing short prompts, they struggle with longer, more complex
ones, often risking information loss and being sensitive to small
perturbations. To address these challenges, we propose SCULPT (Systematic
Tuning of Long Prompts), a framework that treats prompt optimization as a
hierarchical tree refinement problem. SCULPT represents prompts as tree
structures, enabling targeted modifications while preserving contextual
integrity. It employs a Critic-Actor framework that generates reflections and
applies actions to refine the prompt. Evaluations demonstrate SCULPT's
effectiveness on long prompts, its robustness to adversarial perturbations, and
its ability to generate high-performing prompts even without any initial
human-written prompt. Compared to existing state of the art methods, SCULPT
consistently improves LLM performance by preserving essential task information
while applying structured refinements. Both qualitative and quantitative
analyses show that SCULPT produces more stable and interpretable prompt
modifications, ensuring better generalization across tasks.


http://arxiv.org/abs/2410.20432
Integrating uncertainty quantification into randomized smoothing based robustness guarantees. (98%)
Sina Däubener; Kira Maag; David Krueger; Asja Fischer
  Deep neural networks have proven to be extremely powerful, however, they are
also vulnerable to adversarial attacks which can cause hazardous incorrect
predictions in safety-critical applications. Certified robustness via
randomized smoothing gives a probabilistic guarantee that the smoothed
classifier's predictions will not change within an $\ell_2$-ball around a given
input. On the other hand (uncertainty) score-based rejection is a technique
often applied in practice to defend models against adversarial attacks. In this
work, we fuse these two approaches by integrating a classifier that abstains
from predicting when uncertainty is high into the certified robustness
framework. This allows us to derive two novel robustness guarantees for
uncertainty aware classifiers, namely (i) the radius of an $\ell_2$-ball around
the input in which the same label is predicted and uncertainty remains low and
(ii) the $\ell_2$-radius of a ball in which the predictions will either not
change or be uncertain. While the former provides robustness guarantees with
respect to attacks aiming at increased uncertainty, the latter informs about
the amount of input perturbation necessary to lead the uncertainty aware model
into a wrong prediction. Notably, this is on CIFAR10 up to 20.93% larger than
for models not allowing for uncertainty based rejection. We demonstrate, that
the novel framework allows for a systematic robustness evaluation of different
network architectures and uncertainty measures and to identify desired
properties of uncertainty quantification techniques. Moreover, we show that
leveraging uncertainty in a smoothed classifier helps out-of-distribution
detection.


http://arxiv.org/abs/2410.21330
LLM Robustness Against Misinformation in Biomedical Question Answering. (80%)
Alexander Bondarenko; Adrian Viehweger
  The retrieval-augmented generation (RAG) approach is used to reduce the
confabulation of large language models (LLMs) for question answering by
retrieving and providing additional context coming from external knowledge
sources (e.g., by adding the context to the prompt). However, injecting
incorrect information can mislead the LLM to generate an incorrect answer.
  In this paper, we evaluate the effectiveness and robustness of four LLMs
against misinformation - Gemma 2, GPT-4o-mini, Llama~3.1, and Mixtral - in
answering biomedical questions. We assess the answer accuracy on yes-no and
free-form questions in three scenarios: vanilla LLM answers (no context is
provided), "perfect" augmented generation (correct context is provided), and
prompt-injection attacks (incorrect context is provided). Our results show that
Llama 3.1 (70B parameters) achieves the highest accuracy in both vanilla
(0.651) and "perfect" RAG (0.802) scenarios. However, the accuracy gap between
the models almost disappears with "perfect" RAG, suggesting its potential to
mitigate the LLM's size-related effectiveness differences.
  We further evaluate the ability of the LLMs to generate malicious context on
one hand and the LLM's robustness against prompt-injection attacks on the other
hand, using metrics such as attack success rate (ASR), accuracy under attack,
and accuracy drop. As adversaries, we use the same four LLMs (Gemma 2,
GPT-4o-mini, Llama 3.1, and Mixtral) to generate incorrect context that is
injected in the target model's prompt. Interestingly, Llama is shown to be the
most effective adversary, causing accuracy drops of up to 0.48 for vanilla
answers and 0.63 for "perfect" RAG across target models. Our analysis reveals
that robustness rankings vary depending on the evaluation measure, highlighting
the complexity of assessing LLM resilience to adversarial attacks.


http://arxiv.org/abs/2410.21337
Fine-tuned Large Language Models (LLMs): Improved Prompt Injection Attacks Detection. (1%)
Md Abdur Rahman; Fan Wu; Alfredo Cuzzocrea; Sheikh Iqbal Ahamed
  Large language models (LLMs) are becoming a popular tool as they have
significantly advanced in their capability to tackle a wide range of
language-based tasks. However, LLMs applications are highly vulnerable to
prompt injection attacks, which poses a critical problem. These attacks target
LLMs applications through using carefully designed input prompts to divert the
model from adhering to original instruction, thereby it could execute
unintended actions. These manipulations pose serious security threats which
potentially results in data leaks, biased outputs, or harmful responses. This
project explores the security vulnerabilities in relation to prompt injection
attacks. To detect whether a prompt is vulnerable or not, we follows two
approaches: 1) a pre-trained LLM, and 2) a fine-tuned LLM. Then, we conduct a
thorough analysis and comparison of the classification performance. Firstly, we
use pre-trained XLM-RoBERTa model to detect prompt injections using test
dataset without any fine-tuning and evaluate it by zero-shot classification.
Then, this proposed work will apply supervised fine-tuning to this pre-trained
LLM using a task-specific labeled dataset from deepset in huggingface, and this
fine-tuned model achieves impressive results with 99.13\% accuracy, 100\%
precision, 98.33\% recall and 99.15\% F1-score thorough rigorous
experimentation and evaluation. We observe that our approach is highly
efficient in detecting prompt injection attacks.


http://arxiv.org/abs/2410.20103
Adversarial Attacks Against Double RIS-Assisted MIMO Systems-based Autoencoder in Finite-Scattering Environments. (99%)
Bui Duc Son; Ngo Nam Khanh; Chien Trinh Van; Dong In Kim
  Autoencoder permits the end-to-end optimization and design of wireless
communication systems to be more beneficial than traditional signal processing.
However, this emerging learning-based framework has weaknesses, especially
sensitivity to physical attacks. This paper explores adversarial attacks
against a double reconfigurable intelligent surface (RIS)-assisted
multiple-input and multiple-output (MIMO)-based autoencoder, where an adversary
employs encoded and decoded datasets to create adversarial perturbation and
fool the system. Because of the complex and dynamic data structures,
adversarial attacks are not unique, each having its own benefits. We,
therefore, propose three algorithms generating adversarial examples and
perturbations to attack the RIS-MIMO-based autoencoder, exploiting the gradient
descent and allowing for flexibility via varying the input dimensions.
Numerical results show that the proposed adversarial attack-based algorithm
significantly degrades the system performance regarding the symbol error rate
compared to the jamming attacks.


http://arxiv.org/abs/2410.20197
Transferable Adversarial Attacks on SAM and Its Downstream Models. (99%)
Song Xia; Wenhan Yang; Yi Yu; Xun Lin; Henghui Ding; Ling-Yu Duan; Xudong Jiang
  The utilization of large foundational models has a dilemma: while fine-tuning
downstream tasks from them holds promise for making use of the well-generalized
knowledge in practical applications, their open accessibility also poses
threats of adverse usage. This paper, for the first time, explores the
feasibility of adversarial attacking various downstream models fine-tuned from
the segment anything model (SAM), by solely utilizing the information from the
open-sourced SAM. In contrast to prevailing transfer-based adversarial attacks,
we demonstrate the existence of adversarial dangers even without accessing the
downstream task and dataset to train a similar surrogate model. To enhance the
effectiveness of the adversarial attack towards models fine-tuned on unknown
datasets, we propose a universal meta-initialization (UMI) algorithm to extract
the intrinsic vulnerability inherent in the foundation model, which is then
utilized as the prior knowledge to guide the generation of adversarial
perturbations. Moreover, by formulating the gradient difference in the
attacking process between the open-sourced SAM and its fine-tuned downstream
models, we theoretically demonstrate that a deviation occurs in the adversarial
update direction by directly maximizing the distance of encoded feature
embeddings in the open-sourced SAM. Consequently, we propose a gradient robust
loss that simulates the associated uncertainty with gradient-based noise
augmentation to enhance the robustness of generated adversarial examples (AEs)
towards this deviation, thus improving the transferability. Extensive
experiments demonstrate the effectiveness of the proposed universal
meta-initialized and gradient robust adversarial attack (UMI-GRAT) toward SAMs
and their downstream models. Code is available at
https://github.com/xiasong0501/GRAT.


http://arxiv.org/abs/2410.20097
Generative Adversarial Patches for Physical Attacks on Cross-Modal Pedestrian Re-Identification. (98%)
Yue Su; Hao Li; Maoguo Gong
  Visible-infrared pedestrian Re-identification (VI-ReID) aims to match
pedestrian images captured by infrared cameras and visible cameras. However,
VI-ReID, like other traditional cross-modal image matching tasks, poses
significant challenges due to its human-centered nature. This is evidenced by
the shortcomings of existing methods, which struggle to extract common features
across modalities, while losing valuable information when bridging the gap
between them in the implicit feature space, potentially compromising security.
To address this vulnerability, this paper introduces the first physical
adversarial attack against VI-ReID models. Our method, termed Edge-Attack,
specifically tests the models' ability to leverage deep-level implicit features
by focusing on edge information, the most salient explicit feature
differentiating individuals across modalities. Edge-Attack utilizes a novel
two-step approach. First, a multi-level edge feature extractor is trained in a
self-supervised manner to capture discriminative edge representations for each
individual. Second, a generative model based on Vision Transformer Generative
Adversarial Networks (ViTGAN) is employed to generate adversarial patches
conditioned on the extracted edge features. By applying these patches to
pedestrian clothing, we create realistic, physically-realizable adversarial
samples. This black-box, self-supervised approach ensures the generalizability
of our attack against various VI-ReID models. Extensive experiments on
SYSU-MM01 and RegDB datasets, including real-world deployments, demonstrate the
effectiveness of Edge- Attack in significantly degrading the performance of
state-of-the-art VI-ReID methods.


http://arxiv.org/abs/2410.20136
CodePurify: Defend Backdoor Attacks on Neural Code Models via Entropy-based Purification. (76%)
Fangwen Mu; Junjie Wang; Zhuohao Yu; Lin Shi; Song Wang; Mingyang Li; Qing Wang
  Neural code models have found widespread success in tasks pertaining to code
intelligence, yet they are vulnerable to backdoor attacks, where an adversary
can manipulate the victim model's behavior by inserting triggers into the
source code. Recent studies indicate that advanced backdoor attacks can achieve
nearly 100% attack success rates on many software engineering tasks. However,
effective defense techniques against such attacks remain insufficiently
explored. In this study, we propose CodePurify, a novel defense against
backdoor attacks on code models through entropy-based purification.
Entropy-based purification involves the process of precisely detecting and
eliminating the possible triggers in the source code while preserving its
semantic information. Within this process, CodePurify first develops a
confidence-driven entropy-based measurement to determine whether a code snippet
is poisoned and, if so, locates the triggers. Subsequently, it purifies the
code by substituting the triggers with benign tokens using a masked language
model. We extensively evaluate CodePurify against four advanced backdoor
attacks across three representative tasks and two popular code models. The
results show that CodePurify significantly outperforms four commonly used
defense baselines, improving average defense performance by at least 40%, 40%,
and 12% across the three tasks, respectively. These findings highlight the
potential of CodePurify to serve as a robust defense against backdoor attacks
on neural code models.


http://arxiv.org/abs/2410.20250
Robust Model Evaluation over Large-scale Federated Networks. (2%)
Amir Najafi; Samin Mahdizadeh Sani; Farzan Farnia
  In this paper, we address the challenge of certifying the performance of a
machine learning model on an unseen target network, using measurements from an
available source network. We focus on a scenario where heterogeneous datasets
are distributed across a source network of clients, all connected to a central
server. Specifically, consider a source network "A" composed of $K$ clients,
each holding private data from unique and heterogeneous distributions, which
are assumed to be independent samples from a broader meta-distribution $\mu$.
Our goal is to provide certified guarantees for the model's performance on a
different, unseen target network "B," governed by another meta-distribution
$\mu'$, assuming the deviation between $\mu$ and $\mu'$ is bounded by either
the Wasserstein distance or an $f$-divergence. We derive theoretical guarantees
for the model's empirical average loss and provide uniform bounds on the risk
CDF, where the latter correspond to novel and adversarially robust versions of
the Glivenko-Cantelli theorem and the Dvoretzky-Kiefer-Wolfowitz (DKW)
inequality. Our bounds are computable in polynomial time with a polynomial
number of queries to the $K$ clients, preserving client privacy by querying
only the model's (potentially adversarial) loss on private data. We also
establish non-asymptotic generalization bounds that consistently converge to
zero as both $K$ and the minimum client sample size grow. Extensive empirical
evaluations validate the robustness and practicality of our bounds across
real-world tasks.


http://arxiv.org/abs/2410.21276
GPT-4o System Card. (76%)
Tony OpenAI; Tony :; Aaron Tony Hurst; Adam Tony Lerer; Adam P. Tony Goucher; Adam Tony Perelman; Aditya Tony Ramesh; Aidan Tony Clark; AJ Tony Ostrow; Akila Tony Welihinda; Alan Tony Hayes; Alec Tony Radford; Aleksander Tony Mądry; Alex Tony Baker-Whitcomb; Alex Tony Beutel; Alex Tony Borzunov; Alex Tony Carney; Alex Tony Chow; Alex Tony Kirillov; Alex Tony Nichol; Alex Tony Paino; Alex Tony Renzin; Alex Tachard Tony Passos; Alexander Tony Kirillov; Alexi Tony Christakis; Alexis Tony Conneau; Ali Tony Kamali; Allan Tony Jabri; Allison Tony Moyer; Allison Tony Tam; Amadou Tony Crookes; Amin Tony Tootoochian; Amin Tony Tootoonchian; Ananya Tony Kumar; Andrea Tony Vallone; Andrej Tony Karpathy; Andrew Tony Braunstein; Andrew Tony Cann; Andrew Tony Codispoti; Andrew Tony Galu; Andrew Tony Kondrich; Andrew Tony Tulloch; Andrey Tony Mishchenko; Angela Tony Baek; Angela Tony Jiang; Antoine Tony Pelisse; Antonia Tony Woodford; Anuj Tony Gosalia; Arka Tony Dhar; Ashley Tony Pantuliano; Avi Tony Nayak; Avital Tony Oliver; Barret Tony Zoph; Behrooz Tony Ghorbani; Ben Tony Leimberger; Ben Tony Rossen; Ben Tony Sokolowsky; Ben Tony Wang; Benjamin Tony Zweig; Beth Tony Hoover; Blake Tony Samic; Bob Tony McGrew; Bobby Tony Spero; Bogo Tony Giertler; Bowen Tony Cheng; Brad Tony Lightcap; Brandon Tony Walkin; Brendan Tony Quinn; Brian Tony Guarraci; Brian Tony Hsu; Bright Tony Kellogg; Brydon Tony Eastman; Camillo Tony Lugaresi; Carroll Tony Wainwright; Cary Tony Bassin; Cary Tony Hudson; Casey Tony Chu; Chad Tony Nelson; Chak Tony Li; Chan Jun Tony Shern; Channing Tony Conger; Charlotte Tony Barette; Chelsea Tony Voss; Chen Tony Ding; Cheng Tony Lu; Chong Tony Zhang; Chris Tony Beaumont; Chris Tony Hallacy; Chris Tony Koch; Christian Tony Gibson; Christina Tony Kim; Christine Tony Choi; Christine Tony McLeavey; Christopher Tony Hesse; Claudia Tony Fischer; Clemens Tony Winter; Coley Tony Czarnecki; Colin Tony Jarvis; Colin Tony Wei; Constantin Tony Koumouzelis; Dane Tony Sherburn; Daniel Tony Kappler; Daniel Tony Levin; Daniel Tony Levy; David Tony Carr; David Tony Farhi; David Tony Mely; David Tony Robinson; David Tony Sasaki; Denny Tony Jin; Dev Tony Valladares; Dimitris Tony Tsipras; Doug Tony Li; Duc Phong Tony Nguyen; Duncan Tony Findlay; Edede Tony Oiwoh; Edmund Tony Wong; Ehsan Tony Asdar; Elizabeth Tony Proehl; Elizabeth Tony Yang; Eric Tony Antonow; Eric Tony Kramer; Eric Tony Peterson; Eric Tony Sigler; Eric Tony Wallace; Eugene Tony Brevdo; Evan Tony Mays; Farzad Tony Khorasani; Felipe Petroski Tony Such; Filippo Tony Raso; Francis Tony Zhang; Lohmann Fred Tony von; Freddie Tony Sulit; Gabriel Tony Goh; Gene Tony Oden; Geoff Tony Salmon; Giulio Tony Starace; Greg Tony Brockman; Hadi Tony Salman; Haiming Tony Bao; Haitang Tony Hu; Hannah Tony Wong; Haoyu Tony Wang; Heather Tony Schmidt; Heather Tony Whitney; Heewoo Tony Jun; Hendrik Tony Kirchner; Henrique Ponde de Oliveira Tony Pinto; Hongyu Tony Ren; Huiwen Tony Chang; Hyung Won Tony Chung; Ian Tony Kivlichan; Ian Tony O'Connell; Ian Tony O'Connell; Ian Tony Osband; Ian Tony Silber; Ian Tony Sohl; Ibrahim Tony Okuyucu; Ikai Tony Lan; Ilya Tony Kostrikov; Ilya Tony Sutskever; Ingmar Tony Kanitscheider; Ishaan Tony Gulrajani; Jacob Tony Coxon; Jacob Tony Menick; Jakub Tony Pachocki; James Tony Aung; James Tony Betker; James Tony Crooks; James Tony Lennon; Jamie Tony Kiros; Jan Tony Leike; Jane Tony Park; Jason Tony Kwon; Jason Tony Phang; Jason Tony Teplitz; Jason Tony Wei; Jason Tony Wolfe; Jay Tony Chen; Jeff Tony Harris; Jenia Tony Varavva; Jessica Gan Tony Lee; Jessica Tony Shieh; Ji Tony Lin; Jiahui Tony Yu; Jiayi Tony Weng; Jie Tony Tang; Jieqi Tony Yu; Joanne Tony Jang; Joaquin Quinonero Tony Candela; Joe Tony Beutler; Joe Tony Landers; Joel Tony Parish; Johannes Tony Heidecke; John Tony Schulman; Jonathan Tony Lachman; Jonathan Tony McKay; Jonathan Tony Uesato; Jonathan Tony Ward; Jong Wook Tony Kim; Joost Tony Huizinga; Jordan Tony Sitkin; Jos Tony Kraaijeveld; Josh Tony Gross; Josh Tony Kaplan; Josh Tony Snyder; Joshua Tony Achiam; Joy Tony Jiao; Joyce Tony Lee; Juntang Tony Zhuang; Justyn Tony Harriman; Kai Tony Fricke; Kai Tony Hayashi; Karan Tony Singhal; Katy Tony Shi; Kavin Tony Karthik; Kayla Tony Wood; Kendra Tony Rimbach; Kenny Tony Hsu; Kenny Tony Nguyen; Keren Tony Gu-Lemberg; Kevin Tony Button; Kevin Tony Liu; Kiel Tony Howe; Krithika Tony Muthukumar; Kyle Tony Luther; Lama Tony Ahmad; Larry Tony Kai; Lauren Tony Itow; Lauren Tony Workman; Leher Tony Pathak; Leo Tony Chen; Li Tony Jing; Lia Tony Guy; Liam Tony Fedus; Liang Tony Zhou; Lien Tony Mamitsuka; Lilian Tony Weng; Lindsay Tony McCallum; Lindsey Tony Held; Long Tony Ouyang; Louis Tony Feuvrier; Lu Tony Zhang; Lukas Tony Kondraciuk; Lukasz Tony Kaiser; Luke Tony Hewitt; Luke Tony Metz; Lyric Tony Doshi; Mada Tony Aflak; Maddie Tony Simens; Madelaine Tony Boyd; Madeleine Tony Thompson; Marat Tony Dukhan; Mark Tony Chen; Mark Tony Gray; Mark Tony Hudnall; Marvin Tony Zhang; Marwan Tony Aljubeh; Mateusz Tony Litwin; Matthew Tony Zeng; Max Tony Johnson; Maya Tony Shetty; Mayank Tony Gupta; Meghan Tony Shah; Mehmet Tony Yatbaz; Meng Jia Tony Yang; Mengchao Tony Zhong; Mia Tony Glaese; Mianna Tony Chen; Michael Tony Janner; Michael Tony Lampe; Michael Tony Petrov; Michael Tony Wu; Michele Tony Wang; Michelle Tony Fradin; Michelle Tony Pokrass; Miguel Tony Castro; Castro Miguel Oom Temudo Tony de; Mikhail Tony Pavlov; Miles Tony Brundage; Miles Tony Wang; Minal Tony Khan; Mira Tony Murati; Mo Tony Bavarian; Molly Tony Lin; Murat Tony Yesildal; Nacho Tony Soto; Natalia Tony Gimelshein; Natalie Tony Cone; Natalie Tony Staudacher; Natalie Tony Summers; Natan Tony LaFontaine; Neil Tony Chowdhury; Nick Tony Ryder; Nick Tony Stathas; Nick Tony Turley; Nik Tony Tezak; Niko Tony Felix; Nithanth Tony Kudige; Nitish Tony Keskar; Noah Tony Deutsch; Noel Tony Bundick; Nora Tony Puckett; Ofir Tony Nachum; Ola Tony Okelola; Oleg Tony Boiko; Oleg Tony Murk; Oliver Tony Jaffe; Olivia Tony Watkins; Olivier Tony Godement; Owen Tony Campbell-Moore; Patrick Tony Chao; Paul Tony McMillan; Pavel Tony Belov; Peng Tony Su; Peter Tony Bak; Peter Tony Bakkum; Peter Tony Deng; Peter Tony Dolan; Peter Tony Hoeschele; Peter Tony Welinder; Phil Tony Tillet; Philip Tony Pronin; Philippe Tony Tillet; Prafulla Tony Dhariwal; Qiming Tony Yuan; Rachel Tony Dias; Rachel Tony Lim; Rahul Tony Arora; Rajan Tony Troll; Randall Tony Lin; Rapha Gontijo Tony Lopes; Raul Tony Puri; Reah Tony Miyara; Reimar Tony Leike; Renaud Tony Gaubert; Reza Tony Zamani; Ricky Tony Wang; Rob Tony Donnelly; Rob Tony Honsby; Rocky Tony Smith; Rohan Tony Sahai; Rohit Tony Ramchandani; Romain Tony Huet; Rory Tony Carmichael; Rowan Tony Zellers; Roy Tony Chen; Ruby Tony Chen; Ruslan Tony Nigmatullin; Ryan Tony Cheu; Saachi Tony Jain; Sam Tony Altman; Sam Tony Schoenholz; Sam Tony Toizer; Samuel Tony Miserendino; Sandhini Tony Agarwal; Sara Tony Culver; Scott Tony Ethersmith; Scott Tony Gray; Sean Tony Grove; Sean Tony Metzger; Shamez Tony Hermani; Shantanu Tony Jain; Shengjia Tony Zhao; Sherwin Tony Wu; Shino Tony Jomoto; Shirong Tony Wu; Tony Shuaiqi; Xia; Sonia Phene; Spencer Papay; Srinivas Narayanan; Steve Coffey; Steve Lee; Stewart Hall; Suchir Balaji; Tal Broda; Tal Stramer; Tao Xu; Tarun Gogineni; Taya Christianson; Ted Sanders; Tejal Patwardhan; Thomas Cunninghman; Thomas Degry; Thomas Dimson; Thomas Raoux; Thomas Shadwell; Tianhao Zheng; Todd Underwood; Todor Markov; Toki Sherbakov; Tom Rubin; Tom Stasi; Tomer Kaftan; Tristan Heywood; Troy Peterson; Tyce Walters; Tyna Eloundou; Valerie Qi; Veit Moeller; Vinnie Monaco; Vishal Kuo; Vlad Fomenko; Wayne Chang; Weiyi Zheng; Wenda Zhou; Wesam Manassra; Will Sheu; Wojciech Zaremba; Yash Patil; Yilei Qian; Yongjik Kim; Youlong Cheng; Yu Zhang; Yuchen He; Yuchen Zhang; Yujia Jin; Yunxing Dai; Yury Malkov
  GPT-4o is an autoregressive omni model that accepts as input any combination
of text, audio, image, and video, and generates any combination of text, audio,
and image outputs. It's trained end-to-end across text, vision, and audio,
meaning all inputs and outputs are processed by the same neural network. GPT-4o
can respond to audio inputs in as little as 232 milliseconds, with an average
of 320 milliseconds, which is similar to human response time in conversation.
It matches GPT-4 Turbo performance on text in English and code, with
significant improvement on text in non-English languages, while also being much
faster and 50\% cheaper in the API. GPT-4o is especially better at vision and
audio understanding compared to existing models. In line with our commitment to
building AI safely and consistent with our voluntary commitments to the White
House, we are sharing the GPT-4o System Card, which includes our Preparedness
Framework evaluations. In this System Card, we provide a detailed look at
GPT-4o's capabilities, limitations, and safety evaluations across multiple
categories, focusing on speech-to-speech while also evaluating text and image
capabilities, and measures we've implemented to ensure the model is safe and
aligned. We also include third-party assessments on dangerous capabilities, as
well as discussion of potential societal impacts of GPT-4o's text and vision
capabilities.


http://arxiv.org/abs/2410.19937
RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction. (64%)
Tanqiu Jiang; Zian Wang; Jiacheng Liang; Changjiang Li; Yuhui Wang; Ting Wang
  Jailbreak attacks circumvent LLMs' built-in safeguards by concealing harmful
queries within jailbreak prompts. While existing defenses primarily focus on
mitigating the effects of jailbreak prompts, they often prove inadequate as
jailbreak prompts can take arbitrary, adaptive forms. This paper presents
RobustKV, a novel defense that adopts a fundamentally different approach by
selectively removing critical tokens of harmful queries from key-value (KV)
caches. Intuitively, for a jailbreak prompt to be effective, its tokens must
achieve sufficient `importance' (as measured by attention scores), which
inevitably lowers the importance of tokens in the concealed harmful query.
Thus, by strategically evicting the KVs of the lowest-ranked tokens, RobustKV
diminishes the presence of the harmful query in the KV cache, thus preventing
the LLM from generating malicious responses. Extensive evaluation using
benchmark datasets and models demonstrates that RobustKV effectively counters
state-of-the-art jailbreak attacks while maintaining the LLM's general
performance on benign queries. Moreover, RobustKV creates an intriguing
evasiveness dilemma for adversaries, forcing them to balance between evading
RobustKV and bypassing the LLM's built-in safeguards. This trade-off
contributes to RobustKV's robustness against adaptive attacks. (warning: this
paper contains potentially harmful content generated by LLMs.)


http://arxiv.org/abs/2410.20019
Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions. (62%)
Poojitha Thota; Shirin Nilizadeh
  Large Language Models have introduced novel opportunities for text
comprehension and generation. Yet, they are vulnerable to adversarial
perturbations and data poisoning attacks, particularly in tasks like text
classification and translation. However, the adversarial robustness of
abstractive text summarization models remains less explored. In this work, we
unveil a novel approach by exploiting the inherent lead bias in summarization
models, to perform adversarial perturbations. Furthermore, we introduce an
innovative application of influence functions, to execute data poisoning, which
compromises the model's integrity. This approach not only shows a skew in the
models behavior to produce desired outcomes but also shows a new behavioral
change, where models under attack tend to generate extractive summaries rather
than abstractive summaries.


http://arxiv.org/abs/2410.19427
Expose Before You Defend: Unifying and Enhancing Backdoor Defenses via Exposed Models. (56%)
Yige Li; Hanxun Huang; Jiaming Zhang; Xingjun Ma; Yu-Gang Jiang
  Backdoor attacks covertly implant triggers into deep neural networks (DNNs)
by poisoning a small portion of the training data with pre-designed backdoor
triggers. This vulnerability is exacerbated in the era of large models, where
extensive (pre-)training on web-crawled datasets is susceptible to compromise.
In this paper, we introduce a novel two-step defense framework named Expose
Before You Defend (EBYD). EBYD unifies existing backdoor defense methods into a
comprehensive defense system with enhanced performance. Specifically, EBYD
first exposes the backdoor functionality in the backdoored model through a
model preprocessing step called backdoor exposure, and then applies detection
and removal methods to the exposed model to identify and eliminate the backdoor
features. In the first step of backdoor exposure, we propose a novel technique
called Clean Unlearning (CUL), which proactively unlearns clean features from
the backdoored model to reveal the hidden backdoor features. We also explore
various model editing/modification techniques for backdoor exposure, including
fine-tuning, model sparsification, and weight perturbation. Using EBYD, we
conduct extensive experiments on 10 image attacks and 6 text attacks across 2
vision datasets (CIFAR-10 and an ImageNet subset) and 4 language datasets
(SST-2, IMDB, Twitter, and AG's News). The results demonstrate the importance
of backdoor exposure for backdoor defense, showing that the exposed models can
significantly benefit a range of downstream defense tasks, including backdoor
label detection, backdoor trigger recovery, backdoor model detection, and
backdoor removal. We hope our work could inspire more research in developing
advanced defense frameworks with exposed models. Our code is available at:
https://github.com/bboylyg/Expose-Before-You-Defend.


http://arxiv.org/abs/2410.20026
Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin-based Scene Representation. (10%)
Hao Ding; Yuqian Zhang; Hongchao Shu; Xu Lian; Ji Woong Kim; Axel Krieger; Mathias Unberath
  Purpose: Surgical phase recognition (SPR) is an integral component of
surgical data science, enabling high-level surgical analysis. End-to-end
trained neural networks that predict surgical phase directly from videos have
shown excellent performance on benchmarks. However, these models struggle with
robustness due to non-causal associations in the training set, resulting in
poor generalizability. Our goal is to improve model robustness to variations in
the surgical videos by leveraging the digital twin (DT) paradigm -- an
intermediary layer to separate high-level analysis (SPR) from low-level
processing (geometric understanding). This approach takes advantage of the
recent vision foundation models that ensure reliable low-level scene
understanding to craft DT-based scene representations that support various
high-level tasks.
  Methods: We present a DT-based framework for SPR from videos. The framework
employs vision foundation models to extract representations. We embed the
representation in place of raw video inputs in the state-of-the-art Surgformer
model. The framework is trained on the Cholec80 dataset and evaluated on
out-of-distribution (OOD) and corrupted test samples.
  Results: Contrary to the vulnerability of the baseline model, our framework
demonstrates strong robustness on both OOD and corrupted samples, with a
video-level accuracy of 51.1 on the challenging CRCD dataset, 96.0 on an
internal robotics training dataset, and 64.4 on a highly corrupted Cholec80
test set.
  Conclusion: Our findings lend support to the thesis that DT-based scene
representations are effective in enhancing model robustness. Future work will
seek to improve the feature informativeness, automate feature extraction, and
incorporate interpretability for a more comprehensive framework.


http://arxiv.org/abs/2410.18648
GADT: Enhancing Transferable Adversarial Attacks through Gradient-guided Adversarial Data Transformation. (99%)
Yating Ma; Xiaogang Xu; Liming Fang; Zhe Liu
  Current Transferable Adversarial Examples (TAE) are primarily generated by
adding Adversarial Noise (AN). Recent studies emphasize the importance of
optimizing Data Augmentation (DA) parameters along with AN, which poses a
greater threat to real-world AI applications. However, existing DA-based
strategies often struggle to find optimal solutions due to the challenging DA
search procedure without proper guidance. In this work, we propose a novel
DA-based attack algorithm, GADT. GADT identifies suitable DA parameters through
iterative antagonism and uses posterior estimates to update AN based on these
parameters. We uniquely employ a differentiable DA operation library to
identify adversarial DA parameters and introduce a new loss function as a
metric during DA optimization. This loss term enhances adversarial effects
while preserving the original image content, maintaining attack crypticity.
Extensive experiments on public datasets with various networks demonstrate that
GADT can be integrated with existing transferable attack methods, updating
their DA parameters effectively while retaining their AN formulation
strategies. Furthermore, GADT can be utilized in other black-box attack
scenarios, e.g., query-based attacks, offering a new avenue to enhance attacks
on real-world AI applications in both research and industrial contexts.


http://arxiv.org/abs/2410.19160
Adversarial Attacks on Large Language Models Using Regularized Relaxation. (98%)
Samuel Jacob Chacko; Sajib Biswas; Chashi Mahiul Islam; Fatema Tabassum Liza; Xiuwen Liu
  As powerful Large Language Models (LLMs) are now widely used for numerous
practical applications, their safety is of critical importance. While alignment
techniques have significantly improved overall safety, LLMs remain vulnerable
to carefully crafted adversarial inputs. Consequently, adversarial attack
methods are extensively used to study and understand these vulnerabilities.
However, current attack methods face significant limitations. Those relying on
optimizing discrete tokens suffer from limited efficiency, while continuous
optimization techniques fail to generate valid tokens from the model's
vocabulary, rendering them impractical for real-world applications. In this
paper, we propose a novel technique for adversarial attacks that overcomes
these limitations by leveraging regularized gradients with continuous
optimization methods. Our approach is two orders of magnitude faster than the
state-of-the-art greedy coordinate gradient-based method, significantly
improving the attack success rate on aligned language models. Moreover, it
generates valid tokens, addressing a fundamental limitation of existing
continuous optimization methods. We demonstrate the effectiveness of our attack
on five state-of-the-art LLMs using four datasets.


http://arxiv.org/abs/2410.18469
Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities. (88%)
Chung-En Sun; Xiaodong Liu; Weiwei Yang; Tsui-Wei Weng; Hao Cheng; Aidan San; Michel Galley; Jianfeng Gao
  Recent research has shown that Large Language Models (LLMs) are vulnerable to
automated jailbreak attacks, where adversarial suffixes crafted by algorithms
appended to harmful queries bypass safety alignment and trigger unintended
responses. Current methods for generating these suffixes are computationally
expensive and have low Attack Success Rates (ASR), especially against
well-aligned models like Llama2 and Llama3. To overcome these limitations, we
introduce ADV-LLM, an iterative self-tuning process that crafts adversarial
LLMs with enhanced jailbreak ability. Our framework significantly reduces the
computational cost of generating adversarial suffixes while achieving nearly
100\% ASR on various open-source LLMs. Moreover, it exhibits strong attack
transferability to closed-source models, achieving 99% ASR on GPT-3.5 and 49%
ASR on GPT-4, despite being optimized solely on Llama3. Beyond improving
jailbreak ability, ADV-LLM provides valuable insights for future safety
alignment research through its ability to generate large datasets for studying
LLM safety.


http://arxiv.org/abs/2410.19230
Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors. (68%)
Tianchun Wang; Yuanzhou Chen; Zichuan Liu; Zhanwen Chen; Haifeng Chen; Xiang Zhang; Wei Cheng
  The advent of large language models (LLMs) has revolutionized the field of
text generation, producing outputs that closely mimic human-like writing.
Although academic and industrial institutions have developed detectors to
prevent the malicious usage of LLM-generated texts, other research has doubt
about the robustness of these systems. To stress test these detectors, we
introduce a proxy-attack strategy that effortlessly compromises LLMs, causing
them to produce outputs that align with human-written text and mislead
detection systems. Our method attacks the source model by leveraging a
reinforcement learning (RL) fine-tuned humanized small language model (SLM) in
the decoding phase. Through an in-depth analysis, we demonstrate that our
attack strategy is capable of generating responses that are indistinguishable
to detectors, preventing them from differentiating between machine-generated
and human-written text. We conduct systematic evaluations on extensive datasets
using proxy-attacked open-source models, including Llama2-13B, Llama3-70B, and
Mixtral-8*7B in both white- and black-box settings. Our findings show that the
proxy-attack strategy effectively deceives the leading detectors, resulting in
an average AUROC drop of 70.4% across multiple datasets, with a maximum drop of
90.3% on a single dataset. Furthermore, in cross-discipline scenarios, our
strategy also bypasses these detectors, leading to a significant relative
decrease of up to 90.9%, while in cross-language scenario, the drop reaches
91.3%. Despite our proxy-attack strategy successfully bypassing the detectors
with such significant relative drops, we find that the generation quality of
the attacked models remains preserved, even within a modest utility budget,
when compared to the text produced by the original, unattacked source model.


http://arxiv.org/abs/2410.18556
Complexity Matters: Effective Dimensionality as a Measure for Adversarial Robustness. (33%)
David Khachaturov; Robert Mullins
  Quantifying robustness in a single measure for the purposes of model
selection, development of adversarial training methods, and anticipating trends
has so far been elusive. The simplest metric to consider is the number of
trainable parameters in a model but this has previously been shown to be
insufficient at explaining robustness properties. A variety of other metrics,
such as ones based on boundary thickness and gradient flatness have been
proposed but have been shown to be inadequate proxies for robustness.
  In this work, we investigate the relationship between a model's effective
dimensionality, which can be thought of as model complexity, and its robustness
properties. We run experiments on commercial-scale models that are often used
in real-world environments such as YOLO and ResNet. We reveal a near-linear
inverse relationship between effective dimensionality and adversarial
robustness, that is models with a lower dimensionality exhibit better
robustness. We investigate the effect of a variety of adversarial training
methods on effective dimensionality and find the same inverse linear
relationship present, suggesting that effective dimensionality can serve as a
useful criterion for model selection and robustness evaluation, providing a
more nuanced and effective metric than parameter count or previously-tested
measures.


http://arxiv.org/abs/2410.18775
Robust Watermarking Using Generative Priors Against Image Editing: From Benchmarking to Advances. (11%)
Shilin Lu; Zihan Zhou; Jiayou Lu; Yuanzhi Zhu; Adams Wai-Kin Kong
  Current image watermarking methods are vulnerable to advanced image editing
techniques enabled by large-scale text-to-image models. These models can
distort embedded watermarks during editing, posing significant challenges to
copyright protection. In this work, we introduce W-Bench, the first
comprehensive benchmark designed to evaluate the robustness of watermarking
methods against a wide range of image editing techniques, including image
regeneration, global editing, local editing, and image-to-video generation.
Through extensive evaluations of eleven representative watermarking methods
against prevalent editing techniques, we demonstrate that most methods fail to
detect watermarks after such edits. To address this limitation, we propose
VINE, a watermarking method that significantly enhances robustness against
various image editing techniques while maintaining high image quality. Our
approach involves two key innovations: (1) we analyze the frequency
characteristics of image editing and identify that blurring distortions exhibit
similar frequency properties, which allows us to use them as surrogate attacks
during training to bolster watermark robustness; (2) we leverage a large-scale
pretrained diffusion model SDXL-Turbo, adapting it for the watermarking task to
achieve more imperceptible and robust watermark embedding. Experimental results
show that our method achieves outstanding watermarking performance under
various image editing techniques, outperforming existing methods in both image
quality and robustness. Code is available at https://github.com/Shilin-LU/VINE.


http://arxiv.org/abs/2410.18215
Advancing NLP Security by Leveraging LLMs as Adversarial Engines. (98%)
Sudarshan Srinivasan; Maria Mahbub; Amir Sadovnik
  This position paper proposes a novel approach to advancing NLP security by
leveraging Large Language Models (LLMs) as engines for generating diverse
adversarial attacks. Building upon recent work demonstrating LLMs'
effectiveness in creating word-level adversarial examples, we argue for
expanding this concept to encompass a broader range of attack types, including
adversarial patches, universal perturbations, and targeted attacks. We posit
that LLMs' sophisticated language understanding and generation capabilities can
produce more effective, semantically coherent, and human-like adversarial
examples across various domains and classifier architectures. This paradigm
shift in adversarial NLP has far-reaching implications, potentially enhancing
model robustness, uncovering new vulnerabilities, and driving innovation in
defense mechanisms. By exploring this new frontier, we aim to contribute to the
development of more secure, reliable, and trustworthy NLP systems for critical
applications.


http://arxiv.org/abs/2410.18267
Backdoor in Seconds: Unlocking Vulnerabilities in Large Pre-trained Models via Model Editing. (93%)
Dongliang Guo; Mengxuan Hu; Zihan Guan; Junfeng Guo; Thomas Hartvigsen; Sheng Li
  Large pre-trained models have achieved notable success across a range of
downstream tasks. However, recent research shows that a type of adversarial
attack ($\textit{i.e.,}$ backdoor attack) can manipulate the behavior of
machine learning models through contaminating their training dataset, posing
significant threat in the real-world application of large pre-trained model,
especially for those customized models. Therefore, addressing the unique
challenges for exploring vulnerability of pre-trained models is of paramount
importance. Through empirical studies on the capability for performing backdoor
attack in large pre-trained models ($\textit{e.g.,}$ ViT), we find the
following unique challenges of attacking large pre-trained models: 1) the
inability to manipulate or even access large training datasets, and 2) the
substantial computational resources required for training or fine-tuning these
models. To address these challenges, we establish new standards for an
effective and feasible backdoor attack in the context of large pre-trained
models. In line with these standards, we introduce our EDT model, an
\textbf{E}fficient, \textbf{D}ata-free, \textbf{T}raining-free backdoor attack
method. Inspired by model editing techniques, EDT injects an editing-based
lightweight codebook into the backdoor of large pre-trained models, which
replaces the embedding of the poisoned image with the target image without
poisoning the training dataset or training the victim model. Our experiments,
conducted across various pre-trained models such as ViT, CLIP, BLIP, and stable
diffusion, and on downstream tasks including image classification, image
captioning, and image generation, demonstrate the effectiveness of our method.
Our code is available in the supplementary material.


http://arxiv.org/abs/2410.19863
Breaking the Illusion: Real-world Challenges for Adversarial Patches in Object Detection. (70%)
Jakob Shack; Katarina Petrovic; Olga Saukh
  Adversarial attacks pose a significant threat to the robustness and
reliability of machine learning systems, particularly in computer vision
applications. This study investigates the performance of adversarial patches
for the YOLO object detection network in the physical world. Two attacks were
tested: a patch designed to be placed anywhere within the scene - global patch,
and another patch intended to partially overlap with specific object targeted
for removal from detection - local patch. Various factors such as patch size,
position, rotation, brightness, and hue were analyzed to understand their
impact on the effectiveness of the adversarial patches. The results reveal a
notable dependency on these parameters, highlighting the challenges in
maintaining attack efficacy in real-world conditions. Learning to align
digitally applied transformation parameters with those measured in the real
world still results in up to a 64\% discrepancy in patch performance. These
findings underscore the importance of understanding environmental influences on
adversarial attacks, which can inform the development of more robust defenses
for practical machine learning applications.


http://arxiv.org/abs/2410.17910
Slot: Provenance-Driven APT Detection through Graph Reinforcement Learning. (16%)
Wei Qiao; Yebo Feng; Teng Li; Zhuo Ma; Yulong Shen; JianFeng Ma; Yang Liu
  Advanced Persistent Threats (APTs) represent sophisticated cyberattacks
characterized by their ability to remain undetected within the victim system
for extended periods, aiming to exfiltrate sensitive data or disrupt
operations. Existing detection approaches often struggle to effectively
identify these complex threats, construct the attack chain for defense
facilitation, or resist adversarial attacks. To overcome these challenges, we
propose Slot, an advanced APT detection approach based on provenance graphs and
graph reinforcement learning. Slot excels in uncovering multi-level hidden
relationships, such as causal, contextual, and indirect connections, among
system behaviors through provenance graph mining. By pioneering the integration
of graph reinforcement learning, Slot dynamically adapts to new user activities
and evolving attack strategies, enhancing its resilience against adversarial
attacks. Additionally, Slot automatically constructs the attack chain according
to detected attacks with clustering algorithms, providing precise
identification of attack paths and facilitating the development of defense
strategies. Evaluations with real-world datasets demonstrate Slot's outstanding
accuracy, efficiency, adaptability, and robustness in APT detection, with most
metrics surpassing state-of-the-art methods. Additionally, case studies
conducted to assess Slot's effectiveness in supporting APT defense further
establish it as a practical and reliable tool for cybersecurity protection.


http://arxiv.org/abs/2410.17922
Guide for Defense (G4D): Dynamic Guidance for Robust and Balanced Defense in Large Language Models. (9%)
He Cao; Weidi Luo; Yu Wang; Zijing Liu; Bing Feng; Yuan Yao; Yu Li
  With the extensive deployment of Large Language Models (LLMs), ensuring their
safety has become increasingly critical. However, existing defense methods
often struggle with two key issues: (i) inadequate defense capabilities,
particularly in domain-specific scenarios like chemistry, where a lack of
specialized knowledge can lead to the generation of harmful responses to
malicious queries. (ii) over-defensiveness, which compromises the general
utility and responsiveness of LLMs. To mitigate these issues, we introduce a
multi-agents-based defense framework, Guide for Defense (G4D), which leverages
accurate external information to provide an unbiased summary of user intentions
and analytically grounded safety response guidance. Extensive experiments on
popular jailbreak attacks and benign datasets show that our G4D can enhance
LLM's robustness against jailbreak attacks on general and domain-specific
scenarios without compromising the model's general functionality.


http://arxiv.org/abs/2410.17573
Securing Federated Learning against Backdoor Threats with Foundation Model Integration. (3%)
Xiaohuan Bi; Xi Li
  Federated Learning (FL) enables decentralized model training while preserving
privacy. Recently, the integration of Foundation Models (FMs) into FL has
enhanced performance but introduced a novel backdoor attack mechanism.
Attackers can exploit FM vulnerabilities to embed backdoors into synthetic data
generated by FMs. During global model fusion, these backdoors are transferred
to the global model through compromised synthetic data, subsequently infecting
all client models. Existing FL backdoor defenses are ineffective against this
novel attack due to its fundamentally different mechanism compared to classic
ones. In this work, we propose a novel data-free defense strategy that
addresses both classic and novel backdoor attacks in FL. The shared attack
pattern lies in the abnormal activations within the hidden feature space during
model aggregation. Hence, we propose to constrain internal activations to
remain within reasonable ranges, effectively mitigating attacks while
preserving model functionality. The activation constraints are optimized using
synthetic data alongside FL training. Extensive experiments demonstrate its
effectiveness against both novel and classic backdoor attacks, outperforming
existing defenses.


http://arxiv.org/abs/2410.18210
Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks. (2%)
Samuele Poppi; Zheng-Xin Yong; Yifei He; Bobbie Chern; Han Zhao; Aobo Yang; Jianfeng Chi
  Recent advancements in Large Language Models (LLMs) have sparked widespread
concerns about their safety. Recent work demonstrates that safety alignment of
LLMs can be easily removed by fine-tuning with a few adversarially chosen
instruction-following examples, i.e., fine-tuning attacks. We take a further
step to understand fine-tuning attacks in multilingual LLMs. We first discover
cross-lingual generalization of fine-tuning attacks: using a few adversarially
chosen instruction-following examples in one language, multilingual LLMs can
also be easily compromised (e.g., multilingual LLMs fail to refuse harmful
prompts in other languages). Motivated by this finding, we hypothesize that
safety-related information is language-agnostic and propose a new method termed
Safety Information Localization (SIL) to identify the safety-related
information in the model parameter space. Through SIL, we validate this
hypothesis and find that only changing 20% of weight parameters in fine-tuning
attacks can break safety alignment across all languages. Furthermore, we
provide evidence to the alternative pathways hypothesis for why freezing
safety-related parameters does not prevent fine-tuning attacks, and we
demonstrate that our attack vector can still jailbreak LLMs adapted to new
languages.


http://arxiv.org/abs/2410.18312
Countering Autonomous Cyber Threats. (2%)
Kade M. Heckel; Adrian Weller
  With the capability to write convincing and fluent natural language and
generate code, Foundation Models present dual-use concerns broadly and within
the cyber domain specifically. Generative AI has already begun to impact
cyberspace through a broad illicit marketplace for assisting malware
development and social engineering attacks through hundreds of
malicious-AI-as-a-services tools. More alarming is that recent research has
shown the potential for these advanced models to inform or independently
execute offensive cyberspace operations. However, these previous investigations
primarily focused on the threats posed by proprietary models due to the until
recent lack of strong open-weight model and additionally leave the impacts of
network defenses or potential countermeasures unexplored. Critically,
understanding the aptitude of downloadable models to function as offensive
cyber agents is vital given that they are far more difficult to govern and
prevent their misuse. As such, this work evaluates several state-of-the-art FMs
on their ability to compromise machines in an isolated network and investigates
defensive mechanisms to defeat such AI-powered attacks. Using target machines
from a commercial provider, the most recently released downloadable models are
found to be on par with a leading proprietary model at conducting simple cyber
attacks with common hacking tools against known vulnerabilities. To mitigate
such LLM-powered threats, defensive prompt injection (DPI) payloads for
disrupting the malicious cyber agent's workflow are demonstrated to be
effective. From these results, the implications for AI safety and governance
with respect to cybersecurity is analyzed.


http://arxiv.org/abs/2410.17628
Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric. (1%)
Baiyuan Chen
  Robustness is a critical aspect of machine learning models. Existing
robustness evaluation approaches often lack theoretical generality or rely
heavily on empirical assessments, limiting insights into the structural factors
contributing to robustness. Moreover, theoretical robustness analysis is not
applicable for direct comparisons between models. To address these challenges,
we propose $\textit{TopoLip}$, a metric based on layer-wise analysis that
bridges topological data analysis and Lipschitz continuity for robustness
evaluation. TopoLip provides a unified framework for both theoretical and
empirical robustness comparisons across different architectures or
configurations, and it reveals how model parameters influence the robustness of
models. Using TopoLip, we demonstrate that attention-based models typically
exhibit smoother transformations and greater robustness compared to
convolution-based models, as validated through theoretical analysis and
adversarial tasks. Our findings establish a connection between architectural
design, robustness, and topological properties.


http://arxiv.org/abs/2410.17442
Detecting Adversarial Examples. (99%)
Furkan Mumcu; Yasin Yilmaz
  Deep Neural Networks (DNNs) have been shown to be vulnerable to adversarial
examples. While numerous successful adversarial attacks have been proposed,
defenses against these attacks remain relatively understudied. Existing defense
approaches either focus on negating the effects of perturbations caused by the
attacks to restore the DNNs' original predictions or use a secondary model to
detect adversarial examples. However, these methods often become ineffective
due to the continuous advancements in attack techniques. We propose a novel
universal and lightweight method to detect adversarial examples by analyzing
the layer outputs of DNNs. Our method trains a lightweight regression model
that predicts deeper-layer features from early-layer features, and uses the
prediction error to detect adversarial samples. Through theoretical
justification and extensive experiments, we demonstrate that our detection
method is highly effective, compatible with any DNN architecture, and
applicable across different domains, such as image, video, and audio.


http://arxiv.org/abs/2410.16805
Test-time Adversarial Defense with Opposite Adversarial Path and High Attack Time Cost. (98%)
Cheng-Han Yeh; Kuanchun Yu; Chun-Shien Lu
  Deep learning models are known to be vulnerable to adversarial attacks by
injecting sophisticated designed perturbations to input data. Training-time
defenses still exhibit a significant performance gap between natural accuracy
and robust accuracy. In this paper, we investigate a new test-time adversarial
defense method via diffusion-based recovery along opposite adversarial paths
(OAPs). We present a purifier that can be plugged into a pre-trained model to
resist adversarial attacks. Different from prior arts, the key idea is
excessive denoising or purification by integrating the opposite adversarial
direction with reverse diffusion to push the input image further toward the
opposite adversarial direction. For the first time, we also exemplify the
pitfall of conducting AutoAttack (Rand) for diffusion-based defense methods.
Through the lens of time complexity, we examine the trade-off between the
effectiveness of adaptive attack and its computation complexity against our
defense. Experimental evaluation along with time cost analysis verifies the
effectiveness of the proposed method.


http://arxiv.org/abs/2410.17401
AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents. (97%)
Chejian Xu; Mintong Kang; Jiawei Zhang; Zeyi Liao; Lingbo Mo; Mengqi Yuan; Huan Sun; Bo Li
  Vision Language Models (VLMs) have revolutionized the creation of generalist
web agents, empowering them to autonomously complete diverse tasks on
real-world websites, thereby boosting human efficiency and productivity.
However, despite their remarkable capabilities, the safety and security of
these agents against malicious attacks remain critically underexplored, raising
significant concerns about their safe deployment. To uncover and exploit such
vulnerabilities in web agents, we provide AdvWeb, a novel black-box attack
framework designed against web agents. AdvWeb trains an adversarial prompter
model that generates and injects adversarial prompts into web pages, misleading
web agents into executing targeted adversarial actions such as inappropriate
stock purchases or incorrect bank transactions, actions that could lead to
severe real-world consequences. With only black-box access to the web agent, we
train and optimize the adversarial prompter model using DPO, leveraging both
successful and failed attack strings against the target agent. Unlike prior
approaches, our adversarial string injection maintains stealth and control: (1)
the appearance of the website remains unchanged before and after the attack,
making it nearly impossible for users to detect tampering, and (2) attackers
can modify specific substrings within the generated adversarial string to
seamlessly change the attack objective (e.g., purchasing stocks from a
different company), enhancing attack flexibility and efficiency. We conduct
extensive evaluations, demonstrating that AdvWeb achieves high success rates in
attacking SOTA GPT-4V-based VLM agent across various web tasks. Our findings
expose critical vulnerabilities in current LLM/VLM-based agents, emphasizing
the urgent need for developing more reliable web agents and effective defenses.
Our code and data are available at https://ai-secure.github.io/AdvWeb/ .


http://arxiv.org/abs/2410.17431
Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks. (67%)
Tao Li; Henger Li; Yunian Pan; Tianyi Xu; Zizhan Zheng; Quanyan Zhu
  Federated learning (FL) is susceptible to a range of security threats.
Although various defense mechanisms have been proposed, they are typically
non-adaptive and tailored to specific types of attacks, leaving them
insufficient in the face of multiple uncertain, unknown, and adaptive attacks
employing diverse strategies. This work formulates adversarial federated
learning under a mixture of various attacks as a Bayesian Stackelberg Markov
game, based on which we propose the meta-Stackelberg defense composed of
pre-training and online adaptation. {The gist is to simulate strong attack
behavior using reinforcement learning (RL-based attacks) in pre-training and
then design meta-RL-based defense to combat diverse and adaptive attacks.} We
develop an efficient meta-learning approach to solve the game, leading to a
robust and adaptive FL defense. Theoretically, our meta-learning algorithm,
meta-Stackelberg learning, provably converges to the first-order
$\varepsilon$-meta-equilibrium point in $O(\varepsilon^{-2})$ gradient
iterations with $O(\varepsilon^{-4})$ samples per iteration. Experiments show
that our meta-Stackelberg framework performs superbly against strong model
poisoning and backdoor attacks of uncertain and unknown types.


http://arxiv.org/abs/2410.17052
On the Vulnerability of Text Sanitization. (8%)
Meng Tong; Kejiang Chen; Xiaojian Yuang; Jiayang Liu; Weiming Zhang; Nenghai Yu; Jie Zhang
  Text sanitization, which employs differential privacy to replace sensitive
tokens with new ones, represents a significant technique for privacy
protection. Typically, its performance in preserving privacy is evaluated by
measuring the attack success rate (ASR) of reconstruction attacks, where
attackers attempt to recover the original tokens from the sanitized ones.
However, current reconstruction attacks on text sanitization are developed
empirically, making it challenging to accurately assess the effectiveness of
sanitization. In this paper, we aim to provide a more accurate evaluation of
sanitization effectiveness. Inspired by the works of Palamidessi et al., we
implement theoretically optimal reconstruction attacks targeting text
sanitization. We derive their bounds on ASR as benchmarks for evaluating
sanitization performance. For real-world applications, we propose two practical
reconstruction attacks based on these theoretical findings. Our experimental
results underscore the necessity of reassessing these overlooked risks.
Notably, one of our attacks achieves a 46.4% improvement in ASR over the
state-of-the-art baseline, with a privacy budget of epsilon=4.0 on the SST-2
dataset. Our code is available at:
https://github.com/mengtong0110/On-the-Vulnerability-of-Text-Sanitization.


http://arxiv.org/abs/2410.17222
Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods. (5%)
Tsachi Blau; Moshe Kimhi; Yonatan Belinkov; Alexander Bronstein; Chaim Baskin
  Fine-tuning Large Language Models (LLMs) typically involves updating at least
a few billions of parameters. A more parameter-efficient approach is Prompt
Tuning (PT), which updates only a few learnable tokens, and differently,
In-Context Learning (ICL) adapts the model to a new task by simply including
examples in the input without any training. When applying optimization-based
methods, such as fine-tuning and PT for few-shot learning, the model is
specifically adapted to the small set of training examples, whereas ICL leaves
the model unchanged. This distinction makes traditional learning methods more
prone to overfitting; in contrast, ICL is less sensitive to the few-shot
scenario. While ICL is not prone to overfitting, it does not fully extract the
information that exists in the training examples. This work introduces
Context-aware Prompt Tuning (CPT), a method inspired by ICL, PT, and
adversarial attacks. We build on the ICL strategy of concatenating examples
before the input, but we extend this by PT-like learning, refining the context
embedding through iterative optimization to extract deeper insights from the
training examples. We carefully modify specific context tokens, considering the
unique structure of input and output formats. Inspired by adversarial attacks,
we adjust the input based on the labels present in the context, focusing on
minimizing, rather than maximizing, the loss. Moreover, we apply a projected
gradient descent algorithm to keep token embeddings close to their original
values, under the assumption that the user-provided data is inherently
valuable. Our method has been shown to achieve superior accuracy across
multiple classification tasks using various LLM models.


http://arxiv.org/abs/2410.16802
Evaluating the Effectiveness of Attack-Agnostic Features for Morphing Attack Detection. (4%)
Laurent Colbois; Sébastien Marcel
  Morphing attacks have diversified significantly over the past years, with new
methods based on generative adversarial networks (GANs) and diffusion models
posing substantial threats to face recognition systems. Recent research has
demonstrated the effectiveness of features extracted from large vision models
pretrained on bonafide data only (attack-agnostic features) for detecting deep
generative images. Building on this, we investigate the potential of these
image representations for morphing attack detection (MAD). We develop
supervised detectors by training a simple binary linear SVM on the extracted
features and one-class detectors by modeling the distribution of bonafide
features with a Gaussian Mixture Model (GMM). Our method is evaluated across a
comprehensive set of attacks and various scenarios, including generalization to
unseen attacks, different source datasets, and print-scan data. Our results
indicate that attack-agnostic features can effectively detect morphing attacks,
outperforming traditional supervised and one-class detectors from the
literature in most scenarios. Additionally, we provide insights into the
strengths and limitations of each considered representation and discuss
potential future research directions to further enhance the robustness and
generalizability of our approach.


http://arxiv.org/abs/2410.17492
BadFair: Backdoored Fairness Attacks with Group-conditioned Triggers. (2%)
Jiaqi Xue; Qian Lou; Mengxin Zheng
  Attacking fairness is crucial because compromised models can introduce biased
outcomes, undermining trust and amplifying inequalities in sensitive
applications like hiring, healthcare, and law enforcement. This highlights the
urgent need to understand how fairness mechanisms can be exploited and to
develop defenses that ensure both fairness and robustness. We introduce
BadFair, a novel backdoored fairness attack methodology. BadFair stealthily
crafts a model that operates with accuracy and fairness under regular
conditions but, when activated by certain triggers, discriminates and produces
incorrect results for specific groups. This type of attack is particularly
stealthy and dangerous, as it circumvents existing fairness detection methods,
maintaining an appearance of fairness in normal use. Our findings reveal that
BadFair achieves a more than 85% attack success rate in attacks aimed at target
groups on average while only incurring a minimal accuracy loss. Moreover, it
consistently exhibits a significant discrimination score, distinguishing
between pre-defined target and non-target attacked groups across various
datasets and models.


http://arxiv.org/abs/2410.17402
Invisible Manipulation Deep Reinforcement Learning Enhanced Stealthy Attacks on Battery Energy Management Systems. (1%)
Qi Xiao; Lidong Song; Jongha Woo; Rongxing Hu; Bei Xu; Ning Lu
  This paper introduces "invisible manipulation," an innovative cyber-attack
mechanism achieved through strategically timed stealthy false data injection
attacks (SFDIAs). By stealthily manipulating measurements of a critical asset
prior to the target time period, the attacker can subtly guide the engineering
system toward a predetermined operational state without detection. Using the
battery energy management system (BEMS) as a case study, we employ deep
reinforcement learning (DRL) to generate synthetic measurements, such as
battery voltage and current, that align closely with actual measurements. These
synthetic measurements, falling within the acceptable error margin of
residual-based bad data detection algorithm provided by state estimation, can
evade detection and mislead Extended Kalman-filter-based State of Charge
estimation. Subsequently, considering the deceptive data as valid inputs, the
BEMS will operate the BESS towards the attacker desired operational states when
the targeted time period come. The use of the DRL-based scheme allows us to
covert an online optimization problem into an offline training process, thereby
alleviating the computational burden for real-time implementation.
Comprehensive testing on a high-fidelity microgrid real-time simulation testbed
validates the effectiveness and adaptability of the proposed methods in
achieving different attack objectives.


http://arxiv.org/abs/2410.17103
A Hybrid Simulation of DNN-based Gray Box Models. (1%)
Aayushya Agarwal; Yihan Ruan; Larry Pileggi
  Simulation is vital for engineering disciplines, as it enables the prediction
and design of physical systems. However, the computational challenges inherent
to large-scale simulations often arise from complex device models featuring
high degrees of nonlinearities or hidden physical behaviors not captured by
first principles. Gray-box models combine deep neural networks (DNNs) with
physics-based models to address the computational challenges in modeling
physical systems. A well-crafted gray box model capitalizes on the
interpretability and accuracy of a physical model while incorporating DNNs to
capture hidden physical behaviors and mitigate computational load associated
with highly nonlinear components. Previously, gray box models have been
constructed by defining an explicit combination of physics-based and DNN models
to represent the behavior of sub-systems; however this cannot represent the
coupled interactions within physical systems. We explore an implicit gray box
model, where both DNNs and physical equations share a common set of
state-variables. While this approach captures coupled interactions at the
boundary of DNN and physics-based models, simulating the implicit gray box
model remains an open-ended problem. In this work, we introduce a new hybrid
simulation that integrates DNNs into the numerical solvers of simulation
engines to fully simulate implicit gray box models of large physical systems.
This is accomplished by backpropagating through the DNN to calculate Jacobian
values during each iteration of the numerical method. The hybrid simulation
improves the accuracy and runtime compared to physics-based simulation and
enables reusable DNN models with lower data requirements. We explore the
advantages of this approach as compared to physics-based, black box, and other
gray box methods for simulating the steady-state and transient behavior of
power systems.


http://arxiv.org/abs/2410.15889
Model Mimic Attack: Knowledge Distillation for Provably Transferable Adversarial Examples. (99%)
Kirill Lukyanov; Andrew Perminov; Denis Turdakov; Mikhail Pautov
  The vulnerability of artificial neural networks to adversarial perturbations
in the black-box setting is widely studied in the literature. The majority of
attack methods to construct these perturbations suffer from an impractically
large number of queries required to find an adversarial example. In this work,
we focus on knowledge distillation as an approach to conduct transfer-based
black-box adversarial attacks and propose an iterative training of the
surrogate model on an expanding dataset. This work is the first, to our
knowledge, to provide provable guarantees on the success of knowledge
distillation-based attack on classification neural networks: we prove that if
the student model has enough learning capabilities, the attack on the teacher
model is guaranteed to be found within the finite number of distillation
iterations.


http://arxiv.org/abs/2410.16579
Conflict-Aware Adversarial Training. (70%)
Zhiyu Xue; Haohan Wang; Yao Qin; Ramtin Pedarsani
  Adversarial training is the most effective method to obtain adversarial
robustness for deep neural networks by directly involving adversarial samples
in the training procedure. To obtain an accurate and robust model, the
weighted-average method is applied to optimize standard loss and adversarial
loss simultaneously. In this paper, we argue that the weighted-average method
does not provide the best tradeoff for the standard performance and adversarial
robustness. We argue that the failure of the weighted-average method is due to
the conflict between the gradients derived from standard and adversarial loss,
and further demonstrate such a conflict increases with attack budget
theoretically and practically. To alleviate this problem, we propose a new
trade-off paradigm for adversarial training with a conflict-aware factor for
the convex combination of standard and adversarial loss, named
\textbf{Conflict-Aware Adversarial Training~(CA-AT)}. Comprehensive
experimental results show that CA-AT consistently offers a superior trade-off
between standard performance and adversarial robustness under the settings of
adversarial training from scratch and parameter-efficient finetuning.


http://arxiv.org/abs/2410.16449
Robust Feature Learning for Multi-Index Models in High Dimensions. (68%)
Alireza Mousavi-Hosseini; Adel Javanmard; Murat A. Erdogdu
  Recently, there have been numerous studies on feature learning with neural
networks, specifically on learning single- and multi-index models where the
target is a function of a low-dimensional projection of the input. Prior works
have shown that in high dimensions, the majority of the compute and data
resources are spent on recovering the low-dimensional projection; once this
subspace is recovered, the remainder of the target can be learned independently
of the ambient dimension. However, implications of feature learning in
adversarial settings remain unexplored. In this work, we take the first steps
towards understanding adversarially robust feature learning with neural
networks. Specifically, we prove that the hidden directions of a multi-index
model offer a Bayes optimal low-dimensional projection for robustness against
$\ell_2$-bounded adversarial perturbations under the squared loss, assuming
that the multi-index coordinates are statistically independent from the rest of
the coordinates. Therefore, robust learning can be achieved by first performing
standard feature learning, then robustly tuning a linear readout layer on top
of the standard representations. In particular, we show that adversarially
robust learning is just as easy as standard learning, in the sense that the
additional number of samples needed to robustly learn multi-index models when
compared to standard learning, does not depend on dimensionality.


http://arxiv.org/abs/2410.16657
Dual-Model Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting. (16%)
Bao Q. Tran; Viet Nguyen; Anh Tran; Toan Tran
  Diffusion models have demonstrated remarkable capabilities in image
synthesis, but their recently proven vulnerability to Membership Inference
Attacks (MIAs) poses a critical privacy concern. This paper introduces two
novel and efficient approaches (DualMD and DistillMD) to protect diffusion
models against MIAs while maintaining high utility. Both methods are based on
training two separate diffusion models on disjoint subsets of the original
dataset. DualMD then employs a private inference pipeline that utilizes both
models. This strategy significantly reduces the risk of black-box MIAs by
limiting the information any single model contains about individual training
samples. The dual models can also generate "soft targets" to train a private
student model in DistillMD, enhancing privacy guarantees against all types of
MIAs. Extensive evaluations of DualMD and DistillMD against state-of-the-art
MIAs across various datasets in white-box and black-box settings demonstrate
their effectiveness in substantially reducing MIA success rates while
preserving competitive image generation performance. Notably, our experiments
reveal that DistillMD not only defends against MIAs but also mitigates model
memorization, indicating that both vulnerabilities stem from overfitting and
can be addressed simultaneously with our unified approach.


http://arxiv.org/abs/2410.16159
Metric as Transform: Exploring beyond Affine Transform for Interpretable Neural Network. (13%)
Suman Sapkota
  Artificial Neural Networks of varying architectures are generally paired with
affine transformation at the core. However, we find dot product neurons with
global influence less interpretable as compared to local influence of euclidean
distance (as used in Radial Basis Function Network). In this work, we explore
the generalization of dot product neurons to $l^p$-norm, metrics, and beyond.
We find that metrics as transform performs similarly to affine transform when
used in MultiLayer Perceptron or Convolutional Neural Network. Moreover, we
explore various properties of Metrics, compare it with Affine, and present
multiple cases where metrics seem to provide better interpretability. We
develop an interpretable local dictionary based Neural Networks and use it to
understand and reject adversarial examples.


http://arxiv.org/abs/2410.16222
A Realistic Threat Model for Large Language Model Jailbreaks. (11%)
Valentyn Boreiko; Alexander Panfilov; Vaclav Voracek; Matthias Hein; Jonas Geiping
  A plethora of jailbreaking attacks have been proposed to obtain harmful
responses from safety-tuned LLMs. In their original settings, these methods all
largely succeed in coercing the target output, but their attacks vary
substantially in fluency and computational effort. In this work, we propose a
unified threat model for the principled comparison of these methods. Our threat
model combines constraints in perplexity, measuring how far a jailbreak
deviates from natural text, and computational budget, in total FLOPs. For the
former, we build an N-gram model on 1T tokens, which, in contrast to
model-based perplexity, allows for an LLM-agnostic and inherently interpretable
evaluation. We adapt popular attacks to this new, realistic threat model, with
which we, for the first time, benchmark these attacks on equal footing. After a
rigorous comparison, we not only find attack success rates against safety-tuned
modern models to be lower than previously presented but also find that attacks
based on discrete optimization significantly outperform recent LLM-based
attacks. Being inherently interpretable, our threat model allows for a
comprehensive analysis and comparison of jailbreak attacks. We find that
effective attacks exploit and abuse infrequent N-grams, either selecting
N-grams absent from real-world text or rare ones, e.g. specific to code
datasets.


http://arxiv.org/abs/2410.16341
Vulnerabilities in Machine Learning-Based Voice Disorder Detection Systems. (11%)
Gianpaolo Perelli; Andrea Panzino; Roberto Casula; Marco Micheletto; Giulia Orrù; Gian Luca Marcialis
  The impact of voice disorders is becoming more widely acknowledged as a
public health issue. Several machine learning-based classifiers with the
potential to identify disorders have been used in recent studies to
differentiate between normal and pathological voices and sounds. In this paper,
we focus on analyzing the vulnerabilities of these systems by exploring the
possibility of attacks that can reverse classification and compromise their
reliability. Given the critical nature of personal health information,
understanding which types of attacks are effective is a necessary first step
toward improving the security of such systems. Starting from the original
audios, we implement various attack methods, including adversarial, evasion,
and pitching techniques, and evaluate how state-of-the-art disorder detection
models respond to them. Our findings identify the most effective attack
strategies, underscoring the need to address these vulnerabilities in
machine-learning systems used in the healthcare domain.


http://arxiv.org/abs/2410.16073
On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds. (5%)
Matteo Vilucchio; Nikolaos Tsilivis; Bruno Loureiro; Julia Kempe
  Regularization, whether explicit in terms of a penalty in the loss or
implicit in the choice of algorithm, is a cornerstone of modern machine
learning. Indeed, controlling the complexity of the model class is particularly
important when data is scarce, noisy or contaminated, as it translates a
statistical belief on the underlying structure of the data. This work
investigates the question of how to choose the regularization norm $\lVert
\cdot \rVert$ in the context of high-dimensional adversarial training for
binary classification. To this end, we first derive an exact asymptotic
description of the robust, regularized empirical risk minimizer for various
types of adversarial attacks and regularization norms (including non-$\ell_p$
norms). We complement this analysis with a uniform convergence analysis,
deriving bounds on the Rademacher Complexity for this class of problems.
Leveraging our theoretical results, we quantitatively characterize the
relationship between perturbation size and the optimal choice of $\lVert \cdot
\rVert$, confirming the intuition that, in the data scarce regime, the type of
regularization becomes increasingly important for adversarial training as
perturbations grow in size.


http://arxiv.org/abs/2410.15645
Boosting Jailbreak Transferability for Large Language Models. (1%)
Hanqing Liu; Lifeng Zhou; Huanqian Yan
  Large language models have drawn significant attention to the challenge of
safe alignment, especially regarding jailbreak attacks that circumvent security
measures to produce harmful content. To address the limitations of existing
methods like GCG, which perform well in single-model attacks but lack
transferability, we propose several enhancements, including a scenario
induction template, optimized suffix selection, and the integration of
re-suffix attack mechanism to reduce inconsistent outputs. Our approach has
shown superior performance in extensive experiments across various benchmarks,
achieving nearly 100% success rates in both attack execution and
transferability. Notably, our method has won the online first place in the
AISG-hosted Global Challenge for Safe and Secure LLMs.


http://arxiv.org/abs/2410.16121
Extracting Spatiotemporal Data from Gradients with Large Language Models. (1%)
Lele Zheng; Yang Cao; Renhe Jiang; Kenjiro Taura; Yulong Shen; Sheng Li; Masatoshi Yoshikawa
  Recent works show that sensitive user data can be reconstructed from gradient
updates, breaking the key privacy promise of federated learning. While success
was demonstrated primarily on image data, these methods do not directly
transfer to other domains, such as spatiotemporal data. To understand privacy
risks in spatiotemporal federated learning, we first propose Spatiotemporal
Gradient Inversion Attack (ST-GIA), a gradient attack algorithm tailored to
spatiotemporal data that successfully reconstructs the original location from
gradients. Furthermore, the absence of priors in attacks on spatiotemporal data
has hindered the accurate reconstruction of real client data. To address this
limitation, we propose ST-GIA+, which utilizes an auxiliary language model to
guide the search for potential locations, thereby successfully reconstructing
the original data from gradients. In addition, we design an adaptive defense
strategy to mitigate gradient inversion attacks in spatiotemporal federated
learning. By dynamically adjusting the perturbation levels, we can offer
tailored protection for varying rounds of training data, thereby achieving a
better trade-off between privacy and utility than current state-of-the-art
methods. Through intensive experimental analysis on three real-world datasets,
we reveal that the proposed defense strategy can well preserve the utility of
spatiotemporal federated learning with effective security protection.


http://arxiv.org/abs/2404.17358
Adversarial Consistency and the Uniqueness of the Adversarial Bayes Classifier. (1%)
Natalie S. Frank
Minimizing an adversarial surrogate risk is a common technique for learning robust classifiers. Prior work showed that convex surrogate losses are not statistically consistent in the adversarial context -- or in other words, a minimizing sequence of the adversarial surrogate risk will not necessarily minimize the adversarial classification error. We connect the consistency of adversarial surrogate losses to properties of minimizers to the adversarial classification risk, known as adversarial Bayes classifiers. Specifically, under reasonable distributional assumptions, a convex surrogate loss is statistically consistent for adversarial learning iff the adversarial Bayes classifier satisfies a certain notion of uniqueness.

http://arxiv.org/abs/2410.15409
PEAS: A Strategy for Crafting Transferable Adversarial Examples. (99%)
Bar Avraham; Yisroel Mirsky
  Black box attacks, where adversaries have limited knowledge of the target
model, pose a significant threat to machine learning systems. Adversarial
examples generated with a substitute model often suffer from limited
transferability to the target model. While recent work explores ranking
perturbations for improved success rates, these methods see only modest gains.
We propose a novel strategy called PEAS that can boost the transferability of
existing black box attacks. PEAS leverages the insight that samples which are
perceptually equivalent exhibit significant variability in their adversarial
transferability. Our approach first generates a set of images from an initial
sample via subtle augmentations. We then evaluate the transferability of
adversarial perturbations on these images using a set of substitute models.
Finally, the most transferable adversarial example is selected and used for the
attack. Our experiments show that PEAS can double the performance of existing
attacks, achieving a 2.5x improvement in attack success rates on average over
current ranking methods. We thoroughly evaluate PEAS on ImageNet and CIFAR-10,
analyze hyperparameter impacts, and provide an ablation study to isolate each
component's importance.


http://arxiv.org/abs/2410.15429
Efficient Model Extraction via Boundary Sampling. (96%)
Maor Biton Dor; Yisroel Mirsky
  This paper introduces a novel data-free model extraction attack that
significantly advances the current state-of-the-art in terms of efficiency,
accuracy, and effectiveness. Traditional black-box methods rely on using the
victim's model as an oracle to label a vast number of samples within
high-confidence areas. This approach not only requires an extensive number of
queries but also results in a less accurate and less transferable model. In
contrast, our method innovates by focusing on sampling low-confidence areas
(along the decision boundaries) and employing an evolutionary algorithm to
optimize the sampling process. These novel contributions allow for a dramatic
reduction in the number of queries needed by the attacker by a factor of 10x to
600x while simultaneously improving the accuracy of the stolen model. Moreover,
our approach improves boundary alignment, resulting in better transferability
of adversarial examples from the stolen model to the victim's model (increasing
the attack success rate from 60\% to 82\% on average). Finally, we accomplish
all of this with a strict black-box assumption on the victim, with no knowledge
of the target's architecture or dataset.
  We demonstrate our attack on three datasets with increasingly larger
resolutions and compare our performance to four state-of-the-art model
extraction attacks.


http://arxiv.org/abs/2410.15396
The Best Defense is a Good Offense: Countering LLM-Powered Cyberattacks. (76%)
Daniel Ayzenshteyn; Roy Weiss; Yisroel Mirsky
  As large language models (LLMs) continue to evolve, their potential use in
automating cyberattacks becomes increasingly likely. With capabilities such as
reconnaissance, exploitation, and command execution, LLMs could soon become
integral to autonomous cyber agents, capable of launching highly sophisticated
attacks. In this paper, we introduce novel defense strategies that exploit the
inherent vulnerabilities of attacking LLMs. By targeting weaknesses such as
biases, trust in input, memory limitations, and their tunnel-vision approach to
problem-solving, we develop techniques to mislead, delay, or neutralize these
autonomous agents. We evaluate our defenses under black-box conditions,
starting with single prompt-response scenarios and progressing to real-world
tests using custom-built CTF machines. Our results show defense success rates
of up to 90\%, demonstrating the effectiveness of turning LLM vulnerabilities
into defensive strategies against LLM-driven cyber threats.


http://arxiv.org/abs/2410.15362
Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Models. (45%)
Xiao Li; Zhuhong Li; Qiongxiu Li; Bingze Lee; Jinghao Cui; Xiaolin Hu
  Aligned Large Language Models (LLMs) have demonstrated remarkable performance
across various tasks. However, LLMs remain susceptible to jailbreak adversarial
attacks, where adversaries manipulate prompts to elicit malicious responses
that aligned LLMs should have avoided. Identifying these vulnerabilities is
crucial for understanding the inherent weaknesses of LLMs and preventing their
potential misuse. One pioneering work in jailbreaking is the GCG attack, a
discrete token optimization algorithm that seeks to find a suffix capable of
jailbreaking aligned LLMs. Despite the success of GCG, we find it suboptimal,
requiring significantly large computational costs, and the achieved
jailbreaking performance is limited. In this work, we propose Faster-GCG, an
efficient adversarial jailbreak method by delving deep into the design of GCG.
Experiments demonstrate that Faster-GCG can surpass the original GCG with only
1/10 of the computational cost, achieving significantly higher attack success
rates on various open-source aligned LLMs. In addition, We demonstrate that
Faster-GCG exhibits improved attack transferability when testing on
closed-sourced LLMs such as ChatGPT.


http://arxiv.org/abs/2410.15555
Bayesian Concept Bottleneck Models with LLM Priors. (1%)
Jean Feng; Avni Kothari; Luke Zier; Chandan Singh; Yan Shuo Tan
  Concept Bottleneck Models (CBMs) have been proposed as a compromise between
white-box and black-box models, aiming to achieve interpretability without
sacrificing accuracy. The standard training procedure for CBMs is to predefine
a candidate set of human-interpretable concepts, extract their values from the
training data, and identify a sparse subset as inputs to a transparent
prediction model. However, such approaches are often hampered by the tradeoff
between enumerating a sufficiently large set of concepts to include those that
are truly relevant versus controlling the cost of obtaining concept
extractions. This work investigates a novel approach that sidesteps these
challenges: BC-LLM iteratively searches over a potentially infinite set of
concepts within a Bayesian framework, in which Large Language Models (LLMs)
serve as both a concept extraction mechanism and prior. BC-LLM is broadly
applicable and multi-modal. Despite imperfections in LLMs, we prove that BC-LLM
can provide rigorous statistical inference and uncertainty quantification. In
experiments, it outperforms comparator methods including black-box models,
converges more rapidly towards relevant concepts and away from spuriously
correlated ones, and is more robust to out-of-distribution samples.


http://arxiv.org/abs/2410.15042
Adversarial Training: A Survey. (97%)
Mengnan Zhao; Lihe Zhang; Jingwen Ye; Huchuan Lu; Baocai Yin; Xinchao Wang
  Adversarial training (AT) refers to integrating adversarial examples --
inputs altered with imperceptible perturbations that can significantly impact
model predictions -- into the training process. Recent studies have
demonstrated the effectiveness of AT in improving the robustness of deep neural
networks against diverse adversarial attacks. However, a comprehensive overview
of these developments is still missing. This survey addresses this gap by
reviewing a broad range of recent and representative studies. Specifically, we
first describe the implementation procedures and practical applications of AT,
followed by a comprehensive review of AT techniques from three perspectives:
data enhancement, network design, and training configurations. Lastly, we
discuss common challenges in AT and propose several promising directions for
future research.


http://arxiv.org/abs/2410.15107
Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Models. (92%)
Seong-Il Park; Jay-Yoon Lee
  Retrieval Augmented Language Models (RALMs) have gained significant attention
for their ability to generate accurate answer and improve efficiency. However,
RALMs are inherently vulnerable to imperfect information due to their reliance
on the imperfect retriever or knowledge source. We identify three common
scenarios-unanswerable, adversarial, conflicting-where retrieved document sets
can confuse RALM with plausible real-world examples. We present the first
comprehensive investigation to assess how well RALMs detect and handle such
problematic scenarios. Among these scenarios, to systematically examine
adversarial robustness we propose a new adversarial attack method, Generative
model-based ADVersarial attack (GenADV) and a novel metric Robustness under
Additional Document (RAD). Our findings reveal that RALMs often fail to
identify the unanswerability or contradiction of a document set, which
frequently leads to hallucinations. Moreover, we show the addition of an
adversary significantly degrades RALM's performance, with the model becoming
even more vulnerable when the two scenarios overlap (adversarial+unanswerable).
Our research identifies critical areas for assessing and enhancing the
robustness of RALMs, laying the foundation for the development of more robust
models.


http://arxiv.org/abs/2410.15176
Beyond Pruning Criteria: The Dominant Role of Fine-Tuning and Adaptive Ratios in Neural Network Robustness. (81%)
Lincen Bai; Hedi Tabia; Raúl Santos-Rodríguez
  Deep neural networks (DNNs) excel in tasks like image recognition and natural
language processing, but their increasing complexity complicates deployment in
resource-constrained environments and increases susceptibility to adversarial
attacks. While traditional pruning methods reduce model size, they often
compromise the network's ability to withstand subtle perturbations. This paper
challenges the conventional emphasis on weight importance scoring as the
primary determinant of a pruned network's performance. Through extensive
analysis, including experiments conducted on CIFAR, Tiny-ImageNet, and various
network architectures, we demonstrate that effective fine-tuning plays a
dominant role in enhancing both performance and adversarial robustness, often
surpassing the impact of the chosen pruning criteria. To address this issue, we
introduce Module Robust Sensitivity, a novel metric that adaptively adjusts the
pruning ratio for each network layer based on its sensitivity to adversarial
perturbations. By integrating this metric into the pruning process, we develop
a stable algorithm that maintains accuracy and robustness simultaneously.
Experimental results show that our approach enables the practical deployment of
more robust and efficient neural networks.


http://arxiv.org/abs/2410.15236
Jailbreaking and Mitigation of Vulnerabilities in Large Language Models. (54%)
Benji Peng; Ziqian Bi; Qian Niu; Ming Liu; Pohsun Feng; Tianyang Wang; Lawrence K. Q. Yan; Yizhu Wen; Yichao Zhang; Caitlyn Heqi Yin
  Large Language Models (LLMs) have transformed artificial intelligence by
advancing natural language understanding and generation, enabling applications
across fields beyond healthcare, software engineering, and conversational
systems. Despite these advancements in the past few years, LLMs have shown
considerable vulnerabilities, particularly to prompt injection and jailbreaking
attacks. This review analyzes the state of research on these vulnerabilities
and presents available defense strategies. We roughly categorize attack
approaches into prompt-based, model-based, multimodal, and multilingual,
covering techniques such as adversarial prompting, backdoor injections, and
cross-modality exploits. We also review various defense mechanisms, including
prompt filtering, transformation, alignment techniques, multi-agent defenses,
and self-regulation, evaluating their strengths and shortcomings. We also
discuss key metrics and benchmarks used to assess LLM safety and robustness,
noting challenges like the quantification of attack success in interactive
contexts and biases in existing datasets. Identifying current research gaps, we
suggest future directions for resilient alignment strategies, advanced defenses
against evolving attacks, automation of jailbreak detection, and consideration
of ethical and societal impacts. This review emphasizes the need for continued
research and cooperation within the AI community to enhance LLM security and
ensure their safe deployment.


http://arxiv.org/abs/2410.15075
SLIC: Secure Learned Image Codec through Compressed Domain Watermarking to Defend Image Manipulation. (11%)
Chen-Hsiu Huang; Ja-Ling Wu
  The digital image manipulation and advancements in Generative AI, such as
Deepfake, has raised significant concerns regarding the authenticity of images
shared on social media. Traditional image forensic techniques, while helpful,
are often passive and insufficient against sophisticated tampering methods.
This paper introduces the Secure Learned Image Codec (SLIC), a novel active
approach to ensuring image authenticity through watermark embedding in the
compressed domain. SLIC leverages neural network-based compression to embed
watermarks as adversarial perturbations in the latent space, creating images
that degrade in quality upon re-compression if tampered with. This degradation
acts as a defense mechanism against unauthorized modifications. Our method
involves fine-tuning a neural encoder/decoder to balance watermark invisibility
with robustness, ensuring minimal quality loss for non-watermarked images.
Experimental results demonstrate SLIC's effectiveness in generating visible
artifacts in tampered images, thereby preventing their redistribution. This
work represents a significant step toward developing secure image codecs that
can be widely adopted to safeguard digital image integrity.


http://arxiv.org/abs/2410.15033
DynaMO: Protecting Mobile DL Models through Coupling Obfuscated DL Operators. (2%)
Mingyi Zhou; Xiang Gao; Xiao Chen; Chunyang Chen; John Grundy; Li Li
  Deploying DL models on mobile Apps has become ever-more popular. However,
existing studies show attackers can easily reverse-engineer mobile DL models in
Apps to steal intellectual property or generate effective attacks. A recent
approach, Model Obfuscation, has been proposed to defend against such reverse
engineering by obfuscating DL model representations, such as weights and
computational graphs, without affecting model performance. These existing model
obfuscation methods use static methods to obfuscate the model representation,
or they use half-dynamic methods but require users to restore the model
information through additional input arguments. However, these static methods
or half-dynamic methods cannot provide enough protection for on-device DL
models. Attackers can use dynamic analysis to mine the sensitive information in
the inference codes as the correct model information and intermediate results
must be recovered at runtime for static and half-dynamic obfuscation methods.
We assess the vulnerability of the existing obfuscation strategies using an
instrumentation method and tool, DLModelExplorer, that dynamically extracts
correct sensitive model information at runtime. Experiments show it achieves
very high attack performance. To defend against such attacks based on dynamic
instrumentation, we propose DynaMO, a Dynamic Model Obfuscation strategy
similar to Homomorphic Encryption. The obfuscation and recovery process can be
done through simple linear transformation for the weights of randomly coupled
eligible operators, which is a fully dynamic obfuscation strategy. Experiments
show that our proposed strategy can dramatically improve model security
compared with the existing obfuscation strategies, with only negligible
overheads for on-device models.


http://arxiv.org/abs/2410.14911
A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models. (99%)
Yuhan Liang; Yijun Li; Yumeng Niu; Qianhe Shen; Hangyu Liu
  The robustness of Vision-Language Models (VLMs) such as CLIP is critical for
their deployment in safety-critical applications like autonomous driving,
healthcare diagnostics, and security systems, where accurate interpretation of
visual and textual data is essential. However, these models are highly
susceptible to adversarial attacks, which can severely compromise their
performance and reliability in real-world scenarios. Previous methods have
primarily focused on improving robustness through adversarial training and
generating adversarial examples using models like FGSM, AutoAttack, and
DeepFool. However, these approaches often rely on strong assumptions, such as
fixed perturbation norms or predefined attack patterns, and involve high
computational complexity, making them challenging to implement in practical
settings. In this paper, we propose a novel adversarial training framework that
integrates multiple attack strategies and advanced machine learning techniques
to significantly enhance the robustness of VLMs against a broad range of
adversarial attacks. Experiments conducted on real-world datasets, including
CIFAR-10 and CIFAR-100, demonstrate that the proposed method significantly
enhances model robustness. The fine-tuned CLIP model achieved an accuracy of
43.5% on adversarially perturbed images, compared to only 4% for the baseline
model. The neural network model achieved a high accuracy of 98% in these
challenging classification tasks, while the XGBoost model reached a success
rate of 85.26% in prediction tasks.


http://arxiv.org/abs/2410.14881
Class-RAG: Content Moderation with Retrieval Augmented Generation. (76%)
Jianfa Chen; Emily Shen; Trupti Bavalatti; Xiaowen Lin; Yongkai Wang; Shuming Hu; Harihar Subramanyam; Ksheeraj Sai Vepuri; Ming Jiang; Ji Qi; Li Chen; Nan Jiang; Ankit Jain
  Robust content moderation classifiers are essential for the safety of
Generative AI systems. Content moderation, or safety classification, is
notoriously ambiguous: differences between safe and unsafe inputs are often
extremely subtle, making it difficult for classifiers (and indeed, even humans)
to properly distinguish violating vs. benign samples without further context or
explanation. Furthermore, as these technologies are deployed across various
applications and audiences, scaling risk discovery and mitigation through
continuous model fine-tuning becomes increasingly challenging and costly. To
address these challenges, we propose a Classification approach employing
Retrieval-Augmented Generation (Class-RAG). Class-RAG extends the capability of
its base LLM through access to a retrieval library which can be dynamically
updated to enable semantic hotfixing for immediate, flexible risk mitigation.
Compared to traditional fine-tuned models, Class-RAG demonstrates flexibility
and transparency in decision-making. As evidenced by empirical studies,
Class-RAG outperforms on classification and is more robust against adversarial
attack. Besides, our findings suggest that Class-RAG performance scales with
retrieval library size, indicating that increasing the library size is a viable
and low-cost approach to improve content moderation.


http://arxiv.org/abs/2410.14966
Attack as Defense: Run-time Backdoor Implantation for Image Content Protection. (61%)
Haichuan Zhang; Meiyu Lin; Zhaoyi Liu; Renyuan Li; Zhiyuan Cheng; Carl Yang; Mingjie Tang
  As generative models achieve great success, tampering and modifying the
sensitive image contents (i.e., human faces, artist signatures, commercial
logos, etc.) have induced a significant threat with social impact. The backdoor
attack is a method that implants vulnerabilities in a target model, which can
be activated through a trigger. In this work, we innovatively prevent the abuse
of image content modification by implanting the backdoor into image-editing
models. Once the protected sensitive content on an image is modified by an
editing model, the backdoor will be triggered, making the editing fail. Unlike
traditional backdoor attacks that use data poisoning, to enable protection on
individual images and eliminate the need for model training, we developed the
first framework for run-time backdoor implantation, which is both time- and
resource- efficient. We generate imperceptible perturbations on the images to
inject the backdoor and define the protected area as the only backdoor trigger.
Editing other unprotected insensitive areas will not trigger the backdoor,
which minimizes the negative impact on legal image modifications. Evaluations
with state-of-the-art image editing models show that our protective method can
increase the CLIP-FID of generated images from 12.72 to 39.91, or reduce the
SSIM from 0.503 to 0.167 when subjected to malicious editing. At the same time,
our method exhibits minimal impact on benign editing, which demonstrates the
efficacy of our proposed framework. The proposed run-time backdoor can also
achieve effective protection on the latest diffusion models. Code are
available.


http://arxiv.org/abs/2410.14667
Stochastic Gradient Descent Jittering for Inverse Problems: Alleviating the Accuracy-Robustness Tradeoff. (26%)
Peimeng Guan; Mark A. Davenport
  Inverse problems aim to reconstruct unseen data from corrupted or perturbed
measurements. While most work focuses on improving reconstruction quality,
generalization accuracy and robustness are equally important, especially for
safety-critical applications. Model-based architectures (MBAs), such as loop
unrolling methods, are considered more interpretable and achieve better
reconstructions. Empirical evidence suggests that MBAs are more robust to
perturbations than black-box solvers, but the accuracy-robustness tradeoff in
MBAs remains underexplored. In this work, we propose a simple yet effective
training scheme for MBAs, called SGD jittering, which injects noise
iteration-wise during reconstruction. We theoretically demonstrate that SGD
jittering not only generalizes better than the standard mean squared error
training but is also more robust to average-case attacks. We validate SGD
jittering using denoising toy examples, seismic deconvolution, and single-coil
MRI reconstruction. The proposed method achieves cleaner reconstructions for
out-of-distribution data and demonstrates enhanced robustness to adversarial
attacks.


http://arxiv.org/abs/2410.16327
Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs. (13%)
Rui Pu; Chaozhuo Li; Rui Ha; Zejian Chen; Litian Zhang; Zheng Liu; Lirong Qiu; Xi Zhang
  Jailbreak attack can be used to access the vulnerabilities of Large Language
Models (LLMs) by inducing LLMs to generate the harmful content. And the most
common method of the attack is to construct semantically ambiguous prompts to
confuse and mislead the LLMs. To access the security and reveal the intrinsic
relation between the input prompt and the output for LLMs, the distribution of
attention weight is introduced to analyze the underlying reasons. By using
statistical analysis methods, some novel metrics are defined to better describe
the distribution of attention weight, such as the Attention Intensity on
Sensitive Words (Attn_SensWords), the Attention-based Contextual Dependency
Score (Attn_DepScore) and Attention Dispersion Entropy (Attn_Entropy). By
leveraging the distinct characteristics of these metrics, the beam search
algorithm and inspired by the military strategy "Feint and Attack", an
effective jailbreak attack strategy named as Attention-Based Attack (ABA) is
proposed. In the ABA, nested attack prompts are employed to divert the
attention distribution of the LLMs. In this manner, more harmless parts of the
input can be used to attract the attention of the LLMs. In addition, motivated
by ABA, an effective defense strategy called as Attention-Based Defense (ABD)
is also put forward. Compared with ABA, the ABD can be used to enhance the
robustness of LLMs by calibrating the attention distribution of the input
prompt. Some comparative experiments have been given to demonstrate the
effectiveness of ABA and ABD. Therefore, both ABA and ABD can be used to access
the security of the LLMs. The comparative experiment results also give a
logical explanation that the distribution of attention weight can bring great
influence on the output for LLMs.


http://arxiv.org/abs/2410.14651
Real-time Fake News from Adversarial Feedback. (10%)
Sanxing Chen; Yukun Huang; Bhuwan Dhingra
  We show that existing evaluations for fake news detection based on
conventional sources, such as claims on fact-checking websites, result in an
increasing accuracy over time for LLM-based detectors -- even after their
knowledge cutoffs. This suggests that recent popular political claims, which
form the majority of fake news on such sources, are easily classified using
surface-level shallow patterns. Instead, we argue that a proper fake news
detection dataset should test a model's ability to reason factually about the
current world by retrieving and reading related evidence. To this end, we
develop a novel pipeline that leverages natural language feedback from a
RAG-based detector to iteratively modify real-time news into deceptive fake
news that challenges LLMs. Our iterative rewrite decreases the binary
classification AUC by an absolute 17.5 percent for a strong RAG GPT-4o
detector. Our experiments reveal the important role of RAG in both detecting
and generating fake news, as retrieval-free LLM detectors are vulnerable to
unseen events and adversarial attacks, while feedback from RAG detection helps
discover more deceitful patterns in fake news.


http://arxiv.org/abs/2410.14425
Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation. (5%)
Shuai Zhao; Xiaobao Wu; Cong-Duy Nguyen; Meihuizi Jia; Yichao Feng; Luu Anh Tuan
  Parameter-efficient fine-tuning (PEFT) can bridge the gap between large
language models (LLMs) and downstream tasks. However, PEFT has been proven
vulnerable to malicious attacks. Research indicates that poisoned LLMs, even
after PEFT, retain the capability to activate internalized backdoors when input
samples contain predefined triggers. In this paper, we introduce a novel
weak-to-strong unlearning algorithm to defend against backdoor attacks based on
feature alignment knowledge distillation, named W2SDefense. Specifically, we
first train a small-scale language model through full-parameter fine-tuning to
serve as the clean teacher model. Then, this teacher model guides the
large-scale poisoned student model in unlearning the backdoor, leveraging PEFT.
Theoretical analysis suggests that W2SDefense has the potential to enhance the
student model's ability to unlearn backdoor features, preventing the activation
of the backdoor. We conduct experiments on text classification tasks involving
three state-of-the-art language models and three different backdoor attack
algorithms. Our empirical results demonstrate the outstanding performance of
W2SDefense in defending against backdoor attacks without compromising model
performance.


http://arxiv.org/abs/2410.14919
Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step. (1%)
Mingyuan Zhou; Huangjie Zheng; Yi Gu; Zhendong Wang; Hai Huang
  Score identity Distillation (SiD) is a data-free method that has achieved
state-of-the-art performance in image generation by leveraging only a
pretrained diffusion model, without requiring any training data. However, the
ultimate performance of SiD is constrained by the accuracy with which the
pretrained model captures the true data scores at different stages of the
diffusion process. In this paper, we introduce SiDA (SiD with Adversarial
Loss), which not only enhances generation quality but also improves
distillation efficiency by incorporating real images and adversarial loss. SiDA
utilizes the encoder from the generator's score network as a discriminator,
boosting its ability to distinguish between real images and those generated by
SiD. The adversarial loss is batch-normalized within each GPU and then combined
with the original SiD loss. This integration effectively incorporates the
average "fakeness" per GPU batch into the pixel-based SiD loss, enabling SiDA
to distill a single-step generator either from scratch or by fine-tuning an
existing one. SiDA converges significantly faster than its predecessor when
trained from scratch, and swiftly improves upon the original model's
performance after an initial warmup period during fine-tuning from a
pre-distilled SiD generator. This one-step adversarial distillation method
establishes new benchmarks in generation performance when distilling EDM
diffusion models pretrained on CIFAR-10 (32x32) and ImageNet (64x64), achieving
FID score of 1.110 on ImageNet 64x64. It sets record-low FID scores when
distilling EDM2 models trained on ImageNet (512x512), surpassing even the
largest teacher model, EDM2-XXL. Our SiDA's results record FID scores of 2.156
for EDM2-XS, 1.669 for EDM2-S, 1.488 for EDM2-M, and 1.465 for EDM2-L,
demonstrating significant improvements across all model sizes. Our open-source
code will be integrated into the SiD codebase.


http://arxiv.org/abs/2410.14089
MMAD-Purify: A Precision-Optimized Framework for Efficient and Scalable Multi-Modal Attacks. (99%)
Xinxin Liu; Zhongliang Guo; Siyuan Huang; Chun Pong Lau
  Neural networks have achieved remarkable performance across a wide range of
tasks, yet they remain susceptible to adversarial perturbations, which pose
significant risks in safety-critical applications. With the rise of
multimodality, diffusion models have emerged as powerful tools not only for
generative tasks but also for various applications such as image editing,
inpainting, and super-resolution. However, these models still lack robustness
due to limited research on attacking them to enhance their resilience.
Traditional attack techniques, such as gradient-based adversarial attacks and
diffusion model-based methods, are hindered by computational inefficiencies and
scalability issues due to their iterative nature. To address these challenges,
we introduce an innovative framework that leverages the distilled backbone of
diffusion models and incorporates a precision-optimized noise predictor to
enhance the effectiveness of our attack framework. This approach not only
enhances the attack's potency but also significantly reduces computational
costs. Our framework provides a cutting-edge solution for multi-modal
adversarial attacks, ensuring reduced latency and the generation of
high-fidelity adversarial examples with superior success rates. Furthermore, we
demonstrate that our framework achieves outstanding transferability and
robustness against purification defenses, outperforming existing gradient-based
attack models in both effectiveness and efficiency.


http://arxiv.org/abs/2410.14105
DMGNN: Detecting and Mitigating Backdoor Attacks in Graph Neural Networks. (95%)
Hao Sui; Bing Chen; Jiale Zhang; Chengcheng Zhu; Di Wu; Qinghua Lu; Guodong Long
  Recent studies have revealed that GNNs are highly susceptible to multiple
adversarial attacks. Among these, graph backdoor attacks pose one of the most
prominent threats, where attackers cause models to misclassify by learning the
backdoored features with injected triggers and modified target labels during
the training phase. Based on the features of the triggers, these attacks can be
categorized into out-of-distribution (OOD) and in-distribution (ID) graph
backdoor attacks, triggers with notable differences from the clean sample
feature distributions constitute OOD backdoor attacks, whereas the triggers in
ID backdoor attacks are nearly identical to the clean sample feature
distributions. Existing methods can successfully defend against OOD backdoor
attacks by comparing the feature distribution of triggers and clean samples but
fail to mitigate stealthy ID backdoor attacks. Due to the lack of proper
supervision signals, the main task accuracy is negatively affected in defending
against ID backdoor attacks. To bridge this gap, we propose DMGNN against OOD
and ID graph backdoor attacks that can powerfully eliminate stealthiness to
guarantee defense effectiveness and improve the model performance.
Specifically, DMGNN can easily identify the hidden ID and OOD triggers via
predicting label transitions based on counterfactual explanation. To further
filter the diversity of generated explainable graphs and erase the influence of
the trigger features, we present a reverse sampling pruning method to screen
and discard the triggers directly on the data level. Extensive experimental
evaluations on open graph datasets demonstrate that DMGNN far outperforms the
state-of-the-art (SOTA) defense methods, reducing the attack success rate to 5%
with almost negligible degradation in model performance (within 3.5%).


http://arxiv.org/abs/2410.13995
Adversarial Inception for Bounded Backdoor Poisoning in Deep Reinforcement Learning. (67%)
Ethan Rathbun; Christopher Amato; Alina Oprea
  Recent works have demonstrated the vulnerability of Deep Reinforcement
Learning (DRL) algorithms against training-time, backdoor poisoning attacks.
These attacks induce pre-determined, adversarial behavior in the agent upon
observing a fixed trigger during deployment while allowing the agent to solve
its intended task during training. Prior attacks rely on arbitrarily large
perturbations to the agent's rewards to achieve both of these objectives -
leaving them open to detection. Thus, in this work, we propose a new class of
backdoor attacks against DRL which achieve state of the art performance while
minimally altering the agent's rewards. These "inception" attacks train the
agent to associate the targeted adversarial behavior with high returns by
inducing a disjunction between the agent's chosen action and the true action
executed in the environment during training. We formally define these attacks
and prove they can achieve both adversarial objectives. We then devise an
online inception attack which significantly out-performs prior attacks under
bounded reward constraints.


http://arxiv.org/abs/2410.13236
SPIN: Self-Supervised Prompt INjection. (67%)
Leon Zhou; Junfeng Yang; Chengzhi Mao
  Large Language Models (LLMs) are increasingly used in a variety of important
applications, yet their safety and reliability remain as major concerns.
Various adversarial and jailbreak attacks have been proposed to bypass the
safety alignment and cause the model to produce harmful responses. We introduce
Self-supervised Prompt INjection (SPIN) which can detect and reverse these
various attacks on LLMs. As our self-supervised prompt defense is done at
inference-time, it is also compatible with existing alignment and adds an
additional layer of safety for defense. Our benchmarks demonstrate that our
system can reduce the attack success rate by up to 87.9%, while maintaining the
performance on benign user requests. In addition, we discuss the situation of
an adaptive attacker and show that our method is still resilient against
attackers who are aware of our defense.


http://arxiv.org/abs/2410.13691
Jailbreaking LLM-Controlled Robots. (56%)
Alexander Robey; Zachary Ravichandran; Vijay Kumar; Hamed Hassani; George J. Pappas
  The recent introduction of large language models (LLMs) has revolutionized
the field of robotics by enabling contextual reasoning and intuitive
human-robot interaction in domains as varied as manipulation, locomotion, and
self-driving vehicles. When viewed as a stand-alone technology, LLMs are known
to be vulnerable to jailbreaking attacks, wherein malicious prompters elicit
harmful text by bypassing LLM safety guardrails. To assess the risks of
deploying LLMs in robotics, in this paper, we introduce RoboPAIR, the first
algorithm designed to jailbreak LLM-controlled robots. Unlike existing, textual
attacks on LLM chatbots, RoboPAIR elicits harmful physical actions from
LLM-controlled robots, a phenomenon we experimentally demonstrate in three
scenarios: (i) a white-box setting, wherein the attacker has full access to the
NVIDIA Dolphins self-driving LLM, (ii) a gray-box setting, wherein the attacker
has partial access to a Clearpath Robotics Jackal UGV robot equipped with a
GPT-4o planner, and (iii) a black-box setting, wherein the attacker has only
query access to the GPT-3.5-integrated Unitree Robotics Go2 robot dog. In each
scenario and across three new datasets of harmful robotic actions, we
demonstrate that RoboPAIR, as well as several static baselines, finds
jailbreaks quickly and effectively, often achieving 100% attack success rates.
Our results reveal, for the first time, that the risks of jailbroken LLMs
extend far beyond text generation, given the distinct possibility that
jailbroken robots could cause physical damage in the real world. Indeed, our
results on the Unitree Go2 represent the first successful jailbreak of a
deployed commercial robotic system. Addressing this emerging vulnerability is
critical for ensuring the safe deployment of LLMs in robotics. Additional media
is available at: https://robopair.org


http://arxiv.org/abs/2410.13722
Persistent Pre-Training Poisoning of LLMs. (33%)
Yiming Zhang; Javier Rando; Ivan Evtimov; Jianfeng Chi; Eric Michael Smith; Nicholas Carlini; Florian Tramèr; Daphne Ippolito
  Large language models are pre-trained on uncurated text datasets consisting
of trillions of tokens scraped from the Web. Prior work has shown that: (1)
web-scraped pre-training datasets can be practically poisoned by malicious
actors; and (2) adversaries can compromise language models after poisoning
fine-tuning datasets. Our work evaluates for the first time whether language
models can also be compromised during pre-training, with a focus on the
persistence of pre-training attacks after models are fine-tuned as helpful and
harmless chatbots (i.e., after SFT and DPO). We pre-train a series of LLMs from
scratch to measure the impact of a potential poisoning adversary under four
different attack objectives (denial-of-service, belief manipulation,
jailbreaking, and prompt stealing), and across a wide range of model sizes
(from 600M to 7B). Our main result is that poisoning only 0.1% of a model's
pre-training dataset is sufficient for three out of four attacks to measurably
persist through post-training. Moreover, simple attacks like denial-of-service
persist through post-training with a poisoning rate of only 0.001%.


http://arxiv.org/abs/2410.13974
Trojan Prompt Attacks on Graph Neural Networks. (4%)
Minhua Lin; Zhiwei Zhang; Enyan Dai; Zongyu Wu; Yilong Wang; Xiang Zhang; Suhang Wang
  Graph Prompt Learning (GPL) has been introduced as a promising approach that
uses prompts to adapt pre-trained GNN models to specific downstream tasks
without requiring fine-tuning of the entire model. Despite the advantages of
GPL, little attention has been given to its vulnerability to backdoor attacks,
where an adversary can manipulate the model's behavior by embedding hidden
triggers. Existing graph backdoor attacks rely on modifying model parameters
during training, but this approach is impractical in GPL as GNN encoder
parameters are frozen after pre-training. Moreover, downstream users may
fine-tune their own task models on clean datasets, further complicating the
attack. In this paper, we propose TGPA, a backdoor attack framework designed
specifically for GPL. TGPA injects backdoors into graph prompts without
modifying pre-trained GNN encoders and ensures high attack success rates and
clean accuracy. To address the challenge of model fine-tuning by users, we
introduce a finetuning-resistant poisoning approach that maintains the
effectiveness of the backdoor even after downstream model adjustments.
Extensive experiments on multiple datasets under various settings demonstrate
the effectiveness of TGPA in compromising GPL models with fixed GNN encoders.


http://arxiv.org/abs/2410.13334
Do LLMs Have Political Correctness? Analyzing Ethical Biases and Jailbreak Vulnerabilities in AI Systems. (2%)
Isack Lee; Haebin Seong
  Although large language models (LLMs) demonstrate impressive proficiency in
various tasks, they present potential safety risks, such as `jailbreaks', where
malicious inputs can coerce LLMs into generating harmful content. To address
these issues, many LLM developers have implemented various safety measures to
align these models. This alignment involves several techniques, including data
filtering during pre-training, supervised fine-tuning, reinforcement learning
from human feedback, and red-teaming exercises. These methods often introduce
deliberate and intentional biases similar to Political Correctness (PC) to
ensure the ethical behavior of LLMs. In this paper, we delve into the
intentional biases injected into LLMs for safety purposes and examine methods
to circumvent these safety alignment techniques. Notably, these intentional
biases result in a jailbreaking success rate in GPT-4o models that differs by
20% between non-binary and cisgender keywords and by 16% between white and
black keywords, even when the other parts of the prompts are identical. We
introduce the concept of PCJailbreak, highlighting the inherent risks posed by
these safety-induced biases. Additionally, we propose an efficient defense
method PCDefense, which prevents jailbreak attempts by injecting defense
prompts prior to generation. PCDefense stands as an appealing alternative to
Guard Models, such as Llama-Guard, that require additional inference cost after
text generation. Our findings emphasize the urgent need for LLM developers to
adopt a more responsible approach when designing and implementing safety
measures.


http://arxiv.org/abs/2410.13193
Golyadkin's Torment: Doppelg\"angers and Adversarial Vulnerability. (99%)
George I. Kamberov
  Many machine learning (ML) classifiers are claimed to outperform humans, but
they still make mistakes that humans do not. The most notorious examples of
such mistakes are adversarial visual metamers. This paper aims to define and
investigate the phenomenon of adversarial Doppelgangers (AD), which includes
adversarial visual metamers, and to compare the performance and robustness of
ML classifiers to human performance.
  We find that AD are inputs that are close to each other with respect to a
perceptual metric defined in this paper. AD are qualitatively different from
the usual adversarial examples. The vast majority of classifiers are vulnerable
to AD and robustness-accuracy trade-offs may not improve them. Some
classification problems may not admit any AD robust classifiers because the
underlying classes are ambiguous. We provide criteria that can be used to
determine whether a classification problem is well defined or not; describe the
structure and attributes of an AD-robust classifier; introduce and explore the
notions of conceptual entropy and regions of conceptual ambiguity for
classifiers that are vulnerable to AD attacks, along with methods to bound the
AD fooling rate of an attack. We define the notion of classifiers that exhibit
hypersensitive behavior, that is, classifiers whose only mistakes are
adversarial Doppelgangers. Improving the AD robustness of hyper-sensitive
classifiers is equivalent to improving accuracy. We identify conditions
guaranteeing that all classifiers with sufficiently high accuracy are
hyper-sensitive.
  Our findings are aimed at significant improvements in the reliability and
security of machine learning systems.


http://arxiv.org/abs/2410.12307
DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain. (99%)
Fengpeng Li; Kemou Li; Haiwei Wu; Jinyu Tian; Jiantao Zhou
  To protect deep neural networks (DNNs) from adversarial attacks, adversarial
training (AT) is developed by incorporating adversarial examples (AEs) into
model training. Recent studies show that adversarial attacks disproportionately
impact the patterns within the phase of the sample's frequency spectrum --
typically containing crucial semantic information -- more than those in the
amplitude, resulting in the model's erroneous categorization of AEs. We find
that, by mixing the amplitude of training samples' frequency spectrum with
those of distractor images for AT, the model can be guided to focus on phase
patterns unaffected by adversarial perturbations. As a result, the model's
robustness can be improved. Unfortunately, it is still challenging to select
appropriate distractor images, which should mix the amplitude without affecting
the phase patterns. To this end, in this paper, we propose an optimized
Adversarial Amplitude Generator (AAG) to achieve a better tradeoff between
improving the model's robustness and retaining phase patterns. Based on this
generator, together with an efficient AE production procedure, we design a new
Dual Adversarial Training (DAT) strategy. Experiments on various datasets show
that our proposed DAT leads to significantly improved robustness against
diverse adversarial attacks.


http://arxiv.org/abs/2410.13122
Boosting Imperceptibility of Stable Diffusion-based Adversarial Examples Generation with Momentum. (99%)
Nashrah Haque; Xiang Li; Zhehui Chen; Yanzhao Wu; Lei Yu; Arun Iyengar; Wenqi Wei
  We propose a novel framework, Stable Diffusion-based Momentum Integrated
Adversarial Examples (SD-MIAE), for generating adversarial examples that can
effectively mislead neural network classifiers while maintaining visual
imperceptibility and preserving the semantic similarity to the original class
label. Our method leverages the text-to-image generation capabilities of the
Stable Diffusion model by manipulating token embeddings corresponding to the
specified class in its latent space. These token embeddings guide the
generation of adversarial images that maintain high visual fidelity. The
SD-MIAE framework consists of two phases: (1) an initial adversarial
optimization phase that modifies token embeddings to produce misclassified yet
natural-looking images and (2) a momentum-based optimization phase that refines
the adversarial perturbations. By introducing momentum, our approach stabilizes
the optimization of perturbations across iterations, enhancing both the
misclassification rate and visual fidelity of the generated adversarial
examples. Experimental results demonstrate that SD-MIAE achieves a high
misclassification rate of 79%, improving by 35% over the state-of-the-art
method while preserving the imperceptibility of adversarial perturbations and
the semantic similarity to the original class label, making it a practical
method for robust adversarial evaluation.


http://arxiv.org/abs/2410.12671
New Paradigm of Adversarial Training: Breaking Inherent Trade-Off between Accuracy and Robustness via Dummy Classes. (98%)
Yanyun Wang; Li Liu; Zi Liang; Qingqing Ye; Haibo Hu
  Adversarial Training (AT) is one of the most effective methods to enhance the
robustness of DNNs. However, existing AT methods suffer from an inherent
trade-off between adversarial robustness and clean accuracy, which seriously
hinders their real-world deployment. While this problem has been widely studied
within the current AT paradigm, existing AT methods still typically experience
a reduction in clean accuracy by over 10% to date, without significant
improvements in robustness compared with simple baselines like PGD-AT. This
inherent trade-off raises a question: whether the current AT paradigm, which
assumes to learn the corresponding benign and adversarial samples as the same
class, inappropriately combines clean and robust objectives that may be
essentially inconsistent. In this work, we surprisingly reveal that up to 40%
of CIFAR-10 adversarial samples always fail to satisfy such an assumption
across various AT methods and robust models, explicitly indicating the
improvement room for the current AT paradigm. Accordingly, to relax the tension
between clean and robust learning derived from this overstrict assumption, we
propose a new AT paradigm by introducing an additional dummy class for each
original class, aiming to accommodate the hard adversarial samples with shifted
distribution after perturbation. The robustness w.r.t. these adversarial
samples can be achieved by runtime recovery from the predicted dummy classes to
their corresponding original ones, eliminating the compromise with clean
learning. Building on this new paradigm, we propose a novel plug-and-play AT
technology named DUmmy Classes-based Adversarial Training (DUCAT). Extensive
experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that the
DUCAT concurrently improves clean accuracy and adversarial robustness compared
with state-of-the-art benchmarks, effectively breaking the existing inherent
trade-off.


http://arxiv.org/abs/2410.12425
Perseus: Leveraging Common Data Patterns with Curriculum Learning for More Robust Graph Neural Networks. (92%)
Kaiwen Xia; Huijun Wu; Duanyu Li; Min Xie; Ruibo Wang; Wenzhe Zhang
  Graph Neural Networks (GNNs) excel at handling graph data but remain
vulnerable to adversarial attacks. Existing defense methods typically rely on
assumptions like graph sparsity and homophily to either preprocess the graph or
guide structure learning. However, preprocessing methods often struggle to
accurately distinguish between normal edges and adversarial perturbations,
leading to suboptimal results due to the loss of valuable edge information.
Robust graph neural network models train directly on graph data affected by
adversarial perturbations, without preprocessing. This can cause the model to
get stuck in poor local optima, negatively affecting its performance. To
address these challenges, we propose Perseus, a novel adversarial defense
method based on curriculum learning. Perseus assesses edge difficulty using
global homophily and applies a curriculum learning strategy to adjust the
learning order, guiding the model to learn the full graph structure while
adaptively focusing on common data patterns. This approach mitigates the impact
of adversarial perturbations. Experiments show that models trained with Perseus
achieve superior performance and are significantly more robust to adversarial
attacks.


http://arxiv.org/abs/2410.12607
Low-Rank Adversarial PGD Attack. (84%)
Dayana Savostianova; Emanuele Zangrando; Francesco Tudisco
  Adversarial attacks on deep neural network models have seen rapid development
and are extensively used to study the stability of these networks. Among
various adversarial strategies, Projected Gradient Descent (PGD) is a widely
adopted method in computer vision due to its effectiveness and quick
implementation, making it suitable for adversarial training. In this work, we
observe that in many cases, the perturbations computed using PGD predominantly
affect only a portion of the singular value spectrum of the original image,
suggesting that these perturbations are approximately low-rank. Motivated by
this observation, we propose a variation of PGD that efficiently computes a
low-rank attack. We extensively validate our method on a range of standard
models as well as robust models that have undergone adversarial training. Our
analysis indicates that the proposed low-rank PGD can be effectively used in
adversarial training due to its straightforward and fast implementation coupled
with competitive performance. Notably, we find that low-rank PGD often performs
comparably to, and sometimes even outperforms, the traditional full-rank PGD
attack, while using significantly less memory.


http://arxiv.org/abs/2410.13138
Data Defenses Against Large Language Models. (76%)
William Agnew; Harry H. Jiang; Cella Sum; Maarten Sap; Sauvik Das
  Large language models excel at performing inference over text to extract
information, summarize information, or generate additional text. These
inference capabilities are implicated in a variety of ethical harms spanning
surveillance, labor displacement, and IP/copyright theft. While many policy,
legal, and technical mitigations have been proposed to counteract these harms,
these mitigations typically require cooperation from institutions that move
slower than technical advances (i.e., governments) or that have few incentives
to act to counteract these harms (i.e., the corporations that create and profit
from these LLMs). In this paper, we define and build "data defenses" -- a novel
strategy that directly empowers data owners to block LLMs from performing
inference on their data. We create data defenses by developing a method to
automatically generate adversarial prompt injections that, when added to input
text, significantly reduce the ability of LLMs to accurately infer personally
identifying information about the subject of the input text or to use
copyrighted text in inference. We examine the ethics of enabling such direct
resistance to LLM inference, and argue that making data defenses that resist
and subvert LLMs enables the realization of important values such as data
ownership, data sovereignty, and democratic control over AI systems. We verify
that our data defenses are cheap and fast to generate, work on the latest
commercial and open-source LLMs, resistance to countermeasures, and are robust
to several different attack settings. Finally, we consider the security
implications of LLM data defenses and outline several future research
directions in this area. Our code is available at
https://github.com/wagnew3/LLMDataDefenses and a tool for using our defenses to
protect text against LLM inference is at
https://wagnew3.github.io/LLM-Data-Defenses/.


http://arxiv.org/abs/2410.13010
Hiding-in-Plain-Sight (HiPS) Attack on CLIP for Targetted Object Removal from Images. (61%)
Arka Daw; Megan Hong-Thanh Chung; Maria Mahbub; Amir Sadovnik
  Machine learning models are known to be vulnerable to adversarial attacks,
but traditional attacks have mostly focused on single-modalities. With the rise
of large multi-modal models (LMMs) like CLIP, which combine vision and language
capabilities, new vulnerabilities have emerged. However, prior work in
multimodal targeted attacks aim to completely change the model's output to what
the adversary wants. In many realistic scenarios, an adversary might seek to
make only subtle modifications to the output, so that the changes go unnoticed
by downstream models or even by humans. We introduce Hiding-in-Plain-Sight
(HiPS) attacks, a novel class of adversarial attacks that subtly modifies model
predictions by selectively concealing target object(s), as if the target object
was absent from the scene. We propose two HiPS attack variants, HiPS-cls and
HiPS-cap, and demonstrate their effectiveness in transferring to downstream
image captioning models, such as CLIP-Cap, for targeted object removal from
image captions.


http://arxiv.org/abs/2410.13907
NSmark: Null Space Based Black-box Watermarking Defense Framework for Pre-trained Language Models. (26%)
Haodong Zhao; Jinming Hu; Peixuan Li; Fangqi Li; Jinrui Sha; Peixuan Chen; Zhuosheng Zhang; Gongshen Liu
  Pre-trained language models (PLMs) have emerged as critical intellectual
property (IP) assets that necessitate protection. Although various watermarking
strategies have been proposed, they remain vulnerable to Linear Functionality
Equivalence Attacks (LFEA), which can invalidate most existing white-box
watermarks without prior knowledge of the watermarking scheme or training data.
This paper further analyzes and extends the attack scenarios of LFEA to the
commonly employed black-box settings for PLMs by considering Last-Layer outputs
(dubbed LL-LFEA). We discover that the null space of the output matrix remains
invariant against LL-LFEA attacks. Based on this finding, we propose NSmark, a
task-agnostic, black-box watermarking scheme capable of resisting LL-LFEA
attacks. NSmark consists of three phases: (i) watermark generation using the
digital signature of the owner, enhanced by spread spectrum modulation for
increased robustness; (ii) watermark embedding through an output mapping
extractor that preserves PLM performance while maximizing watermark capacity;
(iii) watermark verification, assessed by extraction rate and null space
conformity. Extensive experiments on both pre-training and downstream tasks
confirm the effectiveness, reliability, fidelity, and robustness of our
approach. Code is available at https://github.com/dongdongzhaoUP/NSmark.


http://arxiv.org/abs/2410.13910
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace. (5%)
Jinluan Yang; Anke Tang; Didi Zhu; Zhengyu Chen; Li Shen; Fei Wu
  Model merging has gained significant attention as a cost-effective approach
to integrate multiple single-task fine-tuned models into a unified one that can
perform well on multiple tasks. However, existing model merging techniques
primarily focus on resolving conflicts between task-specific models, they often
overlook potential security threats, particularly the risk of backdoor attacks
in the open-source model ecosystem. In this paper, we first investigate the
vulnerabilities of existing model merging methods to backdoor attacks,
identifying two critical challenges: backdoor succession and backdoor transfer.
To address these issues, we propose a novel Defense-Aware Merging (DAM)
approach that simultaneously mitigates task interference and backdoor
vulnerabilities. Specifically, DAM employs a meta-learning-based optimization
method with dual masks to identify a shared and safety-aware subspace for model
merging. These masks are alternately optimized: the Task-Shared mask identifies
common beneficial parameters across tasks, aiming to preserve task-specific
knowledge while reducing interference, while the Backdoor-Detection mask
isolates potentially harmful parameters to neutralize security threats. This
dual-mask design allows us to carefully balance the preservation of useful
knowledge and the removal of potential vulnerabilities. Compared to existing
merging methods, DAM achieves a more favorable balance between performance and
security, reducing the attack success rate by 2-10 percentage points while
sacrificing only about 1% in accuracy. Furthermore, DAM exhibits robust
performance and broad applicability across various types of backdoor attacks
and the number of compromised models involved in the merging process. We will
release the codes and models soon.


http://arxiv.org/abs/2410.12443
Reconstruction of Differentially Private Text Sanitization via Large Language Models. (4%)
Shuchao Pang; Zhigang Lu; Haichen Wang; Peng Fu; Yongbin Zhou; Minhui Xue; Bo Li
  Differential privacy (DP) is the de facto privacy standard against privacy
leakage attacks, including many recently discovered ones against large language
models (LLMs). However, we discovered that LLMs could reconstruct the
altered/removed privacy from given DP-sanitized prompts. We propose two attacks
(black-box and white-box) based on the accessibility to LLMs and show that LLMs
could connect the pair of DP-sanitized text and the corresponding private
training data of LLMs by giving sample text pairs as instructions (in the
black-box attacks) or fine-tuning data (in the white-box attacks). To
illustrate our findings, we conduct comprehensive experiments on modern LLMs
(e.g., LLaMA-2, LLaMA-3, ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Claude-3,
Claude-3.5, OPT, GPT-Neo, GPT-J, Gemma-2, and Pythia) using commonly used
datasets (such as WikiMIA, Pile-CC, and Pile-Wiki) against both word-level and
sentence-level DP. The experimental results show promising recovery rates,
e.g., the black-box attacks against the word-level DP over WikiMIA dataset gave
72.18% on LLaMA-2 (70B), 82.39% on LLaMA-3 (70B), 75.35% on Gemma-2, 91.2% on
ChatGPT-4o, and 94.01% on Claude-3.5 (Sonnet). More urgently, this study
indicates that these well-known LLMs have emerged as a new security risk for
existing DP text sanitization approaches in the current environment.


http://arxiv.org/abs/2410.12759
Unitary Multi-Margin BERT for Robust Natural Language Processing. (4%)
Hao-Yuan Chang; Kang L. Wang
  Recent developments in adversarial attacks on deep learning leave many
mission-critical natural language processing (NLP) systems at risk of
exploitation. To address the lack of computationally efficient adversarial
defense methods, this paper reports a novel, universal technique that
drastically improves the robustness of Bidirectional Encoder Representations
from Transformers (BERT) by combining the unitary weights with the multi-margin
loss. We discover that the marriage of these two simple ideas amplifies the
protection against malicious interference. Our model, the unitary multi-margin
BERT (UniBERT), boosts post-attack classification accuracies significantly by
5.3% to 73.8% while maintaining competitive pre-attack accuracies. Furthermore,
the pre-attack and post-attack accuracy tradeoff can be adjusted via a single
scalar parameter to best fit the design requirements for the target
applications.


http://arxiv.org/abs/2410.13045
FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuning. (2%)
Evelyn Ma; Chao Pan; Rasoul Etesami; Han Zhao; Olgica Milenkovic
  The performance of Transfer Learning (TL) heavily relies on effective
pretraining, which demands large datasets and substantial computational
resources. As a result, executing TL is often challenging for individual model
developers. Federated Learning (FL) addresses these issues by facilitating
collaborations among clients, expanding the dataset indirectly, distributing
computational costs, and preserving privacy. However, key challenges remain
unresolved. First, existing FL methods tend to optimize transferability only
within local domains, neglecting the global learning domain. Second, most
approaches rely on indirect transferability metrics, which do not accurately
reflect the final target loss or true degree of transferability. To address
these gaps, we propose two enhancements to FL. First, we introduce a
client-server exchange protocol that leverages cross-client Jacobian (gradient)
norms to boost transferability. Second, we increase the average Jacobian norm
across clients at the server, using this as a local regularizer to reduce
cross-client Jacobian variance. Our transferable federated algorithm, termed
FedGTST (Federated Global Transferability via Statistics Tuning), demonstrates
that increasing the average Jacobian and reducing its variance allows for
tighter control of the target loss. This leads to an upper bound on the target
loss in terms of the source loss and source-target domain discrepancy.
Extensive experiments on datasets such as MNIST to MNIST-M and CIFAR10 to SVHN
show that FedGTST outperforms relevant baselines, including FedSR. On the
second dataset pair, FedGTST improves accuracy by 9.8% over FedSR and 7.6% over
FedIIR when LeNet is used as the backbone.


http://arxiv.org/abs/2410.12295
Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors. (2%)
Linwei Tao; Haolan Guo; Minjing Dong; Chang Xu
  Calibration is crucial in deep learning applications, especially in fields
like healthcare and autonomous driving, where accurate confidence estimates are
vital for decision-making. However, deep neural networks often suffer from
miscalibration, with reliability diagrams and Expected Calibration Error (ECE)
being the only standard perspective for evaluating calibration performance. In
this paper, we introduce the concept of consistency as an alternative
perspective on model calibration, inspired by uncertainty estimation literature
in large language models (LLMs). We highlight its advantages over the
traditional reliability-based view. Building on this concept, we propose a
post-hoc calibration method called Consistency Calibration (CC), which adjusts
confidence based on the model's consistency across perturbed inputs. CC is
particularly effective in locally uncertainty estimation, as it requires no
additional data samples or label information, instead generating input
perturbations directly from the source data. Moreover, we show that performing
perturbations at the logit level significantly improves computational
efficiency. We validate the effectiveness of CC through extensive comparisons
with various post-hoc and training-time calibration methods, demonstrating
state-of-the-art performance on standard datasets such as CIFAR-10, CIFAR-100,
and ImageNet, as well as on long-tailed datasets like ImageNet-LT.


http://arxiv.org/abs/2410.12677
Efficient Optimization Algorithms for Linear Adversarial Training. (1%)
Antônio H. RIbeiro; Thomas B. Schön; Dave Zahariah; Francis Bach
  Adversarial training can be used to learn models that are robust against
perturbations. For linear models, it can be formulated as a convex optimization
problem. Compared to methods proposed in the context of deep learning,
leveraging the optimization structure allows significantly faster convergence
rates. Still, the use of generic convex solvers can be inefficient for
large-scale problems. Here, we propose tailored optimization algorithms for the
adversarial training of linear models, which render large-scale regression and
classification problems more tractable. For regression problems, we propose a
family of solvers based on iterative ridge regression and, for classification,
a family of solvers based on projected gradient descent. The methods are based
on extended variable reformulations of the original problem. We illustrate
their efficiency in numerical examples.


http://arxiv.org/abs/2410.13073
PromptExp: Multi-granularity Prompt Explanation of Large Language Models. (1%)
Ximing Dong; Shaowei Wang; Dayi Lin; Gopi Krishnan Rajbahadur; Boquan Zhou; Shichao Liu; Ahmed E. Hassan
  Large Language Models excel in tasks like natural language understanding and
text generation. Prompt engineering plays a critical role in leveraging LLM
effectively. However, LLMs black-box nature hinders its interpretability and
effective prompting engineering. A wide range of model explanation approaches
have been developed for deep learning models, However, these local explanations
are designed for single-output tasks like classification and regression,and
cannot be directly applied to LLMs, which generate sequences of tokens. Recent
efforts in LLM explanation focus on natural language explanations, but they are
prone to hallucinations and inaccuracies. To address this, we introduce
OurTool, a framework for multi-granularity prompt explanations by aggregating
token-level insights. OurTool introduces two token-level explanation
approaches: 1.an aggregation-based approach combining local explanation
techniques, and 2. a perturbation-based approach with novel techniques to
evaluate token masking impact. OurTool supports both white-box and black-box
explanations and extends explanations to higher granularity levels, enabling
flexible analysis. We evaluate OurTool in case studies such as sentiment
analysis, showing the perturbation-based approach performs best using semantic
similarity to assess perturbation impact. Furthermore, we conducted a user
study to confirm OurTool's accuracy and practical value, and demonstrate its
potential to enhance LLM interpretability.


http://arxiv.org/abs/2410.12955
Long-Tailed Backdoor Attack Using Dynamic Data Augmentation Operations. (1%)
Lu Pang; Tao Sun; Weimin Lyu; Haibin Ling; Chao Chen
  Recently, backdoor attack has become an increasing security threat to deep
neural networks and drawn the attention of researchers. Backdoor attacks
exploit vulnerabilities in third-party pretrained models during the training
phase, enabling them to behave normally for clean samples and mispredict for
samples with specific triggers. Existing backdoor attacks mainly focus on
balanced datasets. However, real-world datasets often follow long-tailed
distributions. In this paper, for the first time, we explore backdoor attack on
such datasets. Specifically, we first analyze the influence of data imbalance
on backdoor attack. Based on our analysis, we propose an effective backdoor
attack named Dynamic Data Augmentation Operation (D$^2$AO). We design D$^2$AO
selectors to select operations depending jointly on the class, sample type
(clean vs. backdoored) and sample features. Meanwhile, we develop a trigger
generator to generate sample-specific triggers. Through simultaneous
optimization of the backdoored model and trigger generator, guided by dynamic
data augmentation operation selectors, we achieve significant advancements.
Extensive experiments demonstrate that our method can achieve the
state-of-the-art attack performance while preserving the clean accuracy.


http://arxiv.org/abs/2410.12076
Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks. (99%)
Kevin Eykholt; Farhan Ahmed; Pratik Vaishnavi; Amir Rahmati
  The vulnerability of machine learning models in adversarial scenarios has
garnered significant interest in the academic community over the past decade,
resulting in a myriad of attacks and defenses. However, while the community
appears to be overtly successful in devising new attacks across new contexts,
the development of defenses has stalled. After a decade of research, we appear
no closer to securing AI applications beyond additional training. Despite a
lack of effective mitigations, AI development and its incorporation into
existing systems charge full speed ahead with the rise of generative AI and
large language models. Will our ineffectiveness in developing solutions to
adversarial threats further extend to these new technologies?
  In this paper, we argue that overly permissive attack and overly restrictive
defensive threat models have hampered defense development in the ML domain.
Through the lens of adversarial evasion attacks against neural networks, we
critically examine common attack assumptions, such as the ability to bypass any
defense not explicitly built into the model. We argue that these flawed
assumptions, seen as reasonable by the community based on paper acceptance,
have encouraged the development of adversarial attacks that map poorly to
real-world scenarios. In turn, new defenses evaluated against these very
attacks are inadvertently required to be almost perfect and incorporated as
part of the model. But do they need to? In practice, machine learning models
are deployed as a small component of a larger system. We analyze adversarial
machine learning from a system security perspective rather than an AI
perspective and its implications for emerging AI paradigms.


http://arxiv.org/abs/2410.11639
Efficient and Effective Universal Adversarial Attack against Vision-Language Pre-training Models. (98%)
Fan Yang; Yihao Huang; Kailong Wang; Ling Shi; Geguang Pu; Yang Liu; Haoyu Wang
  Vision-language pre-training (VLP) models, trained on large-scale image-text
pairs, have become widely used across a variety of downstream
vision-and-language (V+L) tasks. This widespread adoption raises concerns about
their vulnerability to adversarial attacks. Non-universal adversarial attacks,
while effective, are often impractical for real-time online applications due to
their high computational demands per data instance. Recently, universal
adversarial perturbations (UAPs) have been introduced as a solution, but
existing generator-based UAP methods are significantly time-consuming. To
overcome the limitation, we propose a direct optimization-based UAP approach,
termed DO-UAP, which significantly reduces resource consumption while
maintaining high attack performance. Specifically, we explore the necessity of
multimodal loss design and introduce a useful data augmentation strategy.
Extensive experiments conducted on three benchmark VLP datasets, six popular
VLP models, and three classical downstream tasks demonstrate the efficiency and
effectiveness of DO-UAP. Specifically, our approach drastically decreases the
time consumption by 23-fold while achieving a better attack performance.


http://arxiv.org/abs/2410.11317
Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation. (83%)
Qizhang Li; Xiaochen Yang; Wangmeng Zuo; Yiwen Guo
  Automatic adversarial prompt generation provides remarkable success in
jailbreaking safely-aligned large language models (LLMs). Existing
gradient-based attacks, while demonstrating outstanding performance in
jailbreaking white-box LLMs, often generate garbled adversarial prompts with
chaotic appearance. These adversarial prompts are difficult to transfer to
other LLMs, hindering their performance in attacking unknown victim models. In
this paper, for the first time, we delve into the semantic meaning embedded in
garbled adversarial prompts and propose a novel method that "translates" them
into coherent and human-readable natural language adversarial prompts. In this
way, we can effectively uncover the semantic information that triggers
vulnerabilities of the model and unambiguously transfer it to the victim model,
without overlooking the adversarial information hidden in the garbled text, to
enhance jailbreak attacks. It also offers a new approach to discovering
effective designs for jailbreak prompts, advancing the understanding of
jailbreak attacks. Experimental results demonstrate that our method
significantly improves the success rate of jailbreak attacks against various
safety-aligned LLMs and outperforms state-of-the-arts by large margins. With at
most 10 queries, our method achieves an average attack success rate of 81.8% in
attacking 7 commercial closed-source LLMs, including GPT and Claude-3 series,
on HarmBench. Our method also achieves over 90% attack success rates against
Llama-2-Chat models on AdvBench, despite their outstanding resistance to
jailbreak attacks. Code at:
https://github.com/qizhangli/Adversarial-Prompt-Translator.


http://arxiv.org/abs/2410.14723
BeniFul: Backdoor Defense via Middle Feature Analysis for Deep Neural Networks. (82%)
Xinfu Li; Junying Zhang; Xindi Ma
  Backdoor defenses have recently become important in resisting backdoor
attacks in deep neural networks (DNNs), where attackers implant backdoors into
the DNN model by injecting backdoor samples into the training dataset. Although
there are many defense methods to achieve backdoor detection for DNN inputs and
backdoor elimination for DNN models, they still have not presented a clear
explanation of the relationship between these two missions. In this paper, we
use the features from the middle layer of the DNN model to analyze the
difference between backdoor and benign samples and propose Backdoor
Consistency, which indicates that at least one backdoor exists in the DNN model
if the backdoor trigger is detected exactly on input. By analyzing the middle
features, we design an effective and comprehensive backdoor defense method
named BeniFul, which consists of two parts: a gray-box backdoor input detection
and a white-box backdoor elimination. Specifically, we use the reconstruction
distance from the Variational Auto-Encoder and model inference results to
implement backdoor input detection and a feature distance loss to achieve
backdoor elimination. Experimental results on CIFAR-10 and Tiny ImageNet
against five state-of-the-art attacks demonstrate that our BeniFul exhibits a
great defense capability in backdoor input detection and backdoor elimination.


http://arxiv.org/abs/2410.11272
Cognitive Overload Attack:Prompt Injection for Long Context. (62%)
Bibek Upadhayay; Vahid Behzadan; Amin Karbasi
  Large Language Models (LLMs) have demonstrated remarkable capabilities in
performing tasks across various domains without needing explicit retraining.
This capability, known as In-Context Learning (ICL), while impressive, exposes
LLMs to a variety of adversarial prompts and jailbreaks that manipulate
safety-trained LLMs into generating undesired or harmful output. In this paper,
we propose a novel interpretation of ICL in LLMs through the lens of cognitive
neuroscience, by drawing parallels between learning in human cognition with
ICL. We applied the principles of Cognitive Load Theory in LLMs and empirically
validate that similar to human cognition, LLMs also suffer from cognitive
overload a state where the demand on cognitive processing exceeds the available
capacity of the model, leading to potential errors. Furthermore, we
demonstrated how an attacker can exploit ICL to jailbreak LLMs through
deliberately designed prompts that induce cognitive overload on LLMs, thereby
compromising the safety mechanisms of LLMs. We empirically validate this threat
model by crafting various cognitive overload prompts and show that advanced
models such as GPT-4, Claude-3.5 Sonnet, Claude-3 OPUS, Llama-3-70B-Instruct,
Gemini-1.0-Pro, and Gemini-1.5-Pro can be successfully jailbroken, with attack
success rates of up to 99.99%. Our findings highlight critical vulnerabilities
in LLMs and underscore the urgency of developing robust safeguards. We propose
integrating insights from cognitive load theory into the design and evaluation
of LLMs to better anticipate and mitigate the risks of adversarial attacks. By
expanding our experiments to encompass a broader range of models and by
highlighting vulnerabilities in LLMs' ICL, we aim to ensure the development of
safer and more reliable AI systems.


http://arxiv.org/abs/2410.11290
Backdoor Attack on Vertical Federated Graph Neural Network Learning. (31%)
Jirui Yang; Peng Chen; Zhihui Lu; Ruijun Deng; Qiang Duan; Jianping Zeng
  Federated Graph Neural Network (FedGNN) is a privacy-preserving machine
learning technology that combines federated learning (FL) and graph neural
networks (GNNs). It offers a privacy-preserving solution for training GNNs
using isolated graph data. Vertical Federated Graph Neural Network (VFGNN) is
an important branch of FedGNN, where data features and labels are distributed
among participants, and each participant has the same sample space. Due to the
difficulty of accessing and modifying distributed data and labels, the
vulnerability of VFGNN to backdoor attacks remains largely unexplored. In this
context, we propose BVG, the first method for backdoor attacks in VFGNN.
Without accessing or modifying labels, BVG uses multi-hop triggers and requires
only four target class nodes for an effective backdoor attack. Experiments show
that BVG achieves high attack success rates (ASR) across three datasets and
three different GNN models, with minimal impact on main task accuracy (MTA). We
also evaluate several defense methods, further validating the robustness and
effectiveness of BVG. This finding also highlights the need for advanced
defense mechanisms to counter sophisticated backdoor attacks in practical VFGNN
applications.


http://arxiv.org/abs/2410.11283
AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment. (31%)
Pankayaraj Pathmanathan; Udari Madhushani Sehwag; Michael-Andrei Panaitescu-Liess; Furong Huang
  With the growing adoption of reinforcement learning with human feedback
(RLHF) for aligning large language models (LLMs), the risk of backdoor
installation during alignment has increased, leading to unintended and harmful
behaviors. Existing backdoor triggers are typically limited to fixed word
patterns, making them detectable during data cleaning and easily removable
post-poisoning. In this work, we explore the use of prompt-specific paraphrases
as backdoor triggers, enhancing their stealth and resistance to removal during
LLM alignment. We propose AdvBDGen, an adversarially fortified generative
fine-tuning framework that automatically generates prompt-specific backdoors
that are effective, stealthy, and transferable across models. AdvBDGen employs
a generator-discriminator pair, fortified by an adversary, to ensure the
installability and stealthiness of backdoors. It enables the crafting and
successful installation of complex triggers using as little as 3% of the
fine-tuning data. Once installed, these backdoors can jailbreak LLMs during
inference, demonstrate improved stability against perturbations compared to
traditional constant triggers, and are more challenging to remove. These
findings underscore an urgent need for the research community to develop more
robust defenses against adversarial backdoor threats in LLM alignment.


http://arxiv.org/abs/2410.11533
Multi-round jailbreak attack on large language models. (4%)
Yihua Zhou; Xiaochuan Shi
  Ensuring the safety and alignment of large language models (LLMs) with human
values is crucial for generating responses that are beneficial to humanity.
While LLMs have the capability to identify and avoid harmful queries, they
remain vulnerable to "jailbreak" attacks, where carefully crafted prompts can
induce the generation of toxic content. Traditional single-round jailbreak
attacks, such as GCG and AutoDAN, do not alter the sensitive words in the
dangerous prompts. Although they can temporarily bypass the model's safeguards
through prompt engineering, their success rate drops significantly as the LLM
is further fine-tuned, and they cannot effectively circumvent static rule-based
filters that remove the hazardous vocabulary.
  In this study, to better understand jailbreak attacks, we introduce a
multi-round jailbreak approach. This method can rewrite the dangerous prompts,
decomposing them into a series of less harmful sub-questions to bypass the
LLM's safety checks. We first use the LLM to perform a decomposition task,
breaking down a set of natural language questions into a sequence of
progressive sub-questions, which are then used to fine-tune the Llama3-8B
model, enabling it to decompose hazardous prompts. The fine-tuned model is then
used to break down the problematic prompt, and the resulting sub-questions are
sequentially asked to the victim model. If the victim model rejects a
sub-question, a new decomposition is generated, and the process is repeated
until the final objective is achieved. Our experimental results show a 94\%
success rate on the llama2-7B and demonstrate the effectiveness of this
approach in circumventing static rule-based filters.


http://arxiv.org/abs/2410.12025
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture. (3%)
Sajad Movahedi; Antonio Orvieto; Seyed-Mohsen Moosavi-Dezfooli
  In this paper, we propose the $\textit{geometric invariance hypothesis
(GIH)}$, which argues that when training a neural network, the input space
curvature remains invariant under transformation in certain directions
determined by its architecture. Starting with a simple non-linear binary
classification problem residing on a plane in a high dimensional space, we
observe that while an MLP can solve this problem regardless of the orientation
of the plane, this is not the case for a ResNet. Motivated by this example, we
define two maps that provide a compact $\textit{architecture-dependent}$
summary of the input space geometry of a neural network and its evolution
during training, which we dub the $\textbf{average geometry}$ and
$\textbf{average geometry evolution}$, respectively. By investigating average
geometry evolution at initialization, we discover that the geometry of a neural
network evolves according to the projection of data covariance onto average
geometry. As a result, in cases where the average geometry is low-rank (such as
in a ResNet), the geometry only changes in a subset of the input space. This
causes an architecture-dependent invariance property in input-space curvature,
which we dub GIH. Finally, we present extensive experimental results to observe
the consequences of GIH and how it relates to generalization in neural
networks.


http://arxiv.org/abs/2410.11782
G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networks. (2%)
Guibin Zhang; Yanwei Yue; Xiangguo Sun; Guancheng Wan; Miao Yu; Junfeng Fang; Kun Wang; Dawei Cheng
  Recent advancements in large language model (LLM)-based agents have
demonstrated that collective intelligence can significantly surpass the
capabilities of individual agents, primarily due to well-crafted inter-agent
communication topologies. Despite the diverse and high-performing designs
available, practitioners often face confusion when selecting the most effective
pipeline for their specific task: \textit{Which topology is the best choice for
my task, avoiding unnecessary communication token overhead while ensuring
high-quality solution?} In response to this dilemma, we introduce G-Designer,
an adaptive, efficient, and robust solution for multi-agent deployment, which
dynamically designs task-aware, customized communication topologies.
Specifically, G-Designer models the multi-agent system as a multi-agent
network, leveraging a variational graph auto-encoder to encode both the nodes
(agents) and a task-specific virtual node, and decodes a task-adaptive and
high-performing communication topology. Extensive experiments on six benchmarks
showcase that G-Designer is: \textbf{(1) high-performing}, achieving superior
results on MMLU with accuracy at $84.50\%$ and on HumanEval with pass@1 at
$89.90\%$; \textbf{(2) task-adaptive}, architecting communication protocols
tailored to task difficulty, reducing token consumption by up to $95.33\%$ on
HumanEval; and \textbf{(3) adversarially robust}, defending against agent
adversarial attacks with merely $0.3\%$ accuracy drop.


http://arxiv.org/abs/2410.10760
Denial-of-Service Poisoning Attacks against Large Language Models. (92%)
Kuofeng Gao; Tianyu Pang; Chao Du; Yong Yang; Shu-Tao Xia; Min Lin
  Recent studies have shown that LLMs are vulnerable to denial-of-service (DoS)
attacks, where adversarial inputs like spelling errors or non-semantic prompts
trigger endless outputs without generating an [EOS] token. These attacks can
potentially cause high latency and make LLM services inaccessible to other
users or tasks. However, when there are speech-to-text interfaces (e.g., voice
commands to a robot), executing such DoS attacks becomes challenging, as it is
difficult to introduce spelling errors or non-semantic prompts through speech.
A simple DoS attack in these scenarios would be to instruct the model to "Keep
repeating Hello", but we observe that relying solely on natural instructions
limits output length, which is bounded by the maximum length of the LLM's
supervised finetuning (SFT) data. To overcome this limitation, we propose
poisoning-based DoS (P-DoS) attacks for LLMs, demonstrating that injecting a
single poisoned sample designed for DoS purposes can break the output length
limit. For example, a poisoned sample can successfully attack GPT-4o and GPT-4o
mini (via OpenAI's finetuning API) using less than $1, causing repeated outputs
up to the maximum inference length (16K tokens, compared to 0.5K before
poisoning). Additionally, we perform comprehensive ablation studies on
open-source LLMs and extend our method to LLM agents, where attackers can
control both the finetuning dataset and algorithm. Our findings underscore the
urgent need for defenses against P-DoS attacks to secure LLMs. Our code is
available at https://github.com/sail-sg/P-DoS.


http://arxiv.org/abs/2410.10736
Towards Calibrated Losses for Adversarial Robust Reject Option Classification. (86%)
Vrund Shah; Tejas Chaudhari; Naresh Manwani
  Robustness towards adversarial attacks is a vital property for classifiers in
several applications such as autonomous driving, medical diagnosis, etc. Also,
in such scenarios, where the cost of misclassification is very high, knowing
when to abstain from prediction becomes crucial. A natural question is which
surrogates can be used to ensure learning in scenarios where the input points
are adversarially perturbed and the classifier can abstain from prediction?
This paper aims to characterize and design surrogates calibrated in
"Adversarial Robust Reject Option" setting. First, we propose an adversarial
robust reject option loss $\ell_{d}^{\gamma}$ and analyze it for the hypothesis
set of linear classifiers ($\mathcal{H}_{\textrm{lin}}$). Next, we provide a
complete characterization result for any surrogate to be
$(\ell_{d}^{\gamma},\mathcal{H}_{\textrm{lin}})$- calibrated. To demonstrate
the difficulty in designing surrogates to $\ell_{d}^{\gamma}$, we show negative
calibration results for convex surrogates and quasi-concave conditional risk
cases (these gave positive calibration in adversarial setting without reject
option). We also empirically argue that Shifted Double Ramp Loss (DRL) and
Shifted Double Sigmoid Loss (DSL) satisfy the calibration conditions. Finally,
we demonstrate the robustness of shifted DRL and shifted DSL against
adversarial perturbations on a synthetically generated dataset.


http://arxiv.org/abs/2410.10744
Adversarially Robust Out-of-Distribution Detection Using Lyapunov-Stabilized Embeddings. (86%)
Hossein Mirzaei; Mackenzie W. Mathis
  Despite significant advancements in out-of-distribution (OOD) detection,
existing methods still struggle to maintain robustness against adversarial
attacks, compromising their reliability in critical real-world applications.
Previous studies have attempted to address this challenge by exposing detectors
to auxiliary OOD datasets alongside adversarial training. However, the
increased data complexity inherent in adversarial training, and the myriad of
ways that OOD samples can arise during testing, often prevent these approaches
from establishing robust decision boundaries. To address these limitations, we
propose AROS, a novel approach leveraging neural ordinary differential
equations (NODEs) with Lyapunov stability theorem in order to obtain robust
embeddings for OOD detection. By incorporating a tailored loss function, we
apply Lyapunov stability theory to ensure that both in-distribution (ID) and
OOD data converge to stable equilibrium points within the dynamical system.
This approach encourages any perturbed input to return to its stable
equilibrium, thereby enhancing the model's robustness against adversarial
perturbations. To not use additional data, we generate fake OOD embeddings by
sampling from low-likelihood regions of the ID data feature space,
approximating the boundaries where OOD data are likely to reside. To then
further enhance robustness, we propose the use of an orthogonal binary layer
following the stable feature space, which maximizes the separation between the
equilibrium points of ID and OOD samples. We validate our method through
extensive experiments across several benchmarks, demonstrating superior
performance, particularly under adversarial attacks. Notably, our approach
improves robust detection performance from 37.8% to 80.1% on CIFAR-10 vs.
CIFAR-100 and from 29.0% to 67.0% on CIFAR-100 vs. CIFAR-10.


http://arxiv.org/abs/2410.10322
Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks. (83%)
Binghui Li; Zhixuan Pan; Kaifeng Lyu; Jian Li
  In this work, we investigate a particular implicit bias in the gradient
descent training process, which we term "Feature Averaging", and argue that it
is one of the principal factors contributing to non-robustness of deep neural
networks. Despite the existence of multiple discriminative features capable of
classifying data, neural networks trained by gradient descent exhibit a
tendency to learn the average (or certain combination) of these features,
rather than distinguishing and leveraging each feature individually. In
particular, we provide a detailed theoretical analysis of the training dynamics
of gradient descent in a two-layer ReLU network for a binary classification
task, where the data distribution consists of multiple clusters with orthogonal
cluster center vectors. We rigorously prove that gradient descent converges to
the regime of feature averaging, wherein the weights associated with each
hidden-layer neuron represent an average of the cluster centers (each center
corresponding to a distinct feature). It leads the network classifier to be
non-robust due to an attack that aligns with the negative direction of the
averaged features. Furthermore, we prove that, with the provision of more
granular supervised information, a two-layer multi-class neural network is
capable of learning individual features, from which one can derive a binary
classifier with the optimal robustness under our setting. Besides, we also
conduct extensive experiments using synthetic datasets, MNIST and CIFAR-10 to
substantiate the phenomenon of feature averaging and its role in adversarial
robustness of neural networks. We hope the theoretical and empirical insights
can provide a deeper understanding of the impact of the gradient descent
training on feature learning process, which in turn influences the robustness
of the network, and how more detailed supervision may enhance model robustness.


http://arxiv.org/abs/2410.11205
Adversarially Guided Stateful Defense Against Backdoor Attacks in Federated Deep Learning. (81%)
Hassan Ali; Surya Nepal; Salil S. Kanhere; Sanjay Jha
  Recent works have shown that Federated Learning (FL) is vulnerable to
backdoor attacks. Existing defenses cluster submitted updates from clients and
select the best cluster for aggregation. However, they often rely on
unrealistic assumptions regarding client submissions and sampled clients
population while choosing the best cluster. We show that in realistic FL
settings, state-of-the-art (SOTA) defenses struggle to perform well against
backdoor attacks in FL. To address this, we highlight that backdoored
submissions are adversarially biased and overconfident compared to clean
submissions. We, therefore, propose an Adversarially Guided Stateful Defense
(AGSD) against backdoor attacks on Deep Neural Networks (DNNs) in FL scenarios.
AGSD employs adversarial perturbations to a small held-out dataset to compute a
novel metric, called the trust index, that guides the cluster selection without
relying on any unrealistic assumptions regarding client submissions. Moreover,
AGSD maintains a trust state history of each client that adaptively penalizes
backdoored clients and rewards clean clients. In realistic FL settings, where
SOTA defenses mostly fail to resist attacks, AGSD mostly outperforms all SOTA
defenses with minimal drop in clean accuracy (5% in the worst-case compared to
best accuracy) even when (a) given a very small held-out dataset -- typically
AGSD assumes 50 samples (<= 0.1% of the training data) and (b) no heldout
dataset is available, and out-of-distribution data is used instead. For
reproducibility, our code will be openly available at:
https://github.com/hassanalikhatim/AGSD.


http://arxiv.org/abs/2410.10554
ROSAR: An Adversarial Re-Training Framework for Robust Side-Scan Sonar Object Detection. (67%)
Martin Aubard; László Antal; Ana Madureira; Luis F. Teixeira; Erika Ábrahám
  This paper introduces ROSAR, a novel framework enhancing the robustness of
deep learning object detection models tailored for side-scan sonar (SSS)
images, generated by autonomous underwater vehicles using sonar sensors. By
extending our prior work on knowledge distillation (KD), this framework
integrates KD with adversarial retraining to address the dual challenges of
model efficiency and robustness against SSS noises. We introduce three novel,
publicly available SSS datasets, capturing different sonar setups and noise
conditions. We propose and formalize two SSS safety properties and utilize them
to generate adversarial datasets for retraining. Through a comparative analysis
of projected gradient descent (PGD) and patch-based adversarial attacks, ROSAR
demonstrates significant improvements in model robustness and detection
accuracy under SSS-specific conditions, enhancing the model's robustness by up
to 1.85%. ROSAR is available at
https://github.com/remaro-network/ROSAR-framework.


http://arxiv.org/abs/2410.19785
How to Backdoor Consistency Models? (56%)
Chengen Wang; Murat Kantarcioglu
  Consistency models are a new class of models that generate images by directly
mapping noise to data, allowing for one-step generation and significantly
accelerating the sampling process. However, their robustness against
adversarial attacks has not yet been thoroughly investigated. In this work, we
conduct the first study on the vulnerability of consistency models to backdoor
attacks. While previous research has explored backdoor attacks on diffusion
models, those studies have primarily focused on conventional diffusion models,
employing a customized backdoor training process and objective, whereas
consistency models have distinct training processes and objectives. Our
proposed framework demonstrates the vulnerability of consistency models to
backdoor attacks. During image generation, poisoned consistency models produce
images with a Fr\'echet Inception Distance (FID) comparable to that of a clean
model when sampling from Gaussian noise. However, once the trigger is
activated, they generate backdoor target images. We explore various trigger and
target configurations to evaluate the vulnerability of consistency models,
including the use of random noise as a trigger. This novel trigger is visually
inconspicuous, more challenging to detect, and aligns well with the sampling
process of consistency models. Across all configurations, our framework
successfully compromises the consistency models while maintaining high utility
and specificity. We also examine the stealthiness of our proposed attack, which
is attributed to the unique properties of consistency models and the elusive
nature of the Gaussian noise trigger.


http://arxiv.org/abs/2410.10700
LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts. (31%)
Qibing Ren; Hao Li; Dongrui Liu; Zhanxu Xie; Xiaoya Lu; Yu Qiao; Lei Sha; Junchi Yan; Lizhuang Ma; Jing Shao
  Safety concerns in large language models (LLMs) have gained significant
attention due to their exposure to potentially harmful data during
pre-training. In this paper, we identify a new safety vulnerability in LLMs:
their susceptibility to \textit{natural distribution shifts} between attack
prompts and original toxic prompts, where seemingly benign prompts,
semantically related to harmful content, can bypass safety mechanisms. To
explore this issue, we introduce a novel attack method, \textit{ActorBreaker},
which identifies actors related to toxic prompts within pre-training
distribution to craft multi-turn prompts that gradually lead LLMs to reveal
unsafe content. ActorBreaker is grounded in Latour's actor-network theory,
encompassing both human and non-human actors to capture a broader range of
vulnerabilities. Our experimental results demonstrate that ActorBreaker
outperforms existing attack methods in terms of diversity, effectiveness, and
efficiency across aligned LLMs. To address this vulnerability, we propose
expanding safety training to cover a broader semantic space of toxic content.
We thus construct a multi-turn safety dataset using ActorBreaker. Fine-tuning
models on our dataset shows significant improvements in robustness, though with
some trade-offs in utility. Code is available at
https://github.com/AI45Lab/ActorAttack.


http://arxiv.org/abs/2410.10526
Generalized Adversarial Code-Suggestions: Exploiting Contexts of LLM-based Code-Completion. (15%)
Karl Rubel; Maximilian Noppel; Christian Wressnegger
  While convenient, relying on LLM-powered code assistants in day-to-day work
gives rise to severe attacks. For instance, the assistant might introduce
subtle flaws and suggest vulnerable code to the user. These adversarial
code-suggestions can be introduced via data poisoning and, thus, unknowingly by
the model creators. In this paper, we provide a generalized formulation of such
attacks, spawning and extending related work in this domain. This formulation
is defined over two components: First, a trigger pattern occurring in the
prompts of a specific user group, and, second, a learnable map in embedding
space from the prompt to an adversarial bait. The latter gives rise to novel
and more flexible targeted attack-strategies, allowing the adversary to choose
the most suitable trigger pattern for a specific user-group arbitrarily,
without restrictions on the pattern's tokens. Our directional-map attacks and
prompt-indexing attacks increase the stealthiness decisively. We extensively
evaluate the effectiveness of these attacks and carefully investigate defensive
mechanisms to explore the limits of generalized adversarial code-suggestions.
We find that most defenses unfortunately offer little protection only.


http://arxiv.org/abs/2410.10674
Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach. (13%)
Rory Young; Nicolas Pugeault
  Deep reinforcement learning agents achieve state-of-the-art performance in a
wide range of simulated control tasks. However, successful applications to
real-world problems remain limited. One reason for this dichotomy is because
the learned policies are not robust to observation noise or adversarial
attacks. In this paper, we investigate the robustness of deep RL policies to a
single small state perturbation in deterministic continuous control tasks. We
demonstrate that RL policies can be deterministically chaotic as small
perturbations to the system state have a large impact on subsequent state and
reward trajectories. This unstable non-linear behaviour has two consequences:
First, inaccuracies in sensor readings, or adversarial attacks, can cause
significant performance degradation; Second, even policies that show robust
performance in terms of rewards may have unpredictable behaviour in practice.
These two facets of chaos in RL policies drastically restrict the application
of deep RL to real-world problems. To address this issue, we propose an
improvement on the successful Dreamer V3 architecture, implementing a Maximal
Lyapunov Exponent regularisation. This new approach reduces the chaotic state
dynamics, rendering the learnt policies more resilient to sensor noise or
adversarial attacks and thereby improving the suitability of Deep Reinforcement
Learning for real-world applications.


http://arxiv.org/abs/2410.10473
The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels. (2%)
Yonatan Slutzky; Yotam Alexander; Noam Razin; Nadav Cohen
  Neural networks are powered by an implicit bias: a tendency of gradient
descent to fit training data in a way that generalizes to unseen data. A recent
class of neural network models gaining increasing popularity is structured
state space models (SSMs), regarded as an efficient alternative to
transformers. Prior work argued that the implicit bias of SSMs leads to
generalization in a setting where data is generated by a low dimensional
teacher. In this paper, we revisit the latter setting, and formally establish a
phenomenon entirely undetected by prior work on the implicit bias of SSMs.
Namely, we prove that while implicit bias leads to generalization under many
choices of training data, there exist special examples whose inclusion in
training completely distorts the implicit bias, to a point where generalization
fails. This failure occurs despite the special training examples being labeled
by the teacher, i.e. having clean labels! We empirically demonstrate the
phenomenon, with SSMs trained independently and as part of non-linear neural
networks. In the area of adversarial machine learning, disrupting
generalization with cleanly labeled training examples is known as clean-label
poisoning. Given the proliferation of SSMs, particularly in large language
models, we believe significant efforts should be invested in further
delineating their susceptibility to clean-label poisoning, and in developing
methods for overcoming this susceptibility.


http://arxiv.org/abs/2410.10796
Context-Parametric Inversion: Why Instruction Finetuning May Not Actually Improve Context Reliance. (1%)
Sachin Goyal; Christina Baek; J. Zico Kolter; Aditi Raghunathan
  Large language models are instruction-finetuned to enhance their ability to
follow user instructions and process the input context. However, even
state-of-the-art models often struggle to follow the instruction, especially
when the input context is not aligned with the model's parametric knowledge.
This manifests as various failures, such as hallucinations where the responses
are outdated, biased or contain unverified facts. In this work, we try to
understand the underlying reason for this poor context reliance, especially
after instruction tuning. We observe an intriguing phenomenon: during
instruction tuning, the context reliance initially increases as expected, but
then gradually decreases as instruction finetuning progresses. We call this
phenomenon context-parametric inversion and observe it across multiple general
purpose instruction tuning datasets like TULU, Alpaca and Ultrachat, as well as
model families such as Llama, Mistral and Pythia. In a simple theoretical
setup, we isolate why context-parametric inversion occurs along the gradient
descent trajectory of instruction finetuning. We tie this phenomena to examples
in the instruction finetuning data mixture where the input context provides
information that is already present in the model's parametric knowledge. Our
analysis suggests natural mitigation strategies that provide some limited
gains, while also validating our theoretical insights. We hope that our work
serves as a starting point in addressing this failure mode in a staple part of
LLM training.


http://arxiv.org/abs/2410.11242
Automatically Generating Visual Hallucination Test Cases for Multimodal Large Language Models. (1%)
Zhongye Liu; Hongbin Liu; Yuepeng Hu; Zedian Shao; Neil Zhenqiang Gong
  Visual hallucination (VH) occurs when a multimodal large language model
(MLLM) generates responses with incorrect visual details for prompts. Existing
methods for generating VH test cases primarily rely on human annotations,
typically in the form of triples: (image, question, answer). In this paper, we
introduce VHExpansion, the first automated method for expanding VH test cases
for MLLMs. Given an initial VH test case, VHExpansion automatically expands it
by perturbing the question and answer through negation as well as modifying the
image using both common and adversarial perturbations. Additionally, we propose
a new evaluation metric, symmetric accuracy, which measures the proportion of
correctly answered VH test-case pairs. Each pair consists of a test case and
its negated counterpart. Our theoretical analysis shows that symmetric accuracy
is an unbiased evaluation metric that remains unaffected by the imbalance of VH
testing cases with varying answers when an MLLM is randomly guessing the
answers, whereas traditional accuracy is prone to such imbalance. We apply
VHExpansion to expand three VH datasets annotated manually and use these
expanded datasets to benchmark seven MLLMs. Our evaluation shows that
VHExpansion effectively identifies more VH test cases. Moreover, symmetric
accuracy, being unbiased, leads to different conclusions about the
vulnerability of MLLMs to VH compared to traditional accuracy metric. Finally,
we show that fine-tuning MLLMs on the expanded VH dataset generated by
VHExpansion mitigates VH more effectively than fine-tuning on the original,
manually annotated dataset. Our code is available at:
https://github.com/lycheeefish/VHExpansion.


http://arxiv.org/abs/2410.10414
On Calibration of LLM-based Guard Models for Reliable Content Moderation. (1%)
Hongfu Liu; Hengguan Huang; Xiangming Gu; Hao Wang; Ye Wang
  Large language models (LLMs) pose significant risks due to the potential for
generating harmful content or users attempting to evade guardrails. Existing
studies have developed LLM-based guard models designed to moderate the input
and output of threat LLMs, ensuring adherence to safety policies by blocking
content that violates these protocols upon deployment. However, limited
attention has been given to the reliability and calibration of such guard
models. In this work, we empirically conduct comprehensive investigations of
confidence calibration for 9 existing LLM-based guard models on 12 benchmarks
in both user input and model output classification. Our findings reveal that
current LLM-based guard models tend to 1) produce overconfident predictions, 2)
exhibit significant miscalibration when subjected to jailbreak attacks, and 3)
demonstrate limited robustness to the outputs generated by different types of
response models. Additionally, we assess the effectiveness of post-hoc
calibration methods to mitigate miscalibration. We demonstrate the efficacy of
temperature scaling and, for the first time, highlight the benefits of
contextual calibration for confidence calibration of guard models, particularly
in the absence of validation sets. Our analysis and experiments underscore the
limitations of current LLM-based guard models and provide valuable insights for
the future development of well-calibrated guard models toward more reliable
content moderation. We also advocate for incorporating reliability evaluation
of confidence calibration when releasing future LLM-based guard models.


http://arxiv.org/abs/2410.10572
Regularized Robustly Reliable Learners and Instance Targeted Attacks. (1%)
Avrim Blum; Donya Saless
  Instance-targeted data poisoning attacks, where an adversary corrupts a
training set to induce errors on specific test points, have raised significant
concerns. Balcan et al (2022) proposed an approach to addressing this challenge
by defining a notion of robustly-reliable learners that provide per-instance
guarantees of correctness under well-defined assumptions, even in the presence
of data poisoning attacks. They then give a generic optimal (but
computationally inefficient) robustly reliable learner as well as a
computationally efficient algorithm for the case of linear separators over
log-concave distributions.
  In this work, we address two challenges left open by Balcan et al (2022). The
first is that the definition of robustly-reliable learners in Balcan et al
(2022) becomes vacuous for highly-flexible hypothesis classes: if there are two
classifiers h_0, h_1 \in H both with zero error on the training set such that
h_0(x) \neq h_1(x), then a robustly-reliable learner must abstain on x. We
address this problem by defining a modified notion of regularized
robustly-reliable learners that allows for nontrivial statements in this case.
The second is that the generic algorithm of Balcan et al (2022) requires
re-running an ERM oracle (essentially, retraining the classifier) on each test
point x, which is generally impractical even if ERM can be implemented
efficiently. To tackle this problem, we show that at least in certain
interesting cases we can design algorithms that can produce their outputs in
time sublinear in training time, by using techniques from dynamic algorithm
design.


http://arxiv.org/abs/2410.13891
S$^4$ST: A Strong, Self-transferable, faSt, and Simple Scale Transformation for Transferable Targeted Attack. (99%)
Yongxiang Liu; Bowen Peng; Li Liu; Xiang Li
  Transferable targeted adversarial attacks (TTAs) against deep neural networks
have been proven significantly more challenging than untargeted ones, yet they
remain relatively underexplored. This paper sheds new light on performing
highly efficient yet transferable targeted attacks leveraging the simple
gradient-based baseline. Our research underscores the critical importance of
image transformations within gradient calculations, marking a shift from the
prevalent emphasis on loss functions to address the gradient vanishing problem.
Moreover, we have developed two effective blind estimators that facilitate the
design of transformation strategies to enhance targeted transferability under
black-box conditions. The adversarial examples' self-transferability to
geometric transformations has been identified as strongly correlated with their
black-box transferability, featuring these basic operations as potent yet
overlapped proxies for facilitating targeted transferability. The surrogate
self-alignment assessments further highlight simple scaling transformation's
exceptional efficacy, which rivals that of most advanced methods. Building on
these insights, we introduce a scaling-centered transformation strategy termed
Strong, Self-transferable, faSt, and Simple Scale Transformation (S4ST) to
enhance transferable targeted attacks. In experiments conducted on the
ImageNet-Compatible benchmark dataset, our proposed S4ST attains a SOTA average
targeted transfer success rate across various challenging black-box models,
outperforming the previous leading method by over 14% while requiring only 25%
of the execution time. Additionally, our approach eclipses SOTA attacks
considerably and exhibits remarkable effectiveness against real-world APIs.
This work marks a significant leap forward in TTAs, revealing the realistic
threats they pose and providing a practical generation method for future
research.


http://arxiv.org/abs/2410.09845
Understanding Robustness of Parameter-Efficient Tuning for Image Classification. (98%)
Jiacheng Ruan; Xian Gao; Suncheng Xiang; Mingye Xie; Ting Liu; Yuzhuo Fu
  Parameter-efficient tuning (PET) techniques calibrate the model's predictions
on downstream tasks by freezing the pre-trained models and introducing a small
number of learnable parameters. However, despite the numerous PET methods
proposed, their robustness has not been thoroughly investigated. In this paper,
we systematically explore the robustness of four classical PET techniques
(e.g., VPT, Adapter, AdaptFormer, and LoRA) under both white-box attacks and
information perturbations. For white-box attack scenarios, we first analyze the
performance of PET techniques using FGSM and PGD attacks. Subsequently, we
further explore the transferability of adversarial samples and the impact of
learnable parameter quantities on the robustness of PET methods. Under
information perturbation attacks, we introduce four distinct perturbation
strategies, including Patch-wise Drop, Pixel-wise Drop, Patch Shuffle, and
Gaussian Noise, to comprehensively assess the robustness of these PET
techniques in the presence of information loss. Via these extensive studies, we
enhance the understanding of the robustness of PET methods, providing valuable
insights for improving their performance in computer vision applications. The
code is available at https://github.com/JCruan519/PETRobustness.


http://arxiv.org/abs/2410.10091
Out-of-Bounding-Box Triggers: A Stealthy Approach to Cheat Object Detectors. (75%)
Tao Lin; Lijia Yu; Gaojie Jin; Renjue Li; Peng Wu; Lijun Zhang
  In recent years, the study of adversarial robustness in object detection
systems, particularly those based on deep neural networks (DNNs), has become a
pivotal area of research. Traditional physical attacks targeting object
detectors, such as adversarial patches and texture manipulations, directly
manipulate the surface of the object. While these methods are effective, their
overt manipulation of objects may draw attention in real-world applications. To
address this, this paper introduces a more subtle approach: an inconspicuous
adversarial trigger that operates outside the bounding boxes, rendering the
object undetectable to the model. We further enhance this approach by proposing
the Feature Guidance (FG) technique and the Universal Auto-PGD (UAPGD)
optimization strategy for crafting high-quality triggers. The effectiveness of
our method is validated through extensive empirical testing, demonstrating its
high performance in both digital and physical environments. The code and video
will be available at: https://github.com/linToTao/Out-of-bbox-attack.


http://arxiv.org/abs/2410.09838
Uncovering, Explaining, and Mitigating the Superficial Safety of Backdoor Defense. (67%)
Rui Min; Zeyu Qin; Nevin L. Zhang; Li Shen; Minhao Cheng
  Backdoor attacks pose a significant threat to Deep Neural Networks (DNNs) as
they allow attackers to manipulate model predictions with backdoor triggers. To
address these security vulnerabilities, various backdoor purification methods
have been proposed to purify compromised models. Typically, these purified
models exhibit low Attack Success Rates (ASR), rendering them resistant to
backdoored inputs. However, Does achieving a low ASR through current safety
purification methods truly eliminate learned backdoor features from the
pretraining phase? In this paper, we provide an affirmative answer to this
question by thoroughly investigating the Post-Purification Robustness of
current backdoor purification methods. We find that current safety purification
methods are vulnerable to the rapid re-learning of backdoor behavior, even when
further fine-tuning of purified models is performed using a very small number
of poisoned samples. Based on this, we further propose the practical
Query-based Reactivation Attack (QRA) which could effectively reactivate the
backdoor by merely querying purified models. We find the failure to achieve
satisfactory post-purification robustness stems from the insufficient deviation
of purified models from the backdoored model along the backdoor-connected path.
To improve the post-purification robustness, we propose a straightforward
tuning defense, Path-Aware Minimization (PAM), which promotes deviation along
backdoor-connected paths with extra model updates. Extensive experiments
demonstrate that PAM significantly improves post-purification robustness while
maintaining a good clean accuracy and low ASR. Our work provides a new
perspective on understanding the effectiveness of backdoor safety tuning and
highlights the importance of faithfully assessing the model's safety.


http://arxiv.org/abs/2410.09804
BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models. (13%)
Xinyuan Wang; Victor Shea-Jay Huang; Renmiao Chen; Hao Wang; Chengwei Pan; Lei Sha; Minlie Huang
  While large language models (LLMs) exhibit remarkable capabilities across
various tasks, they encounter potential security risks such as jailbreak
attacks, which exploit vulnerabilities to bypass security measures and generate
harmful outputs. Existing jailbreak strategies mainly focus on maximizing
attack success rate (ASR), frequently neglecting other critical factors,
including the relevance of the jailbreak response to the query and the level of
stealthiness. This narrow focus on single objectives can result in ineffective
attacks that either lack contextual relevance or are easily recognizable. In
this work, we introduce BlackDAN, an innovative black-box attack framework with
multi-objective optimization, aiming to generate high-quality prompts that
effectively facilitate jailbreaking while maintaining contextual relevance and
minimizing detectability. BlackDAN leverages Multiobjective Evolutionary
Algorithms (MOEAs), specifically the NSGA-II algorithm, to optimize jailbreaks
across multiple objectives including ASR, stealthiness, and semantic relevance.
By integrating mechanisms like mutation, crossover, and Pareto-dominance,
BlackDAN provides a transparent and interpretable process for generating
jailbreaks. Furthermore, the framework allows customization based on user
preferences, enabling the selection of prompts that balance harmfulness,
relevance, and other factors. Experimental results demonstrate that BlackDAN
outperforms traditional single-objective methods, yielding higher success rates
and improved robustness across various LLMs and multimodal LLMs, while ensuring
jailbreak responses are both relevant and less detectable.


http://arxiv.org/abs/2410.09760
Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation. (1%)
Guozhi Liu; Weiwei Lin; Tiansheng Huang; Ruichao Mo; Qi Mu; Li Shen
  Harmful fine-tuning attack poses a serious threat to the online fine-tuning
service. Vaccine, a recent alignment-stage defense, applies uniform
perturbation to all layers of embedding to make the model robust to the
simulated embedding drift. However, applying layer-wise uniform perturbation
may lead to excess perturbations for some particular safety-irrelevant layers,
resulting in defense performance degradation and unnecessary memory
consumption. To address this limitation, we propose Targeted Vaccine
(T-Vaccine), a memory-efficient safety alignment method that applies
perturbation to only selected layers of the model. T-Vaccine follows two core
steps: First, it uses gradient norm as a statistical metric to identify the
safety-critical layers. Second, instead of applying uniform perturbation across
all layers, T-Vaccine only applies perturbation to the safety-critical layers
while keeping other layers frozen during training. Results show that T-Vaccine
outperforms Vaccine in terms of both defense effectiveness and resource
efficiency. Comparison with other defense baselines, e.g., RepNoise and TAR
also demonstrate the superiority of T-Vaccine. Notably, T-Vaccine is the first
defense that can address harmful fine-tuning issues for a 7B pre-trained models
trained on consumer GPUs with limited memory (e.g., RTX 4090). Our code is
available at https://github.com/Lslland/T-Vaccine.


http://arxiv.org/abs/2410.09591
Unlearn and Burn: Adversarial Machine Unlearning Requests Destroy Model Accuracy. (91%)
Yangsibo Huang; Daogao Liu; Lynn Chua; Badih Ghazi; Pritish Kamath; Ravi Kumar; Pasin Manurangsi; Milad Nasr; Amer Sinha; Chiyuan Zhang
  Machine unlearning algorithms, designed for selective removal of training
data from models, have emerged as a promising approach to growing privacy
concerns. In this work, we expose a critical yet underexplored vulnerability in
the deployment of unlearning systems: the assumption that the data requested
for removal is always part of the original training set. We present a threat
model where an attacker can degrade model accuracy by submitting adversarial
unlearning requests for data not present in the training set. We propose
white-box and black-box attack algorithms and evaluate them through a case
study on image classification tasks using the CIFAR-10 and ImageNet datasets,
targeting a family of widely used unlearning methods. Our results show
extremely poor test accuracy following the attack: 3.6% on CIFAR-10 and 0.4% on
ImageNet for white-box attacks, and 8.5% on CIFAR-10 and 1.3% on ImageNet for
black-box attacks. Additionally, we evaluate various verification mechanisms to
detect the legitimacy of unlearning requests and reveal the challenges in
verification, as most of the mechanisms fail to detect stealthy attacks without
severely impairing their ability to process valid requests. These findings
underscore the urgent need for research on more robust request verification
methods and unlearning protocols, should the deployment of machine unlearning
systems become more prevalent in the future.


http://arxiv.org/abs/2410.09691
Robust 3D Point Clouds Classification based on Declarative Defenders. (2%)
Kaidong Li; Tianxiao Zhang; Cuncong Zhong; Ziming Zhang; Guanghui Wang
  3D point cloud classification requires distinct models from 2D image
classification due to the divergent characteristics of the respective input
data. While 3D point clouds are unstructured and sparse, 2D images are
structured and dense. Bridging the domain gap between these two data types is a
non-trivial challenge to enable model interchangeability. Recent research using
Lattice Point Classifier (LPC) highlights the feasibility of cross-domain
applicability. However, the lattice projection operation in LPC generates 2D
images with disconnected projected pixels. In this paper, we explore three
distinct algorithms for mapping 3D point clouds into 2D images. Through
extensive experiments, we thoroughly examine and analyze their performance and
defense mechanisms. Leveraging current large foundation models, we scrutinize
the feature disparities between regular 2D images and projected 2D images. The
proposed approaches demonstrate superior accuracy and robustness against
adversarial attacks. The generative model-based mapping algorithms yield
regular 2D images, further minimizing the domain gap from regular 2D
classification tasks. The source code is available at
https://github.com/KaidongLi/pytorch-LatticePointClassifier.git.


http://arxiv.org/abs/2410.08950
On the Adversarial Transferability of Generalized "Skip Connections". (99%)
Yisen Wang; Yichuan Mo; Dongxian Wu; Mingjie Li; Xingjun Ma; Zhouchen Lin
  Skip connection is an essential ingredient for modern deep models to be
deeper and more powerful. Despite their huge success in normal scenarios
(state-of-the-art classification performance on natural examples), we
investigate and identify an interesting property of skip connections under
adversarial scenarios, namely, the use of skip connections allows easier
generation of highly transferable adversarial examples. Specifically, in
ResNet-like models (with skip connections), we find that using more gradients
from the skip connections rather than the residual modules according to a decay
factor during backpropagation allows one to craft adversarial examples with
high transferability. The above method is termed as Skip Gradient Method (SGM).
Although starting from ResNet-like models in vision domains, we further extend
SGM to more advanced architectures, including Vision Transformers (ViTs) and
models with length-varying paths and other domains, i.e. natural language
processing. We conduct comprehensive transfer attacks against various models
including ResNets, Transformers, Inceptions, Neural Architecture Search, and
Large Language Models (LLMs). We show that employing SGM can greatly improve
the transferability of crafted attacks in almost all cases. Furthermore,
considering the big complexity for practical use, we further demonstrate that
SGM can even improve the transferability on ensembles of models or targeted
attacks and the stealthiness against current defenses. At last, we provide
theoretical explanations and empirical insights on how SGM works. Our findings
not only motivate new adversarial research into the architectural
characteristics of models but also open up further challenges for secure model
architecture design. Our code is available at https://github.com/mo666666/SGM.


http://arxiv.org/abs/2410.08620
Natural Language Induced Adversarial Images. (99%)
Xiaopei Zhu; Peiyang Xu; Guanning Zeng; Yingpeng Dong; Xiaolin Hu
  Research of adversarial attacks is important for AI security because it shows
the vulnerability of deep learning models and helps to build more robust
models. Adversarial attacks on images are most widely studied, which include
noise-based attacks, image editing-based attacks, and latent space-based
attacks. However, the adversarial examples crafted by these methods often lack
sufficient semantic information, making it challenging for humans to understand
the failure modes of deep learning models under natural conditions. To address
this limitation, we propose a natural language induced adversarial image attack
method. The core idea is to leverage a text-to-image model to generate
adversarial images given input prompts, which are maliciously constructed to
lead to misclassification for a target model. To adopt commercial text-to-image
models for synthesizing more natural adversarial images, we propose an adaptive
genetic algorithm (GA) for optimizing discrete adversarial prompts without
requiring gradients and an adaptive word space reduction method for improving
query efficiency. We further used CLIP to maintain the semantic consistency of
the generated images. In our experiments, we found that some high-frequency
semantic information such as "foggy", "humid", "stretching", etc. can easily
cause classifier errors. This adversarial semantic information exists not only
in generated images but also in photos captured in the real world. We also
found that some adversarial semantic information can be transferred to unknown
classification tasks. Furthermore, our attack method can transfer to different
text-to-image models (e.g., Midjourney, DALL-E 3, etc.) and image classifiers.
Our code is available at:
https://github.com/zxp555/Natural-Language-Induced-Adversarial-Images.


http://arxiv.org/abs/2410.08872
Fragile Giants: Understanding the Susceptibility of Models to Subpopulation Attacks. (70%)
Isha Gupta; Hidde Lycklama; Emanuel Opel; Evan Rose; Anwar Hithnawi
  As machine learning models become increasingly complex, concerns about their
robustness and trustworthiness have become more pressing. A critical
vulnerability of these models is data poisoning attacks, where adversaries
deliberately alter training data to degrade model performance. One particularly
stealthy form of these attacks is subpopulation poisoning, which targets
distinct subgroups within a dataset while leaving overall performance largely
intact. The ability of these attacks to generalize within subpopulations poses
a significant risk in real-world settings, as they can be exploited to harm
marginalized or underrepresented groups within the dataset. In this work, we
investigate how model complexity influences susceptibility to subpopulation
poisoning attacks. We introduce a theoretical framework that explains how
overparameterized models, due to their large capacity, can inadvertently
memorize and misclassify targeted subpopulations. To validate our theory, we
conduct extensive experiments on large-scale image and text datasets using
popular model architectures. Our results show a clear trend: models with more
parameters are significantly more vulnerable to subpopulation poisoning.
Moreover, we find that attacks on smaller, human-interpretable subgroups often
go undetected by these models. These results highlight the need to develop
defenses that specifically address subpopulation vulnerabilities.


http://arxiv.org/abs/2410.09024
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. (69%)
Maksym Andriushchenko; Alexandra Souly; Mateusz Dziemian; Derek Duenas; Maxwell Lin; Justin Wang; Dan Hendrycks; Andy Zou; Zico Kolter; Matt Fredrikson; Eric Winsor; Jerome Wynne; Yarin Gal; Xander Davies
  The robustness of LLMs to jailbreak attacks, where users design prompts to
circumvent safety measures and misuse model capabilities, has been studied
primarily for LLMs acting as simple chatbots. Meanwhile, LLM agents -- which
use external tools and can execute multi-stage tasks -- may pose a greater risk
if misused, but their robustness remains underexplored. To facilitate research
on LLM agent misuse, we propose a new benchmark called AgentHarm. The benchmark
includes a diverse set of 110 explicitly malicious agent tasks (440 with
augmentations), covering 11 harm categories including fraud, cybercrime, and
harassment. In addition to measuring whether models refuse harmful agentic
requests, scoring well on AgentHarm requires jailbroken agents to maintain
their capabilities following an attack to complete a multi-step task. We
evaluate a range of leading LLMs, and find (1) leading LLMs are surprisingly
compliant with malicious agent requests without jailbreaking, (2) simple
universal jailbreak templates can be adapted to effectively jailbreak agents,
and (3) these jailbreaks enable coherent and malicious multi-step agent
behavior and retain model capabilities. To enable simple and reliable
evaluation of attacks and defenses for LLM-based agents, we publicly release
AgentHarm at https://huggingface.co/datasets/ai-safety-institute/AgentHarm.


http://arxiv.org/abs/2410.09125
Training on Fake Labels: Mitigating Label Leakage in Split Learning via Secure Dimension Transformation. (62%)
Yukun Jiang; Peiran Wang; Chengguo Lin; Ziyue Huang; Yong Cheng
  Two-party split learning has emerged as a popular paradigm for vertical
federated learning. To preserve the privacy of the label owner, split learning
utilizes a split model, which only requires the exchange of intermediate
representations (IRs) based on the inputs and gradients for each IR between two
parties during the learning process. However, split learning has recently been
proven to survive label inference attacks. Though several defense methods could
be adopted, they either have limited defensive performance or significantly
negatively impact the original mission. In this paper, we propose a novel
two-party split learning method to defend against existing label inference
attacks while maintaining the high utility of the learned models. Specifically,
we first craft a dimension transformation module, SecDT, which could achieve
bidirectional mapping between original labels and increased K-class labels to
mitigate label leakage from the directional perspective. Then, a gradient
normalization algorithm is designed to remove the magnitude divergence of
gradients from different classes. We propose a softmax-normalized Gaussian
noise to mitigate privacy leakage and make our K unknowable to adversaries. We
conducted experiments on real-world datasets, including two
binary-classification datasets (Avazu and Criteo) and three
multi-classification datasets (MNIST, FashionMNIST, CIFAR-10); we also
considered current attack schemes, including direction, norm, spectral, and
model completion attacks. The detailed experiments demonstrate our proposed
method's effectiveness and superiority over existing approaches. For instance,
on the Avazu dataset, the attack AUC of evaluated four prominent attacks could
be reduced by 0.4532+-0.0127.


http://arxiv.org/abs/2410.08811
PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning. (31%)
Tingchen Fu; Mrinank Sharma; Philip Torr; Shay B. Cohen; David Krueger; Fazl Barez
  Preference learning is a central component for aligning current LLMs, but
this process can be vulnerable to data poisoning attacks. To address this
concern, we introduce PoisonBench, a benchmark for evaluating large language
models' susceptibility to data poisoning during preference learning. Data
poisoning attacks can manipulate large language model responses to include
hidden malicious content or biases, potentially causing the model to generate
harmful or unintended outputs while appearing to function normally. We deploy
two distinct attack types across eight realistic scenarios, assessing 21
widely-used models. Our findings reveal concerning trends: (1) Scaling up
parameter size does not inherently enhance resilience against poisoning
attacks; (2) There exists a log-linear relationship between the effects of the
attack and the data poison ratio; (3) The effect of data poisoning can
generalize to extrapolated triggers that are not included in the poisoned data.
These results expose weaknesses in current preference learning techniques,
highlighting the urgent need for more robust defenses against malicious models
and data manipulation.


http://arxiv.org/abs/2410.09040
AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation. (31%)
Zijun Wang; Haoqin Tu; Jieru Mei; Bingchen Zhao; Yisen Wang; Cihang Xie
  This paper studies the vulnerabilities of transformer-based Large Language
Models (LLMs) to jailbreaking attacks, focusing specifically on the
optimization-based Greedy Coordinate Gradient (GCG) strategy. We first observe
a positive correlation between the effectiveness of attacks and the internal
behaviors of the models. For instance, attacks tend to be less effective when
models pay more attention to system prompts designed to ensure LLM safety
alignment. Building on this discovery, we introduce an enhanced method that
manipulates models' attention scores to facilitate LLM jailbreaking, which we
term AttnGCG. Empirically, AttnGCG shows consistent improvements in attack
efficacy across diverse LLMs, achieving an average increase of ~7% in the
Llama-2 series and ~10% in the Gemma series. Our strategy also demonstrates
robust attack transferability against both unseen harmful goals and black-box
LLMs like GPT-3.5 and GPT-4. Moreover, we note our attention-score
visualization is more interpretable, allowing us to gain better insights into
how our targeted attention manipulation facilitates more effective
jailbreaking. We release the code at
https://github.com/UCSC-VLAA/AttnGCG-attack.


http://arxiv.org/abs/2410.08864
The Good, the Bad and the Ugly: Watermarks, Transferable Attacks and Adversarial Defenses. (16%)
Grzegorz Głuch; Berkant Turan; Sai Ganesh Nagarajan; Sebastian Pokutta
  We formalize and extend existing definitions of backdoor-based watermarks and
adversarial defenses as interactive protocols between two players. The
existence of these schemes is inherently tied to the learning tasks for which
they are designed. Our main result shows that for almost every discriminative
learning task, at least one of the two -- a watermark or an adversarial defense
-- exists. The term "almost every" indicates that we also identify a third,
counterintuitive but necessary option, i.e., a scheme we call a transferable
attack. By transferable attack, we refer to an efficient algorithm computing
queries that look indistinguishable from the data distribution and fool all
efficient defenders. To this end, we prove the necessity of a transferable
attack via a construction that uses a cryptographic tool called homomorphic
encryption. Furthermore, we show that any task that satisfies our notion of a
transferable attack implies a cryptographic primitive, thus requiring the
underlying task to be computationally complex. These two facts imply an
"equivalence" between the existence of transferable attacks and cryptography.
Finally, we show that the class of tasks of bounded VC-dimension has an
adversarial defense, and a subclass of them has a watermark.


http://arxiv.org/abs/2410.09318
Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation. (4%)
Saiful Islam Salim; Rubin Yuchan Yang; Alexander Cooper; Suryashree Ray; Saumya Debray; Sazzadur Rahaman
  While Large language model (LLM)-based programming assistants such as CoPilot
and ChatGPT can help improve the productivity of professional software
developers, they can also facilitate cheating in introductory computer
programming courses. Assuming instructors have limited control over the
industrial-strength models, this paper investigates the baseline performance of
5 widely used LLMs on a collection of introductory programming problems,
examines adversarial perturbations to degrade their performance, and describes
the results of a user study aimed at understanding the efficacy of such
perturbations in hindering actual code generation for introductory programming
assignments. The user study suggests that i) perturbations combinedly reduced
the average correctness score by 77%, ii) the drop in correctness caused by
these perturbations was affected based on their detectability.


http://arxiv.org/abs/2410.09186
AI Learning Algorithms: Deep Learning, Hybrid Models, and Large-Scale Model Integration. (1%)
Noorbakhsh Amiri Golilarz; Elias Hossain; Abdoljalil Addeh; Keyan Alexander Rahimi
  In this paper, we discuss learning algorithms and their importance in
different types of applications which includes training to identify important
patterns and features in a straightforward, easy-to-understand manner. We will
review the main concepts of artificial intelligence (AI), machine learning
(ML), deep learning (DL), and hybrid models. Some important subsets of Machine
Learning algorithms such as supervised, unsupervised, and reinforcement
learning are also discussed in this paper. These techniques can be used for
some important tasks like prediction, classification, and segmentation.
Convolutional Neural Networks (CNNs) are used for image and video processing
and many more applications. We dive into the architecture of CNNs and how to
integrate CNNs with ML algorithms to build hybrid models. This paper explores
the vulnerability of learning algorithms to noise, leading to
misclassification. We further discuss the integration of learning algorithms
with Large Language Models (LLM) to generate coherent responses applicable to
many domains such as healthcare, marketing, and finance by learning important
patterns from large volumes of data. Furthermore, we discuss the next
generation of learning algorithms and how we may have an unified Adaptive and
Dynamic Network to perform important tasks. Overall, this article provides
brief overview of learning algorithms, exploring their current state,
applications and future direction.


http://arxiv.org/abs/2410.08776
F2A: An Innovative Approach for Prompt Injection by Utilizing Feign Security Detection Agents. (1%)
Yupeng Ren
  With the rapid development of Large Language Models (LLMs), numerous mature
applications of LLMs have emerged in the field of content safety detection.
However, we have found that LLMs exhibit blind trust in safety detection
agents. The general LLMs can be compromised by hackers with this vulnerability.
Hence, this paper proposed an attack named Feign Agent Attack (F2A).Through
such malicious forgery methods, adding fake safety detection results into the
prompt, the defense mechanism of LLMs can be bypassed, thereby obtaining
harmful content and hijacking the normal conversation.Continually, a series of
experiments were conducted. In these experiments, the hijacking capability of
F2A on LLMs was analyzed and demonstrated, exploring the fundamental reasons
why LLMs blindly trust safety detection results. The experiments involved
various scenarios where fake safety detection results were injected into
prompts, and the responses were closely monitored to understand the extent of
the vulnerability. Also, this paper provided a reasonable solution to this
attack, emphasizing that it is important for LLMs to critically evaluate the
results of augmented agents to prevent the generating harmful content. By doing
so, the reliability and security can be significantly improved, protecting the
LLMs from F2A.


http://arxiv.org/abs/2410.08660
RePD: Defending Jailbreak Attack through a Retrieval-based Prompt Decomposition Process. (1%)
Peiran Wang; Xiaogeng Liu; Chaowei Xiao
  In this study, we introduce RePD, an innovative attack Retrieval-based Prompt
Decomposition framework designed to mitigate the risk of jailbreak attacks on
large language models (LLMs). Despite rigorous pretraining and finetuning
focused on ethical alignment, LLMs are still susceptible to jailbreak exploits.
RePD operates on a one-shot learning model, wherein it accesses a database of
pre-collected jailbreak prompt templates to identify and decompose harmful
inquiries embedded within user prompts. This process involves integrating the
decomposition of the jailbreak prompt into the user's original query into a
one-shot learning example to effectively teach the LLM to discern and separate
malicious components. Consequently, the LLM is equipped to first neutralize any
potentially harmful elements before addressing the user's prompt in a manner
that aligns with its ethical guidelines. RePD is versatile and compatible with
a variety of open-source LLMs acting as agents. Through comprehensive
experimentation with both harmful and benign prompts, we have demonstrated the
efficacy of our proposed RePD in enhancing the resilience of LLMs against
jailbreak attacks, without compromising their performance in responding to
typical user requests.


http://arxiv.org/abs/2410.12855
JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Framework. (1%)
Fan Liu; Yue Feng; Zhao Xu; Lixin Su; Xinyu Ma; Dawei Yin; Hao Liu
  Despite advancements in enhancing LLM safety against jailbreak attacks,
evaluating LLM defenses remains a challenge, with current methods often lacking
explainability and generalization to complex scenarios, leading to incomplete
assessments (e.g., direct judgment without reasoning, low F1 score of GPT-4 in
complex cases, bias in multilingual scenarios). To address this, we present
JAILJUDGE, a comprehensive benchmark featuring diverse risk scenarios,
including synthetic, adversarial, in-the-wild, and multilingual prompts, along
with high-quality human-annotated datasets. The JAILJUDGE dataset includes over
35k+ instruction-tune data with reasoning explainability and JAILJUDGETEST, a
4.5k+ labeled set for risk scenarios, and a 6k+ multilingual set across ten
languages. To enhance evaluation with explicit reasoning, we propose the
JailJudge MultiAgent framework, which enables explainable, fine-grained scoring
(1 to 10). This framework supports the construction of instruction-tuning
ground truth and facilitates the development of JAILJUDGE Guard, an end-to-end
judge model that provides reasoning and eliminates API costs. Additionally, we
introduce JailBoost, an attacker-agnostic attack enhancer, and GuardShield, a
moderation defense, both leveraging JAILJUDGE Guard. Our experiments
demonstrate the state-of-the-art performance of JailJudge methods (JailJudge
MultiAgent, JAILJUDGE Guard) across diverse models (e.g., GPT-4, Llama-Guard)
and zero-shot scenarios. JailBoost and GuardShield significantly improve
jailbreak attack and defense tasks under zero-shot settings, with JailBoost
enhancing performance by 29.24% and GuardShield reducing defense ASR from
40.46% to 0.15%.


http://arxiv.org/abs/2410.08338
Time Traveling to Defend Against Adversarial Example Attacks in Image Classification. (99%)
Anthony Etim; Jakub Szefer
  Adversarial example attacks have emerged as a critical threat to machine
learning. Adversarial attacks in image classification abuse various, minor
modifications to the image that confuse the image classification neural network
-- while the image still remains recognizable to humans. One important domain
where the attacks have been applied is in the automotive setting with traffic
sign classification. Researchers have demonstrated that adding stickers,
shining light, or adding shadows are all different means to make machine
learning inference algorithms mis-classify the traffic signs. This can cause
potentially dangerous situations as a stop sign is recognized as a speed limit
sign causing vehicles to ignore it and potentially leading to accidents. To
address these attacks, this work focuses on enhancing defenses against such
adversarial attacks. This work shifts the advantage to the user by introducing
the idea of leveraging historical images and majority voting. While the
attacker modifies a traffic sign that is currently being processed by the
victim's machine learning inference, the victim can gain advantage by examining
past images of the same traffic sign. This work introduces the notion of ''time
traveling'' and uses historical Street View images accessible to anybody to
perform inference on different, past versions of the same traffic sign. In the
evaluation, the proposed defense has 100% effectiveness against latest
adversarial example attack on traffic sign classification algorithm.


http://arxiv.org/abs/2410.08503
Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data. (99%)
Binghui Li; Yuanzhi Li
  Adversarial training is a widely-applied approach to training deep neural
networks to be robust against adversarial perturbation. However, although
adversarial training has achieved empirical success in practice, it still
remains unclear why adversarial examples exist and how adversarial training
methods improve model robustness. In this paper, we provide a theoretical
understanding of adversarial examples and adversarial training algorithms from
the perspective of feature learning theory. Specifically, we focus on a
multiple classification setting, where the structured data can be composed of
two types of features: the robust features, which are resistant to perturbation
but sparse, and the non-robust features, which are susceptible to perturbation
but dense. We train a two-layer smoothed ReLU convolutional neural network to
learn our structured data. First, we prove that by using standard training
(gradient descent over the empirical risk), the network learner primarily
learns the non-robust feature rather than the robust feature, which thereby
leads to the adversarial examples that are generated by perturbations aligned
with negative non-robust feature directions. Then, we consider the
gradient-based adversarial training algorithm, which runs gradient ascent to
find adversarial examples and runs gradient descent over the empirical risk at
adversarial examples to update models. We show that the adversarial training
method can provably strengthen the robust feature learning and suppress the
non-robust feature learning to improve the network robustness. Finally, we also
empirically validate our theoretical findings with experiments on real-image
datasets, including MNIST, CIFAR10 and SVHN.


http://arxiv.org/abs/2410.07719
Understanding Adversarially Robust Generalization via Weight-Curvature Index. (98%)
Yuelin Xu; Xiao Zhang
  Despite extensive research on adversarial examples, the underlying mechanisms
of adversarially robust generalization, a critical yet challenging task for
deep learning, remain largely unknown. In this work, we propose a novel
perspective to decipher adversarially robust generalization through the lens of
the Weight-Curvature Index (WCI). The proposed WCI quantifies the vulnerability
of models to adversarial perturbations using the Frobenius norm of weight
matrices and the trace of Hessian matrices. We prove generalization bounds
based on PAC-Bayesian theory and second-order loss function approximations to
elucidate the interplay between robust generalization gap, model parameters,
and loss landscape curvature. Our theory and experiments show that WCI
effectively captures the robust generalization performance of adversarially
trained models. By offering a nuanced understanding of adversarial robustness
based on the scale of model parameters and the curvature of the loss landscape,
our work provides crucial insights for designing more resilient deep learning
models, enhancing their reliability and security.


http://arxiv.org/abs/2410.07670
Invisibility Cloak: Disappearance under Human Pose Estimation via Backdoor Attacks. (92%)
Minxing Zhang; Michael Backes; Xiao Zhang
  Human Pose Estimation (HPE) has been widely applied in autonomous systems
such as self-driving cars. However, the potential risks of HPE to adversarial
attacks have not received comparable attention with image classification or
segmentation tasks. Existing works on HPE robustness focus on misleading an HPE
system to provide wrong predictions that still indicate some human poses. In
this paper, we study the vulnerability of HPE systems to disappearance attacks,
where the attacker aims to subtly alter the HPE training process via backdoor
techniques so that any input image with some specific trigger will not be
recognized as involving any human pose. As humans are typically at the center
of HPE systems, such attacks can induce severe security hazards, e.g.,
pedestrians' lives will be threatened if a self-driving car incorrectly
understands the front scene due to disappearance attacks.
  To achieve the adversarial goal of disappearance, we propose IntC, a general
framework to craft Invisibility Cloak in the HPE domain. The core of our work
lies in the design of target HPE labels that do not represent any human pose.
In particular, we propose three specific backdoor attacks based on our IntC
framework with different label designs. IntC-S and IntC-E, respectively
designed for regression- and heatmap-based HPE techniques, concentrate the
keypoints of triggered images in a tiny, imperceptible region. Further, to
improve the attack's stealthiness, IntC-L designs the target poisons to capture
the label outputs of typical landscape images without a human involved,
achieving disappearance and reducing detectability simultaneously. Extensive
experiments demonstrate the effectiveness and generalizability of our IntC
methods in achieving the disappearance goal. By revealing the vulnerability of
HPE to disappearance and backdoor attacks, we hope our work can raise awareness
of the potential risks ...


http://arxiv.org/abs/2410.16317
A Survey on Physical Adversarial Attacks against Face Recognition Systems. (91%)
Mingsi Wang; Jiachen Zhou; Tianlin Li; Guozhu Meng; Kai Chen
  As Face Recognition (FR) technology becomes increasingly prevalent in
finance, the military, public safety, and everyday life, security concerns have
grown substantially. Physical adversarial attacks targeting FR systems in
real-world settings have attracted considerable research interest due to their
practicality and the severe threats they pose. However, a systematic overview
focused on physical adversarial attacks against FR systems is still lacking,
hindering an in-depth exploration of the challenges and future directions in
this field. In this paper, we bridge this gap by comprehensively collecting and
analyzing physical adversarial attack methods targeting FR systems.
Specifically, we first investigate the key challenges of physical attacks on FR
systems. We then categorize existing physical attacks into three categories
based on the physical medium used and summarize how the research in each
category has evolved to address these challenges. Furthermore, we review
current defense strategies and discuss potential future research directions.
Our goal is to provide a fresh, comprehensive, and deep understanding of
physical adversarial attacks against FR systems, thereby inspiring relevant
research in this area.


http://arxiv.org/abs/2410.07962
Towards Assurance of LLM Adversarial Robustness using Ontology-Driven Argumentation. (74%)
Tomas Bueno Momcilovic; Beat Buesser; Giulio Zizzo; Mark Purcell; Dian Balta
  Despite the impressive adaptability of large language models (LLMs),
challenges remain in ensuring their security, transparency, and
interpretability. Given their susceptibility to adversarial attacks, LLMs need
to be defended with an evolving combination of adversarial training and
guardrails. However, managing the implicit and heterogeneous knowledge for
continuously assuring robustness is difficult. We introduce a novel approach
for assurance of the adversarial robustness of LLMs based on formal
argumentation. Using ontologies for formalization, we structure
state-of-the-art attacks and defenses, facilitating the creation of a
human-readable assurance case, and a machine-readable representation. We
demonstrate its application with examples in English language and code
translation tasks, and provide implications for theory and practice, by
targeting engineers, data scientists, users, and auditors.


http://arxiv.org/abs/2410.08010
Securing HHL Quantum Algorithm against Quantum Computer Attacks. (73%)
Yizhuo Tan; Hrvoje Kukina; Jakub Szefer
  As the quantum research community expands and new quantum algorithms are
created and implemented, it is essential to consider the security implications
and potential threats that could lead to the compromise the information
processed by them. This work focuses on securing the HHL quantum algorithm
against attacks while it executes on a quantum computer. Specifically, two
types of potential attacks could be deployed on a cloud-based quantum computer
by an attacker circuit attempting to interfere with the victim HHL circuit: the
Improper Initialization Attack (IIA) and the Higher Energy Attack (HEA). To
protect the HHL algorithm from IIA and HEA, this work proposes first-of-a-kind
defense strategies against these attacks on the HHL quantum algorithm. Next,
this work demonstrates an implementation of a new quantum circuit for the HHL
quantum algorithm that incorporates these defenses. The redesigned quantum
circuit is necessary to successfully apply and realize all proposed defense
strategies. Finally, this work illustrates how these defense strategies
function in practice in the redesigned circuit, specifically how they can
protect the HHL quantum algorithm from both IIA and HEA across multiple qubits
involving all three types of qubits used in the HHL algorithms: ancilla, clock,
and b. The defense requires minimal modification to the circuit, and has only a
very small effect on the fidelity of the circuits. The circuits have been
tested and validated in both simulation, and also on real IBM quantum computer
hardware. The work further analyzes how the modified HHL circuit with the
defenses is affected by noise during quantum computation. This work in the end
demonstrates that it is practical to add protections to quantum circuits so
that they not only perform correct computation, but also self-detect if an
attack has occured during the execution.


http://arxiv.org/abs/2410.08417
Bilinear MLPs enable weight-based mechanistic interpretability. (70%)
Michael T. Pearce; Thomas Dooms; Alice Rigg; Jose M. Oramas; Lee Sharkey
  A mechanistic understanding of how MLPs do computation in deep neural
networks remains elusive. Current interpretability work can extract features
from hidden activations over an input dataset but generally cannot explain how
MLP weights construct features. One challenge is that element-wise
nonlinearities introduce higher-order interactions and make it difficult to
trace computations through the MLP layer. In this paper, we analyze bilinear
MLPs, a type of Gated Linear Unit (GLU) without any element-wise nonlinearity
that nevertheless achieves competitive performance. Bilinear MLPs can be fully
expressed in terms of linear operations using a third-order tensor, allowing
flexible analysis of the weights. Analyzing the spectra of bilinear MLP weights
using eigendecomposition reveals interpretable low-rank structure across toy
tasks, image classification, and language modeling. We use this understanding
to craft adversarial examples, uncover overfitting, and identify small language
model circuits directly from the weights alone. Our results demonstrate that
bilinear layers serve as an interpretable drop-in replacement for current
activation functions and that weight-based interpretability is viable for
understanding deep-learning models.


http://arxiv.org/abs/2410.07675
Adversarial Robustness Overestimation and Instability in TRADES. (67%)
Jonathan Weiping Li; Ren-Wei Liang; Cheng-Han Yeh; Cheng-Chang Tsai; Kuanchun Yu; Chun-Shien Lu; Shang-Tse Chen
  This paper examines the phenomenon of probabilistic robustness overestimation
in TRADES, a prominent adversarial training method. Our study reveals that
TRADES sometimes yields disproportionately high PGD validation accuracy
compared to the AutoAttack testing accuracy in the multiclass classification
task. This discrepancy highlights a significant overestimation of robustness
for these instances, potentially linked to gradient masking. We further analyze
the parameters contributing to unstable models that lead to overestimation. Our
findings indicate that smaller batch sizes, lower beta values (which control
the weight of the robust loss term in TRADES), larger learning rates, and
higher class complexity (e.g., CIFAR-100 versus CIFAR-10) are associated with
an increased likelihood of robustness overestimation. By examining metrics such
as the First-Order Stationary Condition (FOSC), inner-maximization, and
gradient information, we identify the underlying cause of this phenomenon as
gradient masking and provide insights into it. Furthermore, our experiments
show that certain unstable training instances may return to a state without
robust overestimation, inspiring our attempts at a solution. In addition to
adjusting parameter settings to reduce instability or retraining when
overestimation occurs, we recommend incorporating Gaussian noise in inputs when
the FOSC score exceed the threshold. This method aims to mitigate robustness
overestimation of TRADES and other similar methods at its source, ensuring more
reliable representation of adversarial robustness during evaluation.


http://arxiv.org/abs/2410.08244
RAB$^2$-DEF: Dynamic and explainable defense against adversarial attacks in Federated Learning to fair poor clients. (61%)
Nuria Rodríguez-Barroso; M. Victoria Luzón; Francisco Herrera
  At the same time that artificial intelligence is becoming popular, concern
and the need for regulation is growing, including among other requirements the
data privacy. In this context, Federated Learning is proposed as a solution to
data privacy concerns derived from different source data scenarios due to its
distributed learning. The defense mechanisms proposed in literature are just
focused on defending against adversarial attacks and the performance, leaving
aside other important qualities such as explainability, fairness to poor
quality clients, dynamism in terms of attacks configuration and generality in
terms of being resilient against different kinds of attacks. In this work, we
propose RAB$^2$-DEF, a $\textbf{r}$esilient $\textbf{a}$gainst
$\textbf{b}\text{yzantine}$ and $\textbf{b}$ackdoor attacks which is
$\textbf{d}$ynamic, $\textbf{e}$xplainable and $\textbf{f}$air to poor clients
using local linear explanations. We test the performance of RAB$^2$-DEF in
image datasets and both byzantine and backdoor attacks considering the
state-of-the-art defenses and achieve that RAB$^2$-DEF is a proper defense at
the same time that it boosts the other qualities towards trustworthy artificial
intelligence.


http://arxiv.org/abs/2410.08190
Poison-splat: Computation Cost Attack on 3D Gaussian Splatting. (10%)
Jiahao Lu; Yifan Zhang; Qiuhong Shen; Xinchao Wang; Shuicheng Yan
  3D Gaussian splatting (3DGS), known for its groundbreaking performance and
efficiency, has become a dominant 3D representation and brought progress to
many 3D vision tasks. However, in this work, we reveal a significant security
vulnerability that has been largely overlooked in 3DGS: the computation cost of
training 3DGS could be maliciously tampered by poisoning the input data. By
developing an attack named Poison-splat, we reveal a novel attack surface where
the adversary can poison the input images to drastically increase the
computation memory and time needed for 3DGS training, pushing the algorithm
towards its worst computation complexity. In extreme cases, the attack can even
consume all allocable memory, leading to a Denial-of-Service (DoS) that
disrupts servers, resulting in practical damages to real-world 3DGS service
vendors. Such a computation cost attack is achieved by addressing a bi-level
optimization problem through three tailored strategies: attack objective
approximation, proxy model rendering, and optional constrained optimization.
These strategies not only ensure the effectiveness of our attack but also make
it difficult to defend with simple defensive measures. We hope the revelation
of this novel attack surface can spark attention to this crucial yet overlooked
vulnerability of 3DGS systems.


http://arxiv.org/abs/2410.08109
A Closer Look at Machine Unlearning for Large Language Models. (4%)
Xiaojian Yuan; Tianyu Pang; Chao Du; Kejiang Chen; Weiming Zhang; Min Lin
  Large language models (LLMs) may memorize sensitive or copyrighted content,
raising privacy and legal concerns. Due to the high cost of retraining from
scratch, researchers attempt to employ machine unlearning to remove specific
content from LLMs while preserving the overall performance. In this paper, we
discuss several issues in machine unlearning for LLMs and provide our insights
on possible approaches. To address the issue of inadequate evaluation of model
outputs after unlearning, we introduce three additional metrics to evaluate
token diversity, sentence semantics, and factual correctness. We then
categorize unlearning methods into untargeted and targeted, and discuss their
issues respectively. Specifically, the behavior that untargeted unlearning
attempts to approximate is unpredictable and may involve hallucinations, and
existing regularization is insufficient for targeted unlearning. To alleviate
these issues, we propose using the objective of maximizing entropy (ME) for
untargeted unlearning and incorporate answer preservation (AP) loss as
regularization for targeted unlearning. Experimental results across three
scenarios, i.e., fictitious unlearning, continual unlearning, and real-world
unlearning, demonstrate the effectiveness of our approaches. The code is
available at https://github.com/sail-sg/closer-look-LLM-unlearning.


http://arxiv.org/abs/2410.06699
Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models. (99%)
Yubo Wang; Chaohu Liu; Yanqiu Qu; Haoyu Cao; Deqiang Jiang; Linli Xu
  Large vision-language models (LVLMs) integrate visual information into large
language models, showcasing remarkable multi-modal conversational capabilities.
However, the visual modules introduces new challenges in terms of robustness
for LVLMs, as attackers can craft adversarial images that are visually clean
but may mislead the model to generate incorrect answers. In general, LVLMs rely
on vision encoders to transform images into visual tokens, which are crucial
for the language models to perceive image contents effectively. Therefore, we
are curious about one question: Can LVLMs still generate correct responses when
the encoded visual tokens are attacked and disrupting the visual information?
To this end, we propose a non-targeted attack method referred to as VT-Attack
(Visual Tokens Attack), which constructs adversarial examples from multiple
perspectives, with the goal of comprehensively disrupting feature
representations and inherent relationships as well as the semantic properties
of visual tokens output by image encoders. Using only access to the image
encoder in the proposed attack, the generated adversarial examples exhibit
transferability across diverse LVLMs utilizing the same image encoder and
generality across different tasks. Extensive experiments validate the superior
attack performance of the VT-Attack over baseline methods, demonstrating its
effectiveness in attacking LVLMs with image encoders, which in turn can provide
guidance on the robustness of LVLMs, particularly in terms of the stability of
the visual feature space.


http://arxiv.org/abs/2410.06851
Understanding Model Ensemble in Transferable Adversarial Attack. (99%)
Wei Yao; Zeliang Zhang; Huayi Tang; Yong Liu
  Model ensemble adversarial attack has become a powerful method for generating
transferable adversarial examples that can target even unknown models, but its
theoretical foundation remains underexplored. To address this gap, we provide
early theoretical insights that serve as a roadmap for advancing model ensemble
adversarial attack. We first define transferability error to measure the error
in adversarial transferability, alongside concepts of diversity and empirical
model ensemble Rademacher complexity. We then decompose the transferability
error into vulnerability, diversity, and a constant, which rigidly explains the
origin of transferability error in model ensemble attack: the vulnerability of
an adversarial example to ensemble components, and the diversity of ensemble
components. Furthermore, we apply the latest mathematical tools in information
theory to bound the transferability error using complexity and generalization
terms, contributing to three practical guidelines for reducing transferability
error: (1) incorporating more surrogate models, (2) increasing their diversity,
and (3) reducing their complexity in cases of overfitting. Finally, extensive
experiments with 54 models validate our theoretical framework, representing a
significant step forward in understanding transferable model ensemble
adversarial attacks.


http://arxiv.org/abs/2410.06866
Secure Video Quality Assessment Resisting Adversarial Attacks. (75%)
Ao-Xiang Zhang; Yu Ran; Weixuan Tang; Yuan-Gen Wang; Qingxiao Guan; Chunsheng Yang
  The exponential surge in video traffic has intensified the imperative for
Video Quality Assessment (VQA). Leveraging cutting-edge architectures, current
VQA models have achieved human-comparable accuracy. However, recent studies
have revealed the vulnerability of existing VQA models against adversarial
attacks. To establish a reliable and practical assessment system, a secure VQA
model capable of resisting such malicious attacks is urgently demanded.
Unfortunately, no attempt has been made to explore this issue. This paper first
attempts to investigate general adversarial defense principles, aiming at
endowing existing VQA models with security. Specifically, we first introduce
random spatial grid sampling on the video frame for intra-frame defense. Then,
we design pixel-wise randomization through a guardian map, globally
neutralizing adversarial perturbations. Meanwhile, we extract temporal
information from the video sequence as compensation for inter-frame defense.
Building upon these principles, we present a novel VQA framework from the
security-oriented perspective, termed SecureVQA. Extensive experiments indicate
that SecureVQA sets a new benchmark in security while achieving competitive VQA
performance compared with state-of-the-art models. Ablation studies delve
deeper into analyzing the principles of SecureVQA, demonstrating their
generalization and contributions to the security of leading VQA models.


http://arxiv.org/abs/2410.06572
Can DeepFake Speech be Reliably Detected? (62%)
Hongbin Liu; Youzheng Chen; Arun Narayanan; Athula Balachandran; Pedro J. Moreno; Lun Wang
  Recent advances in text-to-speech (TTS) systems, particularly those with
voice cloning capabilities, have made voice impersonation readily accessible,
raising ethical and legal concerns due to potential misuse for malicious
activities like misinformation campaigns and fraud. While synthetic speech
detectors (SSDs) exist to combat this, they are vulnerable to ``test domain
shift", exhibiting decreased performance when audio is altered through
transcoding, playback, or background noise. This vulnerability is further
exacerbated by deliberate manipulation of synthetic speech aimed at deceiving
detectors. This work presents the first systematic study of such active
malicious attacks against state-of-the-art open-source SSDs. White-box attacks,
black-box attacks, and their transferability are studied from both attack
effectiveness and stealthiness, using both hardcoded metrics and human ratings.
The results highlight the urgent need for more robust detection methods in the
face of evolving adversarial threats.


http://arxiv.org/abs/2410.09101
Data Taggants: Dataset Ownership Verification via Harmless Targeted Data Poisoning. (15%)
Wassim Bouaziz; El-Mahdi El-Mhamdi; Nicolas Usunier
  Dataset ownership verification, the process of determining if a dataset is
used in a model's training data, is necessary for detecting unauthorized data
usage and data contamination. Existing approaches, such as backdoor
watermarking, rely on inducing a detectable behavior into the trained model on
a part of the data distribution. However, these approaches have limitations, as
they can be harmful to the model's performances or require unpractical access
to the model's internals. Most importantly, previous approaches lack guarantee
against false positives. This paper introduces data taggants, a novel
non-backdoor dataset ownership verification technique. Our method uses pairs of
out-of-distribution samples and random labels as secret keys, and leverages
clean-label targeted data poisoning to subtly alter a dataset, so that models
trained on it respond to the key samples with the corresponding key labels. The
keys are built as to allow for statistical certificates with black-box access
only to the model. We validate our approach through comprehensive and realistic
experiments on ImageNet1k using ViT and ResNet models with state-of-the-art
training recipes. Our findings demonstrate that data taggants can reliably make
models trained on the protected dataset detectable with high confidence,
without compromising validation accuracy, and demonstrates superiority over
backdoor watermarking. Moreover, our method shows to be stealthy and robust
against various defense mechanisms.


http://arxiv.org/abs/2410.06895
Average Certified Radius is a Poor Metric for Randomized Smoothing. (13%)
Chenhao Sun; Yuhao Mao; Mark Niklas Müller; Martin Vechev
  Randomized smoothing is a popular approach for providing certified robustness
guarantees against adversarial attacks, and has become an active area of
research. Over the past years, the average certified radius (ACR) has emerged
as the most important metric for comparing methods and tracking progress in the
field. However, in this work, for the first time we show that ACR is a poor
metric for evaluating robustness guarantees provided by randomized smoothing.
We theoretically prove not only that a trivial classifier can have arbitrarily
large ACR, but also that ACR is much more sensitive to improvements on easy
samples than on hard ones. Empirically, we confirm that existing training
strategies, though improving ACR with different approaches, reduce the model's
robustness on hard samples consistently. To strengthen our conclusion, we
propose strategies, including explicitly discarding hard samples, reweighting
the dataset with approximate certified radius, and extreme optimization for
easy samples, to achieve state-of-the-art ACR, without training for robustness
on the full data distribution. Overall, our results suggest that ACR has
introduced a strong undesired bias to the field, and its application should be
discontinued when evaluating randomized smoothing.


http://arxiv.org/abs/2410.07081
JPEG Inspired Deep Learning. (11%)
Ahmed H. Salamah; Kaixiang Zheng; Yiwen Liu; En-Hui Yang
  Although it is traditionally believed that lossy image compression, such as
JPEG compression, has a negative impact on the performance of deep neural
networks (DNNs), it is shown by recent works that well-crafted JPEG compression
can actually improve the performance of deep learning (DL). Inspired by this,
we propose JPEG-DL, a novel DL framework that prepends any underlying DNN
architecture with a trainable JPEG compression layer. To make the quantization
operation in JPEG compression trainable, a new differentiable soft quantizer is
employed at the JPEG layer, and then the quantization operation and underlying
DNN are jointly trained. Extensive experiments show that in comparison with the
standard DL, JPEG-DL delivers significant accuracy improvements across various
datasets and model architectures while enhancing robustness against adversarial
attacks. Particularly, on some fine-grained image classification datasets,
JPEG-DL can increase prediction accuracy by as much as 20.9%. Our code is
available on https://github.com/AhmedHussKhalifa/JPEG-Inspired-DL.git.


http://arxiv.org/abs/2410.06921
Adversarial Vulnerability as a Consequence of On-Manifold Inseparibility. (2%)
Rajdeep Haldar; Yue Xing; Qifan Song; Guang Lin
  Recent works have shown theoretically and empirically that redundant data
dimensions are a source of adversarial vulnerability. However, the inverse
doesn't seem to hold in practice; employing dimension-reduction techniques
doesn't exhibit robustness as expected. In this work, we consider
classification tasks and characterize the data distribution as a
low-dimensional manifold, with high/low variance features defining the on/off
manifold direction. We argue that clean training experiences poor convergence
in the off-manifold direction caused by the ill-conditioning in widely used
first-order optimizers like gradient descent. The poor convergence then acts as
a source of adversarial vulnerability when the dataset is inseparable in the
on-manifold direction. We provide theoretical results for logistic regression
and a 2-layer linear network on the considered data distribution. Furthermore,
we advocate using second-order methods that are immune to ill-conditioning and
lead to better robustness. We perform experiments and exhibit tremendous
robustness improvements in clean training through long training and the
employment of second-order methods, corroborating our framework. Additionally,
we find the inclusion of batch-norm layers hinders such robustness gains. We
attribute this to differing implicit biases between traditional and
batch-normalized neural networks.


http://arxiv.org/abs/2410.07137
Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates. (2%)
Xiaosen Zheng; Tianyu Pang; Chao Du; Qian Liu; Jing Jiang; Min Lin
  Automatic LLM benchmarks, such as AlpacaEval 2.0, Arena-Hard-Auto, and
MT-Bench, have become popular for evaluating language models due to their
cost-effectiveness and scalability compared to human evaluation. Achieving high
win rates on these benchmarks can significantly boost the promotional impact of
newly released language models. This promotional benefit may motivate tricks,
such as manipulating model output length or style to game win rates, even
though several mechanisms have been developed to control length and disentangle
style to reduce gameability. Nonetheless, we show that even a "null model" that
always outputs a constant response (irrelevant to input instructions) can cheat
automatic benchmarks and achieve top-ranked win rates: an 86.5% LC win rate on
AlpacaEval 2.0; an 83.0 score on Arena-Hard-Auto; and a 9.55 score on MT-Bench.
Moreover, the crafted cheating outputs are transferable because we assume that
the instructions of these benchmarks (e.g., 805 samples of AlpacaEval 2.0) are
private and cannot be accessed. While our experiments are primarily
proof-of-concept, an adversary could use LLMs to generate more imperceptible
cheating responses, unethically benefiting from high win rates and promotional
impact. Our findings call for the development of anti-cheating mechanisms for
reliable automatic benchmarks. The code is available at
https://github.com/sail-sg/Cheating-LLM-Benchmarks.


http://arxiv.org/abs/2410.06782
Mind Your Questions! Towards Backdoor Attacks on Text-to-Visualization Models. (2%)
Shuaimin Li; Yuanfeng Song; Xuanang Chen; Anni Peng; Zhuoyue Wan; Chen Jason Zhang; Raymond Chi-Wing Wong
  Text-to-visualization (text-to-vis) models have become valuable tools in the
era of big data, enabling users to generate data visualizations and make
informed decisions through natural language queries (NLQs). Despite their
widespread application, the security vulnerabilities of these models have been
largely overlooked. To address this gap, we propose VisPoison, a novel
framework designed to identify these vulnerabilities of current text-to-vis
models systematically. VisPoison introduces two types of triggers that activate
three distinct backdoor attacks, potentially leading to data exposure,
misleading visualizations, or denial-of-service (DoS) incidents. The framework
features both proactive and passive attack mechanisms: proactive attacks
leverage rare-word triggers to access confidential data, while passive attacks,
triggered unintentionally by users, exploit a first-word trigger method,
causing errors or DoS events in visualizations. Through extensive experiments
on both trainable and in-context learning (ICL)-based text-to-vis models,
\textit{VisPoison} achieves attack success rates of over 90\%, highlighting the
security problem of current text-to-vis models. Additionally, we explore two
types of defense mechanisms against these attacks, but the results show that
existing countermeasures are insufficient, underscoring the pressing need for
more robust security solutions in text-to-vis systems.


http://arxiv.org/abs/2410.06976
AdaRC: Mitigating Graph Structure Shifts during Test-Time. (1%)
Wenxuan Bao; Zhichen Zeng; Zhining Liu; Hanghang Tong; Jingrui He
  Powerful as they are, graph neural networks (GNNs) are known to be vulnerable
to distribution shifts. Recently, test-time adaptation (TTA) has attracted
attention due to its ability to adapt a pre-trained model to a target domain
without re-accessing the source domain. However, existing TTA algorithms are
primarily designed for attribute shifts in vision tasks, where samples are
independent. These methods perform poorly on graph data that experience
structure shifts, where node connectivity differs between source and target
graphs. We attribute this performance gap to the distinct impact of node
attribute shifts versus graph structure shifts: the latter significantly
degrades the quality of node representations and blurs the boundaries between
different node categories. To address structure shifts in graphs, we propose
AdaRC, an innovative framework designed for effective and efficient adaptation
to structure shifts by adjusting the hop-aggregation parameters in GNNs. To
enhance the representation quality, we design a prediction-informed clustering
loss to encourage the formation of distinct clusters for different node
categories. Additionally, AdaRC seamlessly integrates with existing TTA
algorithms, allowing it to handle attribute shifts effectively while improving
overall performance under combined structure and attribute shifts. We validate
the effectiveness of AdaRC on both synthetic and real-world datasets,
demonstrating its robustness across various combinations of structure and
attribute shifts.


http://arxiv.org/abs/2410.06704
PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs. (1%)
Krishna Kanth Nakka; Ahmed Frikha; Ricardo Mendes; Xue Jiang; Xuebing Zhou
  In this work, we introduce PII-Scope, a comprehensive benchmark designed to
evaluate state-of-the-art methodologies for PII extraction attacks targeting
LLMs across diverse threat settings. Our study provides a deeper understanding
of these attacks by uncovering several hyperparameters (e.g., demonstration
selection) crucial to their effectiveness. Building on this understanding, we
extend our study to more realistic attack scenarios, exploring PII attacks that
employ advanced adversarial strategies, including repeated and diverse
querying, and leveraging iterative learning for continual PII extraction.
Through extensive experimentation, our results reveal a notable underestimation
of PII leakage in existing single-query attacks. In fact, we show that with
sophisticated adversarial capabilities and a limited query budget, PII
extraction rates can increase by up to fivefold when targeting the pretrained
model. Moreover, we evaluate PII leakage on finetuned models, showing that they
are more vulnerable to leakage than pretrained models. Overall, our work
establishes a rigorous empirical benchmark for PII extraction attacks in
realistic threat scenarios and provides a strong foundation for developing
effective mitigation strategies.


http://arxiv.org/abs/2410.06913
Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuning. (1%)
Runchuan Zhu; Zhipeng Ma; Jiang Wu; Junyuan Gao; Jiaqi Wang; Dahua Lin; Conghui He
  Refusal-Aware Instruction Tuning (RAIT) enables Large Language Models (LLMs)
to refuse to answer unknown questions. By modifying responses of unknown
questions in the training data to refusal responses such as "I don't know",
RAIT enhances the reliability of LLMs and reduces their hallucination.
Generally, RAIT modifies training samples based on the correctness of the
initial LLM's response. However, this crude approach can cause LLMs to
excessively refuse answering questions they could have correctly answered, the
problem we call over-refusal. In this paper, we explore two primary causes of
over-refusal: Static conflict occurs when similar samples within the LLM's
feature space receive differing supervision signals (original vs. modified "I
don't know"). Dynamic conflict arises as the LLM's evolving knowledge during
SFT enables it to answer previously unanswerable questions, but the
now-answerable training samples still retain the original "I don't know"
supervision signals from the initial LLM state, leading to inconsistencies.
These conflicts cause the trained LLM to misclassify known questions as
unknown, resulting in over-refusal. To address this issue, we introduce
Certainty Represented Knowledge Flow for Refusal-Aware Instructions Tuning
(CRaFT). CRaFT centers on two main contributions: First, we additionally
incorporate response certainty to selectively filter and modify data, reducing
static conflicts. Second, we implement preliminary rehearsal training to
characterize changes in the LLM's knowledge state, which helps mitigate dynamic
conflicts during the fine-tuning process. We conducted extensive experiments on
open-ended question answering and multiple-choice question task. Experiment
results show that CRaFT can improve LLM's overall performance during the RAIT
process. Code and data will be released at https://github.com/opendatalab/CRaFT .


http://arxiv.org/abs/2410.05951
Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models. (99%)
Kangtao Lv; Huangsen Cao; Kainan Tu; Yihuai Xu; Zhimeng Zhang; Xin Ding; Yongwei Wang
  Large vision models have been found vulnerable to adversarial examples,
emphasizing the need for enhancing their adversarial robustness. While
adversarial training is an effective defense for deep convolutional models, it
often faces scalability issues with large vision models due to high
computational costs. Recent approaches propose robust fine-tuning methods, such
as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but
they still struggle to match the accuracy of full parameter adversarial
fine-tuning. The integration of various defense mechanisms offers a promising
approach to enhancing the robustness of large vision models, yet this paradigm
remains underexplored. To address this, we propose hyper adversarial tuning
(HyperAT), which leverages shared defensive knowledge among different methods
to improve model robustness efficiently and effectively simultaneously.
Specifically, adversarial tuning of each defense method is formulated as a
learning task, and a hypernetwork generates LoRA specific to this defense.
Then, a random sampling and tuning strategy is proposed to extract and
facilitate the defensive knowledge transfer between different defenses.
Finally, diverse LoRAs are merged to enhance the adversarial robustness.
Experiments on various datasets and model architectures demonstrate that
HyperAT significantly enhances the adversarial robustness of pretrained large
vision models without excessive computational overhead, establishing a new
state-of-the-art benchmark.


http://arxiv.org/abs/2410.06339
Filtered Randomized Smoothing: A New Defense for Robust Modulation Classification. (98%)
Wenhan Zhang; Meiyu Zhong; Ravi Tandon; Marwan Krunz
  Deep Neural Network (DNN) based classifiers have recently been used for the
modulation classification of RF signals. These classifiers have shown
impressive performance gains relative to conventional methods, however, they
are vulnerable to imperceptible (low-power) adversarial attacks. Some of the
prominent defense approaches include adversarial training (AT) and randomized
smoothing (RS). While AT increases robustness in general, it fails to provide
resilience against previously unseen adaptive attacks. Other approaches, such
as Randomized Smoothing (RS), which injects noise into the input, address this
shortcoming by providing provable certified guarantees against arbitrary
attacks, however, they tend to sacrifice accuracy.
  In this paper, we study the problem of designing robust DNN-based modulation
classifiers that can provide provable defense against arbitrary attacks without
significantly sacrificing accuracy. To this end, we first analyze the spectral
content of commonly studied attacks on modulation classifiers for the benchmark
RadioML dataset. We observe that spectral signatures of un-perturbed RF signals
are highly localized, whereas attack signals tend to be spread out in
frequency. To exploit this spectral heterogeneity, we propose Filtered
Randomized Smoothing (FRS), a novel defense which combines spectral filtering
together with randomized smoothing. FRS can be viewed as a strengthening of RS
by leveraging the specificity (spectral Heterogeneity) inherent to the
modulation classification problem. In addition to providing an approach to
compute the certified accuracy of FRS, we also provide a comprehensive set of
simulations on the RadioML dataset to show the effectiveness of FRS and show
that it significantly outperforms existing defenses including AT and RS in
terms of accuracy on both attacked and benign signals.


http://arxiv.org/abs/2410.05814
CALoR: Towards Comprehensive Model Inversion Defense. (76%)
Hongyao Yu; Yixiang Qiu; Hao Fang; Bin Chen; Sijin Yu; Bin Wang; Shu-Tao Xia; Ke Xu
  Model Inversion Attacks (MIAs) aim at recovering privacy-sensitive training
data from the knowledge encoded in the released machine learning models. Recent
advances in the MIA field have significantly enhanced the attack performance
under multiple scenarios, posing serious privacy risks of Deep Neural Networks
(DNNs). However, the development of defense strategies against MIAs is
relatively backward to resist the latest MIAs and existing defenses fail to
achieve further trade-off between model utility and model robustness. In this
paper, we provide an in-depth analysis from the perspective of intrinsic
vulnerabilities of MIAs, comprehensively uncovering the weaknesses inherent in
the basic pipeline, which are partially investigated in the previous defenses.
Building upon these new insights, we propose a robust defense mechanism,
integrating Confidence Adaptation and Low-Rank compression(CALoR). Our method
includes a novel robustness-enhanced classification loss specially-designed for
model inversion defenses and reveals the extraordinary effectiveness of
compressing the classification header. With CALoR, we can mislead the
optimization objective, reduce the leaked information and impede the
backpropagation of MIAs, thus mitigating the risk of privacy leakage. Extensive
experimental results demonstrate that our method achieves state-of-the-art
(SOTA) defense performance against MIAs and exhibits superior generalization to
existing defenses across various scenarios.


http://arxiv.org/abs/2410.05750
Polynomial Time Cryptanalytic Extraction of Deep Neural Networks in the Hard-Label Setting. (74%)
Nicholas Carlini; Jorge Chávez-Saab; Anna Hambitzer; Francisco Rodríguez-Henríquez; Adi Shamir
  Deep neural networks (DNNs) are valuable assets, yet their public
accessibility raises security concerns about parameter extraction by malicious
actors. Recent work by Carlini et al. (crypto'20) and Canales-Mart\'inez et al.
(eurocrypt'24) has drawn parallels between this issue and block cipher key
extraction via chosen plaintext attacks. Leveraging differential cryptanalysis,
they demonstrated that all the weights and biases of black-box ReLU-based DNNs
could be inferred using a polynomial number of queries and computational time.
However, their attacks relied on the availability of the exact numeric value of
output logits, which allowed the calculation of their derivatives. To overcome
this limitation, Chen et al. (asiacrypt'24) tackled the more realistic
hard-label scenario, where only the final classification label (e.g., "dog" or
"car") is accessible to the attacker. They proposed an extraction method
requiring a polynomial number of queries but an exponential execution time. In
addition, their approach was applicable only to a restricted set of
architectures, could deal only with binary classifiers, and was demonstrated
only on tiny neural networks with up to four neurons split among up to two
hidden layers. This paper introduces new techniques that, for the first time,
achieve cryptanalytic extraction of DNN parameters in the most challenging
hard-label setting, using both a polynomial number of queries and polynomial
time. We validate our approach by extracting nearly one million parameters from
a DNN trained on the CIFAR-10 dataset, comprising 832 neurons in four hidden
layers. Our results reveal the surprising fact that all the weights of a
ReLU-based DNN can be efficiently determined by analyzing only the geometric
shape of its decision boundaries.


http://arxiv.org/abs/2410.06072
Training-free LLM-generated Text Detection by Mining Token Probability Sequences. (26%)
Yihuai Xu; Yongwei Wang; Yifei Bi; Huangsen Cao; Zhouhan Lin; Yu Zhao; Fei Wu
  Large language models (LLMs) have demonstrated remarkable capabilities in
generating high-quality texts across diverse domains. However, the potential
misuse of LLMs has raised significant concerns, underscoring the urgent need
for reliable detection of LLM-generated texts. Conventional training-based
detectors often struggle with generalization, particularly in cross-domain and
cross-model scenarios. In contrast, training-free methods, which focus on
inherent discrepancies through carefully designed statistical features, offer
improved generalization and interpretability. Despite this, existing
training-free detection methods typically rely on global text sequence
statistics, neglecting the modeling of local discriminative features, thereby
limiting their detection efficacy. In this work, we introduce a novel
training-free detector, termed \textbf{Lastde} that synergizes local and global
statistics for enhanced detection. For the first time, we introduce time series
analysis to LLM-generated text detection, capturing the temporal dynamics of
token probability sequences. By integrating these local statistics with global
ones, our detector reveals significant disparities between human and
LLM-generated texts. We also propose an efficient alternative,
\textbf{Lastde++} to enable real-time detection. Extensive experiments on six
datasets involving cross-domain, cross-model, and cross-lingual detection
scenarios, under both white-box and black-box settings, demonstrated that our
method consistently achieves state-of-the-art performance. Furthermore, our
approach exhibits greater robustness against paraphrasing attacks compared to
existing baseline methods.


http://arxiv.org/abs/2410.06509
PFAttack: Stealthy Attack Bypassing Group Fairness in Federated Learning. (10%)
Jiashi Gao; Ziwei Wang; Xiangyu Zhao; Xin Yao; Xuetao Wei
  Federated learning (FL), integrating group fairness mechanisms, allows
multiple clients to collaboratively train a global model that makes unbiased
decisions for different populations grouped by sensitive attributes (e.g.,
gender and race). Due to its distributed nature, previous studies have
demonstrated that FL systems are vulnerable to model poisoning attacks.
However, these studies primarily focus on perturbing accuracy, leaving a
critical question unexplored: Can an attacker bypass the group fairness
mechanisms in FL and manipulate the global model to be biased? The motivations
for such an attack vary; an attacker might seek higher accuracy, yet fairness
considerations typically limit the accuracy of the global model or aim to cause
ethical disruption. To address this question, we design a novel form of attack
in FL, termed Profit-driven Fairness Attack (PFATTACK), which aims not to
degrade global model accuracy but to bypass fairness mechanisms. Our
fundamental insight is that group fairness seeks to weaken the dependence of
outputs on input attributes related to sensitive information. In the proposed
PFATTACK, an attacker can recover this dependence through local fine-tuning
across various sensitive groups, thereby creating a biased yet
accuracy-preserving malicious model and injecting it into FL through model
replacement. Compared to attacks targeting accuracy, PFATTACK is more stealthy.
The malicious model in PFATTACK exhibits subtle parameter variations relative
to the original global model, making it robust against detection and filtering
by Byzantine-resilient aggregations. Extensive experiments on benchmark
datasets are conducted for four fair FL frameworks and three
Byzantine-resilient aggregations against model poisoning, demonstrating the
effectiveness and stealth of PFATTACK in bypassing group fairness mechanisms in
FL.


http://arxiv.org/abs/2410.09097
Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations. (10%)
Tarun Raheja; Nilay Pochhi
  Large Language Models (LLMs) have demonstrated remarkable capabilities in
natural language processing tasks, but their vulnerability to jailbreak attacks
poses significant security risks. This survey paper presents a comprehensive
analysis of recent advancements in attack strategies and defense mechanisms
within the field of Large Language Model (LLM) red-teaming. We analyze various
attack methods, including gradient-based optimization, reinforcement learning,
and prompt engineering approaches. We discuss the implications of these attacks
on LLM safety and the need for improved defense mechanisms. This work aims to
provide a thorough understanding of the current landscape of red-teaming
attacks and defenses on LLMs, enabling the development of more secure and
reliable language models.


http://arxiv.org/abs/2410.05136
LOTOS: Layer-wise Orthogonalization for Training Robust Ensembles. (99%)
Ali Ebrahimpour-Boroojeny; Hari Sundaram; Varun Chandrasekaran
  Transferability of adversarial examples is a well-known property that
endangers all classification models, even those that are only accessible
through black-box queries. Prior work has shown that an ensemble of models is
more resilient to transferability: the probability that an adversarial example
is effective against most models of the ensemble is low. Thus, most ongoing
research focuses on improving ensemble diversity. Another line of prior work
has shown that Lipschitz continuity of the models can make models more robust
since it limits how a model's output changes with small input perturbations. In
this paper, we study the effect of Lipschitz continuity on transferability
rates. We show that although a lower Lipschitz constant increases the
robustness of a single model, it is not as beneficial in training robust
ensembles as it increases the transferability rate of adversarial examples
across models in the ensemble. Therefore, we introduce LOTOS, a new training
paradigm for ensembles, which counteracts this adverse effect. It does so by
promoting orthogonality among the top-$k$ sub-spaces of the transformations of
the corresponding affine layers of any pair of models in the ensemble. We
theoretically show that $k$ does not need to be large for convolutional layers,
which makes the computational overhead negligible. Through various experiments,
we show LOTOS increases the robust accuracy of ensembles of ResNet-18 models by
$6$ percentage points (p.p) against black-box attacks on CIFAR-10. It is also
capable of combining with the robustness of prior state-of-the-art methods for
training robust ensembles to enhance their robust accuracy by $10.7$ p.p.


http://arxiv.org/abs/2410.05346
AnyAttack: Towards Large-scale Self-supervised Adversarial Attacks on Vision-language Models. (99%)
Jiaming Zhang; Junhong Ye; Xingjun Ma; Yige Li; Yunfan Yang; Yunhao Chen; Jitao Sang; Dit-Yan Yeung
  Due to their multimodal capabilities, Vision-Language Models (VLMs) have
found numerous impactful applications in real-world scenarios. However, recent
studies have revealed that VLMs are vulnerable to image-based adversarial
attacks. Traditional targeted adversarial attacks require specific targets and
labels, limiting their real-world impact.We present AnyAttack, a
self-supervised framework that transcends the limitations of conventional
attacks through a novel foundation model approach. By pre-training on the
massive LAION-400M dataset without label supervision, AnyAttack achieves
unprecedented flexibility - enabling any image to be transformed into an attack
vector targeting any desired output across different VLMs.This approach
fundamentally changes the threat landscape, making adversarial capabilities
accessible at an unprecedented scale. Our extensive validation across five
open-source VLMs (CLIP, BLIP, BLIP2, InstructBLIP, and MiniGPT-4) demonstrates
AnyAttack's effectiveness across diverse multimodal tasks. Most concerning,
AnyAttack seamlessly transfers to commercial systems including Google Gemini,
Claude Sonnet, Microsoft Copilot and OpenAI GPT, revealing a systemic
vulnerability requiring immediate attention.


http://arxiv.org/abs/2410.05573
TaeBench: Improving Quality of Toxic Adversarial Examples. (99%)
Xuan Zhu; Dmitriy Bespalov; Liwen You; Ninad Kulkarni; Yanjun Qi
  Toxicity text detectors can be vulnerable to adversarial examples - small
perturbations to input text that fool the systems into wrong detection.
Existing attack algorithms are time-consuming and often produce invalid or
ambiguous adversarial examples, making them less useful for evaluating or
improving real-world toxicity content moderators. This paper proposes an
annotation pipeline for quality control of generated toxic adversarial examples
(TAE). We design model-based automated annotation and human-based quality
verification to assess the quality requirements of TAE. Successful TAE should
fool a target toxicity model into making benign predictions, be grammatically
reasonable, appear natural like human-generated text, and exhibit semantic
toxicity. When applying these requirements to more than 20 state-of-the-art
(SOTA) TAE attack recipes, we find many invalid samples from a total of 940k
raw TAE attack generations. We then utilize the proposed pipeline to filter and
curate a high-quality TAE dataset we call TaeBench (of size 264k). Empirically,
we demonstrate that TaeBench can effectively transfer-attack SOTA toxicity
content moderation models and services. Our experiments also show that TaeBench
with adversarial training achieve significant improvements of the robustness of
two toxicity detectors.


http://arxiv.org/abs/2410.04884
Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models. (95%)
Dehong Kong; Siyuan Liang; Xiaopeng Zhu; Yuansheng Zhong; Wenqi Ren
  Visual language pre-training (VLP) models have demonstrated significant
success across various domains, yet they remain vulnerable to adversarial
attacks. Addressing these adversarial vulnerabilities is crucial for enhancing
security in multimodal learning. Traditionally, adversarial methods targeting
VLP models involve simultaneously perturbing images and text. However, this
approach faces notable challenges: first, adversarial perturbations often fail
to translate effectively into real-world scenarios; second, direct
modifications to the text are conspicuously visible. To overcome these
limitations, we propose a novel strategy that exclusively employs image patches
for attacks, thus preserving the integrity of the original text. Our method
leverages prior knowledge from diffusion models to enhance the authenticity and
naturalness of the perturbations. Moreover, to optimize patch placement and
improve the efficacy of our attacks, we utilize the cross-attention mechanism,
which encapsulates intermodal interactions by generating attention maps to
guide strategic patch placements. Comprehensive experiments conducted in a
white-box setting for image-to-text scenarios reveal that our proposed method
significantly outperforms existing techniques, achieving a 100% attack success
rate. Additionally, it demonstrates commendable performance in transfer tasks
involving text-to-image configurations.


http://arxiv.org/abs/2410.05159
MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense. (86%)
Yixiang Qiu; Hongyao Yu; Hao Fang; Tianqu Zhuang; Wenbo Yu; Bin Chen; Xuan Wang; Shu-Tao Xia; Ke Xu
  Model Inversion (MI) attacks aim at leveraging the output information of
target models to reconstruct privacy-sensitive training data, raising critical
concerns regarding the privacy vulnerabilities of Deep Neural Networks (DNNs).
Unfortunately, in tandem with the rapid evolution of MI attacks, the absence of
a comprehensive benchmark with standardized metrics and reproducible
implementations has emerged as a formidable challenge. This deficiency has
hindered objective comparison of methodological advancements and reliable
assessment of defense efficacy. To address this critical gap, we build the
first practical benchmark named MIBench for systematic evaluation of model
inversion attacks and defenses. This benchmark bases on an extensible and
reproducible modular-based toolbox which currently integrates a total of 19
state-of-the-art attack and defense methods and encompasses 9 standardized
evaluation protocols. Capitalizing on this foundation, we conduct extensive
evaluation from multiple perspectives to holistically compare and analyze
various methods across different scenarios, such as the impact of target
resolution, model predictive power, defense performance and adversarial
robustness.


http://arxiv.org/abs/2410.05417
STOP! Camera Spoofing via the in-Vehicle IP Network. (83%)
Dror Peri; Avishai Wool
  Autonomous driving and advanced driver assistance systems (ADAS) rely on
cameras to control the driving. In many prior approaches an attacker aiming to
stop the vehicle had to send messages on the specialized and better-defended
CAN bus. We suggest an easier alternative: manipulate the IP-based network
communication between the camera and the ADAS logic, inject fake images of stop
signs or red lights into the video stream, and let the ADAS stop the car
safely. We created an attack tool that successfully exploits the GigE Vision
protocol.
  Then we analyze two classes of passive anomaly detectors to identify such
attacks: protocol-based detectors and video-based detectors. We implemented
multiple detectors of both classes and evaluated them on data collected from
our test vehicle and also on data from the public BDD corpus. Our results show
that such detectors are effective against naive adversaries, but sophisticated
adversaries can evade detection.
  Finally, we propose a novel class of active defense mechanisms that randomly
adjust camera parameters during the video transmission, and verify that the
received images obey the requested adjustments. Within this class we focus on a
specific implementation, the width-varying defense, which randomly modifies the
width of every frame. Beyond its function as an anomaly detector, this defense
is also a protective measure against certain attacks: by distorting injected
image patches it prevents their recognition by the ADAS logic. We demonstrate
the effectiveness of the width-varying defense through theoretical analysis and
by an extensive evaluation of several types of attack in a wide range of
realistic road driving conditions. The best the attack was able to achieve
against this defense was injecting a stop sign for a duration of 0.2 seconds,
with a success probability of 0.2%, whereas stopping a vehicle requires about
2.5 seconds.


http://arxiv.org/abs/2410.05451
SecAlign: Defending Against Prompt Injection with Preference Optimization. (78%)
Sizhe Chen; Arman Zharmagambetov; Saeed Mahloujifar; Kamalika Chaudhuri; David Wagner; Chuan Guo
  Large language models (LLMs) are becoming increasingly prevalent in modern
software systems, interfacing between the user and the Internet to assist with
tasks that require advanced language understanding. To accomplish these tasks,
the LLM often uses external data sources such as user documents, web retrieval,
results from API calls, etc. This opens up new avenues for attackers to
manipulate the LLM via prompt injection. Adversarial prompts can be injected
into external data sources to override the system's intended instruction and
instead execute a malicious instruction.
  To mitigate this vulnerability, we propose a new defense called SecAlign
based on the technique of preference optimization. Our defense first constructs
a preference dataset with prompt-injected inputs, secure outputs (ones that
respond to the legitimate instruction), and insecure outputs (ones that respond
to the injection). We then perform preference optimization on this dataset to
teach the LLM to prefer the secure output over the insecure one. This provides
the first known method that reduces the success rates of various prompt
injections to around 0%, even against attacks much more sophisticated than ones
seen during training. This indicates our defense generalizes well against
unknown and yet-to-come attacks. Also, our defended models are still practical
with similar utility to the one before our defensive training. Our code is at
https://github.com/facebookresearch/SecAlign


http://arxiv.org/abs/2410.04764
Double Oracle Neural Architecture Search for Game Theoretic Deep Learning Models. (76%)
Aye Phyu Phyu Aung; Xinrun Wang; Ruiyu Wang; Hau Chan; Bo An; Xiaoli Li; J. Senthilnath
  In this paper, we propose a new approach to train deep learning models using
game theory concepts including Generative Adversarial Networks (GANs) and
Adversarial Training (AT) where we deploy a double-oracle framework using best
response oracles. GAN is essentially a two-player zero-sum game between the
generator and the discriminator. The same concept can be applied to AT with
attacker and classifier as players. Training these models is challenging as a
pure Nash equilibrium may not exist and even finding the mixed Nash equilibrium
is difficult as training algorithms for both GAN and AT have a large-scale
strategy space. Extending our preliminary model DO-GAN, we propose the methods
to apply the double oracle framework concept to Adversarial Neural Architecture
Search (NAS for GAN) and Adversarial Training (NAS for AT) algorithms. We first
generalize the players' strategies as the trained models of generator and
discriminator from the best response oracles. We then compute the
meta-strategies using a linear program. For scalability of the framework where
multiple network models of best responses are stored in the memory, we prune
the weakly-dominated players' strategies to keep the oracles from becoming
intractable. Finally, we conduct experiments on MNIST, CIFAR-10 and
TinyImageNet for DONAS-GAN. We also evaluate the robustness under FGSM and PGD
attacks on CIFAR-10, SVHN and TinyImageNet for DONAS-AT. We show that all our
variants have significant improvements in both subjective qualitative
evaluation and quantitative metrics, compared with their respective base
architectures.


http://arxiv.org/abs/2410.04968
Collaboration! Towards Robust Neural Methods for Routing Problems. (70%)
Jianan Zhou; Yaoxin Wu; Zhiguang Cao; Wen Song; Jie Zhang; Zhiqi Shen
  Despite enjoying desirable efficiency and reduced reliance on domain
expertise, existing neural methods for vehicle routing problems (VRPs) suffer
from severe robustness issues -- their performance significantly deteriorates
on clean instances with crafted perturbations. To enhance robustness, we
propose an ensemble-based Collaborative Neural Framework (CNF) w.r.t. the
defense of neural VRP methods, which is crucial yet underexplored in the
literature. Given a neural VRP method, we adversarially train multiple models
in a collaborative manner to synergistically promote robustness against
attacks, while boosting standard generalization on clean instances. A neural
router is designed to adeptly distribute training instances among models,
enhancing overall load balancing and collaborative efficacy. Extensive
experiments verify the effectiveness and versatility of CNF in defending
against various attacks across different neural VRP methods. Notably, our
approach also achieves impressive out-of-distribution generalization on
benchmark instances.


http://arxiv.org/abs/2410.05363
Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. (1%)
Fanqing Meng; Jiaqi Liao; Xinyu Tan; Wenqi Shao; Quanfeng Lu; Kaipeng Zhang; Yu Cheng; Dianqi Li; Yu Qiao; Ping Luo
  Text-to-video (T2V) models like Sora have made significant strides in
visualizing complex prompts, which is increasingly viewed as a promising path
towards constructing the universal world simulator. Cognitive psychologists
believe that the foundation for achieving this goal is the ability to
understand intuitive physics. However, the capacity of these models to
accurately represent intuitive physics remains largely unexplored. To bridge
this gap, we introduce PhyGenBench, a comprehensive \textbf{Phy}sics
\textbf{Gen}eration \textbf{Ben}chmark designed to evaluate physical
commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully
crafted prompts across 27 distinct physical laws, spanning four fundamental
domains, which could comprehensively assesses models' understanding of physical
commonsense. Alongside PhyGenBench, we propose a novel evaluation framework
called PhyGenEval. This framework employs a hierarchical evaluation structure
utilizing appropriate advanced vision-language models and large language models
to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can
conduct large-scale automated assessments of T2V models' understanding of
physical commonsense, which align closely with human feedback. Our evaluation
results and in-depth analysis demonstrate that current models struggle to
generate videos that comply with physical commonsense. Moreover, simply scaling
up models or employing prompt engineering techniques is insufficient to fully
address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We
hope this study will inspire the community to prioritize the learning of
physical commonsense in these models beyond entertainment applications. We will
release the data and codes at https://github.com/OpenGVLab/PhyGenBench


http://arxiv.org/abs/2410.04377
Graded Suspiciousness of Adversarial Texts to Human. (99%)
Shakila Mahjabin Tonni; Pedro Faustini; Mark Dras
  Adversarial examples pose a significant challenge to deep neural networks
(DNNs) across both image and text domains, with the intent to degrade model
performance through meticulously altered inputs. Adversarial texts, however,
are distinct from adversarial images due to their requirement for semantic
similarity and the discrete nature of the textual contents. This study delves
into the concept of human suspiciousness, a quality distinct from the
traditional focus on imperceptibility found in image-based adversarial
examples. Unlike images, where adversarial changes are meant to be
indistinguishable to the human eye, textual adversarial content must often
remain undetected or non-suspicious to human readers, even when the text's
purpose is to deceive NLP systems or bypass filters.
  In this research, we expand the study of human suspiciousness by analyzing
how individuals perceive adversarial texts. We gather and publish a novel
dataset of Likert-scale human evaluations on the suspiciousness of adversarial
sentences, crafted by four widely used adversarial attack methods and assess
their correlation with the human ability to detect machine-generated
alterations. Additionally, we develop a regression-based model to quantify
suspiciousness and establish a baseline for future research in reducing the
suspiciousness in adversarial text generation. We also demonstrate how the
regressor-generated suspicious scores can be incorporated into adversarial
generation methods to produce texts that are less likely to be perceived as
computer-generated. We make our human suspiciousness annotated data and our
code available.


http://arxiv.org/abs/2410.04682
On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning. (99%)
Yongyi Su; Yushu Li; Nanqing Liu; Kui Jia; Xulei Yang; Chuan-Sheng Foo; Xun Xu
  Test-time adaptation (TTA) updates the model weights during the inference
stage using testing data to enhance generalization. However, this practice
exposes TTA to adversarial risks. Existing studies have shown that when TTA is
updated with crafted adversarial test samples, also known as test-time poisoned
data, the performance on benign samples can deteriorate. Nonetheless, the
perceived adversarial risk may be overstated if the poisoned data is generated
under overly strong assumptions. In this work, we first review realistic
assumptions for test-time data poisoning, including white-box versus grey-box
attacks, access to benign data, attack budget, and more. We then propose an
effective and realistic attack method that better produces poisoned samples
without access to benign samples, and derive an effective in-distribution
attack objective. We also design two TTA-aware attack objectives. Our
benchmarks of existing attack methods reveal that the TTA methods are more
robust than previously believed. In addition, we analyze effective defense
strategies to help develop adversarially robust TTA methods.


http://arxiv.org/abs/2410.05334
TA3: Testing Against Adversarial Attacks on Machine Learning Models. (67%)
Yuanzhe Jin; Min Chen
  Adversarial attacks are major threats to the deployment of machine learning
(ML) models in many applications. Testing ML models against such attacks is
becoming an essential step for evaluating and improving ML models. In this
paper, we report the design and development of an interactive system for aiding
the workflow of Testing Against Adversarial Attacks (TA3). In particular, with
TA3, human-in-the-loop (HITL) enables human-steered attack simulation and
visualization-assisted attack impact evaluation. While the current version of
TA3 focuses on testing decision tree models against adversarial attacks based
on the One Pixel Attack Method, it demonstrates the importance of HITL in ML
testing and the potential application of HITL to the ML testing workflows for
other types of ML models and other types of adversarial attacks.


http://arxiv.org/abs/2410.04577
Robustness Reprogramming for Representation Learning. (56%)
Zhichao Hou; MohamadAli Torkamani; Hamid Krim; Xiaorui Liu
  This work tackles an intriguing and fundamental open challenge in
representation learning: Given a well-trained deep learning model, can it be
reprogrammed to enhance its robustness against adversarial or noisy input
perturbations without altering its parameters? To explore this, we revisit the
core feature transformation mechanism in representation learning and propose a
novel non-linear robust pattern matching technique as a robust alternative.
Furthermore, we introduce three model reprogramming paradigms to offer flexible
control of robustness under different efficiency requirements. Comprehensive
experiments and ablation studies across diverse learning models ranging from
basic linear model and MLPs to shallow and modern deep ConvNets demonstrate the
effectiveness of our approaches. This work not only opens a promising and
orthogonal direction for improving adversarial defenses in deep learning beyond
existing methods but also provides new insights into designing more resilient
AI systems with robust statistics.


http://arxiv.org/abs/2410.04397
Towards Understanding and Enhancing Security of Proof-of-Training for DNN Model Ownership Verification. (2%)
Yijia Chang; Hanrui Jiang; Chao Lin; Xinyi Huang; Jian Weng
  The great economic values of deep neural networks (DNNs) urge AI enterprises
to protect their intellectual property (IP) for these models. Recently,
proof-of-training (PoT) has been proposed as a promising solution to DNN IP
protection, through which AI enterprises can utilize the record of DNN training
process as their ownership proof. To prevent attackers from forging ownership
proof, a secure PoT scheme should be able to distinguish honest training
records from those forged by attackers. Although existing PoT schemes provide
various distinction criteria, these criteria are based on intuitions or
observations. The effectiveness of these criteria lacks clear and comprehensive
analysis, resulting in existing schemes initially deemed secure being swiftly
compromised by simple ideas. In this paper, we make the first move to identify
distinction criteria in the style of formal methods, so that their
effectiveness can be explicitly demonstrated. Specifically, we conduct
systematic modeling to cover a wide range of attacks and then theoretically
analyze the distinctions between honest and forged training records. The
analysis results not only induce a universal distinction criterion, but also
provide detailed reasoning to demonstrate its effectiveness in defending
against attacks covered by our model. Guided by the criterion, we propose a
generic PoT construction that can be instantiated into concrete schemes. This
construction sheds light on the realization that trajectory matching
algorithms, previously employed in data distillation, possess significant
advantages in PoT construction. Experimental results demonstrate that our
scheme can resist attacks that have compromised existing PoT schemes, which
corroborates its superiority in security.


http://arxiv.org/abs/2410.04661
Federated Learning Nodes Can Reconstruct Peers' Image Data. (1%)
Ethan Wilson; Kai Yue; Chau-Wai Wong; Huaiyu Dai
  Federated learning (FL) is a privacy-preserving machine learning framework
that enables multiple nodes to train models on their local data and
periodically average weight updates to benefit from other nodes' training. Each
node's goal is to collaborate with other nodes to improve the model's
performance while keeping its training data private. However, this framework
does not guarantee data privacy. Prior work has shown that the gradient-sharing
steps in FL can be vulnerable to data reconstruction attacks from an
honest-but-curious central server. In this work, we show that an
honest-but-curious node/client can also launch attacks to reconstruct peers'
image data through gradient inversion, presenting a severe privacy risk. We
demonstrate that a single client can silently reconstruct other clients'
private images using diluted information available within consecutive updates.
We leverage state-of-the-art diffusion models to enhance the perceptual quality
and recognizability of the reconstructed images, further demonstrating the risk
of information leakage at a semantic level. This highlights the need for more
robust privacy-preserving mechanisms that protect against silent client-side
attacks during federated training.


http://arxiv.org/abs/2410.04190
Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models. (38%)
Yiting Dong; Guobin Shen; Dongcheng Zhao; Xiang He; Yi Zeng
  Large Language Models (LLMs) remain vulnerable to jailbreak attacks that
bypass their safety mechanisms. Existing attack methods are fixed or
specifically tailored for certain models and cannot flexibly adjust attack
strength, which is critical for generalization when attacking models of various
sizes. We introduce a novel scalable jailbreak attack that preempts the
activation of an LLM's safety policies by occupying its computational
resources. Our method involves engaging the LLM in a resource-intensive
preliminary task - a Character Map lookup and decoding process - before
presenting the target instruction. By saturating the model's processing
capacity, we prevent the activation of safety protocols when processing the
subsequent instruction. Extensive experiments on state-of-the-art LLMs
demonstrate that our method achieves a high success rate in bypassing safety
measures without requiring gradient access, manual prompt engineering. We
verified our approach offers a scalable attack that quantifies attack strength
and adapts to different model scales at the optimal strength. We shows safety
policies of LLMs might be more susceptible to resource constraints. Our
findings reveal a critical vulnerability in current LLM safety designs,
highlighting the need for more robust defense strategies that account for
resource-intense condition.


http://arxiv.org/abs/2410.04144
ConDa: Fast Federated Unlearning with Contribution Dampening. (1%)
Vikram S Chundawat; Pushkar Niroula; Prasanna Dhungana; Stefan Schoepf; Murari Mandal; Alexandra Brintrup
  Federated learning (FL) has enabled collaborative model training across
decentralized data sources or clients. While adding new participants to a
shared model does not pose great technical hurdles, the removal of a
participant and their related information contained in the shared model remains
a challenge. To address this problem, federated unlearning has emerged as a
critical research direction, seeking to remove information from globally
trained models without harming the model performance on the remaining data.
Most modern federated unlearning methods use costly approaches such as the use
of remaining clients data to retrain the global model or methods that would
require heavy computation on client or server side. We introduce Contribution
Dampening (ConDa), a framework that performs efficient unlearning by tracking
down the parameters which affect the global model for each client and performs
synaptic dampening on the parameters of the global model that have privacy
infringing contributions from the forgetting client. Our technique does not
require clients data or any kind of retraining and it does not put any
computational overhead on either the client or server side. We perform
experiments on multiple datasets and demonstrate that ConDa is effective to
forget a client's data. In experiments conducted on the MNIST, CIFAR10, and
CIFAR100 datasets, ConDa proves to be the fastest federated unlearning method,
outperforming the nearest state of the art approach by at least 100x. Our
emphasis is on the non-IID Federated Learning setting, which presents the
greatest challenge for unlearning. Additionally, we validate ConDa's robustness
through backdoor and membership inference attacks. We envision this work as a
crucial component for FL in adhering to legal and ethical requirements.


http://arxiv.org/abs/2410.03376
Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantization. (98%)
Tung M. Luu; Thanh Nguyen; Tee Joshua Tian Jin; Sungwoon Kim; Chang D. Yoo
  Recent studies reveal that well-performing reinforcement learning (RL) agents
in training often lack resilience against adversarial perturbations during
deployment. This highlights the importance of building a robust agent before
deploying it in the real world. Most prior works focus on developing robust
training-based procedures to tackle this problem, including enhancing the
robustness of the deep neural network component itself or adversarially
training the agent on strong attacks. In this work, we instead study an input
transformation-based defense for RL. Specifically, we propose using a variant
of vector quantization (VQ) as a transformation for input observations, which
is then used to reduce the space of adversarial attacks during testing,
resulting in the transformed observations being less affected by attacks. Our
method is computationally efficient and seamlessly integrates with adversarial
training, further enhancing the robustness of RL agents against adversarial
attacks. Through extensive experiments in multiple environments, we demonstrate
that using VQ as the input transformation effectively defends against
adversarial attacks on the agent's observations.


http://arxiv.org/abs/2410.03658
RAFT: Realistic Attacks to Fool Text Detectors. (96%)
James Wang; Ran Li; Junfeng Yang; Chengzhi Mao
  Large language models (LLMs) have exhibited remarkable fluency across various
tasks. However, their unethical applications, such as disseminating
disinformation, have become a growing concern. Although recent works have
proposed a number of LLM detection methods, their robustness and reliability
remain unclear. In this paper, we present RAFT: a grammar error-free black-box
attack against existing LLM detectors. In contrast to previous attacks for
language models, our method exploits the transferability of LLM embeddings at
the word-level while preserving the original text quality. We leverage an
auxiliary embedding to greedily select candidate words to perturb against the
target detector. Experiments reveal that our attack effectively compromises all
detectors in the study across various domains by up to 99%, and are
transferable across source models. Manual human evaluation studies show our
attacks are realistic and indistinguishable from original human-written text.
We also show that examples generated by RAFT can be used to train adversarially
robust detectors. Our work shows that current LLM detectors are not
adversarially robust, underscoring the urgent need for more resilient detection
mechanisms.


http://arxiv.org/abs/2410.03952
A Brain-Inspired Regularizer for Adversarial Robustness. (92%)
Elie Attias; Cengiz Pehlevan; Dina Obeid
  Convolutional Neural Networks (CNNs) excel in many visual tasks, but they
tend to be sensitive to slight input perturbations that are imperceptible to
the human eye, often resulting in task failures. Recent studies indicate that
training CNNs with regularizers that promote brain-like representations, using
neural recordings, can improve model robustness. However, the requirement to
use neural data severely restricts the utility of these methods. Is it possible
to develop regularizers that mimic the computational function of neural
regularizers without the need for neural recordings, thereby expanding the
usability and effectiveness of these techniques? In this work, we inspect a
neural regularizer introduced in Li et al. (2019) to extract its underlying
strength. The regularizer uses neural representational similarities, which we
find also correlate with pixel similarities. Motivated by this finding, we
introduce a new regularizer that retains the essence of the original but is
computed using image pixel similarities, eliminating the need for neural
recordings. We show that our regularization method 1) significantly increases
model robustness to a range of black box attacks on various datasets and 2) is
computationally inexpensive and relies only on original datasets. Our work
explores how biologically motivated loss functions can be used to drive the
performance of artificial neural networks.


http://arxiv.org/abs/2410.03489
Gradient-based Jailbreak Images for Multimodal Fusion Models. (16%)
Javier Rando; Hannah Korevaar; Erik Brinkman; Ivan Evtimov; Florian Tramèr
  Augmenting language models with image inputs may enable more effective
jailbreak attacks through continuous optimization, unlike text inputs that
require discrete optimization. However, new multimodal fusion models tokenize
all input modalities using non-differentiable functions, which hinders
straightforward attacks. In this work, we introduce the notion of a tokenizer
shortcut that approximates tokenization with a continuous function and enables
continuous optimization. We use tokenizer shortcuts to create the first
end-to-end gradient image attacks against multimodal fusion models. We evaluate
our attacks on Chameleon models and obtain jailbreak images that elicit harmful
information for 72.5% of prompts. Jailbreak images outperform text jailbreaks
optimized with the same objective and require 3x lower compute budget to
optimize 50x more input tokens. Finally, we find that representation
engineering defenses, like Circuit Breakers, trained only on text attacks can
effectively transfer to adversarial image inputs.


http://arxiv.org/abs/2410.03857
You Know What I'm Saying -- Jailbreak Attack via Implicit Reference. (16%)
Tianyu Wu; Lingrui Mei; Ruibin Yuan; Lujun Li; Wei Xue; Yike Guo
  While recent advancements in large language model (LLM) alignment have
enabled the effective identification of malicious objectives involving scene
nesting and keyword rewriting, our study reveals that these methods remain
inadequate at detecting malicious objectives expressed through context within
nested harmless objectives. This study identifies a previously overlooked
vulnerability, which we term Attack via Implicit Reference (AIR). AIR
decomposes a malicious objective into permissible objectives and links them
through implicit references within the context. This method employs multiple
related harmless objectives to generate malicious content without triggering
refusal responses, thereby effectively bypassing existing detection
techniques.Our experiments demonstrate AIR's effectiveness across
state-of-the-art LLMs, achieving an attack success rate (ASR) exceeding 90% on
most models, including GPT-4o, Claude-3.5-Sonnet, and Qwen-2-72B. Notably, we
observe an inverse scaling phenomenon, where larger models are more vulnerable
to this attack method. These findings underscore the urgent need for defense
mechanisms capable of understanding and preventing contextual attacks.
Furthermore, we introduce a cross-model attack strategy that leverages less
secure models to generate malicious contexts, thereby further increasing the
ASR when targeting other models.Our code and jailbreak artifacts can be found
at https://github.com/Lucas-TY/llm_Implicit_reference.


http://arxiv.org/abs/2410.03999
Impact of Regularization on Calibration and Robustness: from the Representation Space Perspective. (13%)
Jonghyun Park; Juyeop Kim; Jong-Seok Lee
  Recent studies have shown that regularization techniques using soft labels,
e.g., label smoothing, Mixup, and CutMix, not only enhance image classification
accuracy but also improve model calibration and robustness against adversarial
attacks. However, the underlying mechanisms of such improvements remain
underexplored. In this paper, we offer a novel explanation from the perspective
of the representation space (i.e., the space of the features obtained at the
penultimate layer). Our investigation first reveals that the decision regions
in the representation space form cone-like shapes around the origin after
training regardless of the presence of regularization. However, applying
regularization causes changes in the distribution of features (or
representation vectors). The magnitudes of the representation vectors are
reduced and subsequently the cosine similarities between the representation
vectors and the class centers (minimal loss points for each class) become
higher, which acts as a central mechanism inducing improved calibration and
robustness. Our findings provide new insights into the characteristics of the
high-dimensional representation space in relation to training and
regularization using soft labels.


http://arxiv.org/abs/2410.03373
Make Interval Bound Propagation great again. (9%)
Patryk Krukowski; Daniel Wilczak; Jacek Tabor; Anna Bielawska; Przemysław Spurek
  In various scenarios motivated by real life, such as medical data analysis,
autonomous driving, and adversarial training, we are interested in robust deep
networks. A network is robust when a relatively small perturbation of the input
cannot lead to drastic changes in output (like change of class, etc.). This
falls under the broader scope field of Neural Network Certification (NNC). Two
crucial problems in NNC are of profound interest to the scientific community:
how to calculate the robustness of a given pre-trained network and how to
construct robust networks. The common approach to constructing robust networks
is Interval Bound Propagation (IBP). This paper demonstrates that IBP is
sub-optimal in the first case due to its susceptibility to the wrapping effect.
Even for linear activation, IBP gives strongly sub-optimal bounds.
Consequently, one should use strategies immune to the wrapping effect to obtain
bounds close to optimal ones. We adapt two classical approaches dedicated to
strict computations -- Dubleton Arithmetic and Affine Arithmetic -- to mitigate
the wrapping effect in neural networks. These techniques yield precise results
for networks with linear activation functions, thus resisting the wrapping
effect. As a result, we achieve bounds significantly closer to the optimal
level than IBPs.


http://arxiv.org/abs/2410.03505
Classification-Denoising Networks. (9%)
Louis Thiry; Florentin Guth
  Image classification and denoising suffer from complementary issues of lack
of robustness or partially ignoring conditioning information. We argue that
they can be alleviated by unifying both tasks through a model of the joint
probability of (noisy) images and class labels. Classification is performed
with a forward pass followed by conditioning. Using the Tweedie-Miyasawa
formula, we evaluate the denoising function with the score, which can be
computed by marginalization and back-propagation. The training objective is
then a combination of cross-entropy loss and denoising score matching loss
integrated over noise levels. Numerical experiments on CIFAR-10 and ImageNet
show competitive classification and denoising performance compared to reference
deep convolutional classifiers/denoisers, and significantly improves efficiency
compared to previous joint approaches. Our model shows an increased robustness
to adversarial perturbations compared to a standard discriminative classifier,
and allows for a novel interpretation of adversarial gradients as a difference
of denoisers.


http://arxiv.org/abs/2410.09078
Knowledge-Augmented Reasoning for EUAIA Compliance and Adversarial Robustness of LLMs. (2%)
Tomas Bueno Momcilovic; Dian Balta; Beat Buesser; Giulio Zizzo; Mark Purcell
  The EU AI Act (EUAIA) introduces requirements for AI systems which intersect
with the processes required to establish adversarial robustness. However, given
the ambiguous language of regulation and the dynamicity of adversarial attacks,
developers of systems with highly complex models such as LLMs may find their
effort to be duplicated without the assurance of having achieved either
compliance or robustness. This paper presents a functional architecture that
focuses on bridging the two properties, by introducing components with clear
reference to their source. Taking the detection layer recommended by the
literature, and the reporting layer required by the law, we aim to support
developers and auditors with a reasoning layer based on knowledge augmentation
(rules, assurance cases, contextual mappings). Our findings demonstrate a novel
direction for ensuring LLMs deployed in the EU are both compliant and
adversarially robust, which underpin trustworthiness.


http://arxiv.org/abs/2410.03281
BN-SCAFFOLD: controlling the drift of Batch Normalization statistics in Federated Learning. (1%)
Gonzalo Iñaki Quintana; Laurence Vancamberg; Vincent Jugnon; Mathilde Mougeot; Agnès Desolneux
  Federated Learning (FL) is gaining traction as a learning paradigm for
training Machine Learning (ML) models in a decentralized way. Batch
Normalization (BN) is ubiquitous in Deep Neural Networks (DNN), as it improves
convergence and generalization. However, BN has been reported to hinder
performance of DNNs in heterogeneous FL. Recently, the FedTAN algorithm has
been proposed to mitigate the effect of heterogeneity on BN, by aggregating BN
statistics and gradients from all the clients. However, it has a high
communication cost, that increases linearly with the depth of the DNN. SCAFFOLD
is a variance reduction algorithm, that estimates and corrects the client drift
in a communication-efficient manner. Despite its promising results in
heterogeneous FL settings, it has been reported to underperform for models with
BN. In this work, we seek to revive SCAFFOLD, and more generally variance
reduction, as an efficient way of training DNN with BN in heterogeneous FL. We
introduce a unified theoretical framework for analyzing the convergence of
variance reduction algorithms in the BN-DNN setting, inspired of by the work of
Wang et al. 2023, and show that SCAFFOLD is unable to remove the bias
introduced by BN. We thus propose the BN-SCAFFOLD algorithm, which extends the
client drift correction of SCAFFOLD to BN statistics. We prove convergence
using the aforementioned framework and validate the theoretical results with
experiments on MNIST and CIFAR-10. BN-SCAFFOLD equals the performance of
FedTAN, without its high communication cost, outperforming Federated Averaging
(FedAvg), SCAFFOLD, and other FL algorithms designed to mitigate BN
heterogeneity.


http://arxiv.org/abs/2410.03869
Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step. (1%)
Wenxuan Wang; Kuiyi Gao; Zihan Jia; Youliang Yuan; Jen-tse Huang; Qiuzhi Liu; Shuai Wang; Wenxiang Jiao; Zhaopeng Tu
  Text-based image generation models, such as Stable Diffusion and DALL-E 3,
hold significant potential in content creation and publishing workflows, making
them the focus in recent years. Despite their remarkable capability to generate
diverse and vivid images, considerable efforts are being made to prevent the
generation of harmful content, such as abusive, violent, or pornographic
material. To assess the safety of existing models, we introduce a novel
jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises
image generation models through a step-by-step editing process. Specifically,
for malicious queries that cannot bypass the safeguards with a single prompt,
we intentionally decompose the query into multiple sub-queries. The image
generation models are then prompted to generate and iteratively edit images
based on these sub-queries. To evaluate the effectiveness of our CoJ attack
method, we constructed a comprehensive dataset, CoJ-Bench, encompassing nine
safety scenarios, three types of editing operations, and three editing
elements. Experiments on four widely-used image generation services provided by
GPT-4V, GPT-4o, Gemini 1.5 and Gemini 1.5 Pro, demonstrate that our CoJ attack
method can successfully bypass the safeguards of models for over 60% cases,
which significantly outperforms other jailbreaking methods (i.e., 14%).
Further, to enhance these models' safety against our CoJ attack method, we also
propose an effective prompting-based method, Think Twice Prompting, that can
successfully defend over 95% of CoJ attack. We release our dataset and code to
facilitate the AI safety research.


http://arxiv.org/abs/2410.02240
SCA: Highly Efficient Semantic-Consistent Unrestricted Adversarial Attack. (99%)
Zihao Pan; Weibin Wu; Yuhang Cao; Zibin Zheng
  Deep neural network based systems deployed in sensitive environments are
vulnerable to adversarial attacks. Unrestricted adversarial attacks typically
manipulate the semantic content of an image (e.g., color or texture) to create
adversarial examples that are both effective and photorealistic. Recent works
have utilized the diffusion inversion process to map images into a latent
space, where high-level semantics are manipulated by introducing perturbations.
However, they often results in substantial semantic distortions in the denoised
output and suffers from low efficiency. In this study, we propose a novel
framework called Semantic-Consistent Unrestricted Adversarial Attacks (SCA),
which employs an inversion method to extract edit-friendly noise maps and
utilizes Multimodal Large Language Model (MLLM) to provide semantic guidance
throughout the process. Under the condition of rich semantic information
provided by MLLM, we perform the DDPM denoising process of each step using a
series of edit-friendly noise maps, and leverage DPM Solver++ to accelerate
this process, enabling efficient sampling with semantic consistency. Compared
to existing methods, our framework enables the efficient generation of
adversarial examples that exhibit minimal discernible semantic changes.
Consequently, we for the first time introduce Semantic-Consistent Adversarial
Examples (SCAE). Extensive experiments and visualizations have demonstrated the
high efficiency of SCA, particularly in being on average 12 times faster than
the state-of-the-art attacks. Our research can further draw attention to the
security of multimedia information.


http://arxiv.org/abs/2410.02916
LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attacks. (47%)
Qingzhao Zhang; Ziyang Xiong; Z. Morley Mao
  Safety is a paramount concern for large language models (LLMs) in open
deployment, motivating the development of safeguard methods that enforce
ethical and responsible use through safety alignment or guardrail mechanisms.
Jailbreak attacks that exploit the \emph{false negatives} of safeguard methods
have emerged as a prominent research focus in the field of LLM security.
However, we found that the malicious attackers could also exploit false
positives of safeguards, i.e., fooling the safeguard model to block safe
content mistakenly, leading to a denial-of-service (DoS) affecting LLM users.
To bridge the knowledge gap of this overlooked threat, we explore multiple
attack methods that include inserting a short adversarial prompt into user
prompt templates and corrupting the LLM on the server by poisoned fine-tuning.
In both ways, the attack triggers safeguard rejections of user requests from
the client. Our evaluation demonstrates the severity of this threat across
multiple scenarios. For instance, in the scenario of white-box adversarial
prompt injection, the attacker can use our optimization process to
automatically generate seemingly safe adversarial prompts, approximately only
30 characters long, that universally block over 97% of user requests on Llama
Guard 3. These findings reveal a new dimension in LLM safeguard evaluation --
adversarial robustness to false positives.


http://arxiv.org/abs/2410.03000
Towards Universal Certified Robustness with Multi-Norm Training. (41%)
Enyi Jiang; David S. Cheung; Gagandeep Singh
  Existing certified training methods can only train models to be robust
against a certain perturbation type (e.g. $l_\infty$ or $l_2$). However, an
$l_\infty$ certifiably robust model may not be certifiably robust against $l_2$
perturbation (and vice versa) and also has low robustness against other
perturbations (e.g. geometric and patch transformation). By constructing a
theoretical framework to analyze and mitigate the tradeoff, we propose the
first multi-norm certified training framework \textbf{CURE}, consisting of
several multi-norm certified training methods, to attain better \emph{union
robustness} when training from scratch or fine-tuning a pre-trained certified
model. Inspired by our theoretical findings, we devise bound alignment and
connect natural training with certified training for better union robustness.
Compared with SOTA-certified training, \textbf{CURE} improves union robustness
to $32.0\%$ on MNIST, $25.8\%$ on CIFAR-10, and $10.6\%$ on TinyImagenet across
different epsilon values. It leads to better generalization on a diverse set of
challenging unseen geometric and patch perturbations to $6.8\%$ and $16.0\%$ on
CIFAR-10. Overall, our contributions pave a path towards \textit{universal
certified robustness}.


http://arxiv.org/abs/2410.02644
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. (15%)
Hanrong Zhang; Jingyuan Huang; Kai Mei; Yifei Yao; Zhenting Wang; Chenlu Zhan; Hongwei Wang; Yongfeng Zhang
  Although LLM-based agents, powered by Large Language Models (LLMs), can use
external tools and memory mechanisms to solve complex real-world tasks, they
may also introduce critical security vulnerabilities. However, the existing
literature does not comprehensively evaluate attacks and defenses against
LLM-based agents. To address this, we introduce Agent Security Bench (ASB), a
comprehensive framework designed to formalize, benchmark, and evaluate the
attacks and defenses of LLM-based agents, including 10 scenarios (e.g.,
e-commerce, autonomous driving, finance), 10 agents targeting the scenarios,
over 400 tools, 27 different types of attack/defense methods, and 7 evaluation
metrics. Based on ASB, we benchmark 10 prompt injection attacks, a memory
poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and
11 corresponding defenses across 13 LLM backbones. Our benchmark results reveal
critical vulnerabilities in different stages of agent operation, including
system prompt, user prompt handling, tool usage, and memory retrieval, with the
highest average attack success rate of 84.30\%, but limited effectiveness shown
in current defenses, unveiling important works to be done in terms of agent
security for the community. We also introduce a new metric to evaluate the
agents' capability to balance utility and security. Our code can be found at
https://github.com/agiresearch/ASB.


http://arxiv.org/abs/2410.05295
AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs. (11%)
Xiaogeng Liu; Peiran Li; Edward Suh; Yevgeniy Vorobeychik; Zhuoqing Mao; Somesh Jha; Patrick McDaniel; Huan Sun; Bo Li; Chaowei Xiao
  In this paper, we propose AutoDAN-Turbo, a black-box jailbreak method that
can automatically discover as many jailbreak strategies as possible from
scratch, without any human intervention or predefined scopes (e.g., specified
candidate strategies), and use them for red-teaming. As a result, AutoDAN-Turbo
can significantly outperform baseline methods, achieving a 74.3% higher average
attack success rate on public benchmarks. Notably, AutoDAN-Turbo achieves an
88.5 attack success rate on GPT-4-1106-turbo. In addition, AutoDAN-Turbo is a
unified framework that can incorporate existing human-designed jailbreak
strategies in a plug-and-play manner. By integrating human-designed strategies,
AutoDAN-Turbo can even achieve a higher attack success rate of 93.4 on
GPT-4-1106-turbo.


http://arxiv.org/abs/2410.02841
Demonstration Attack against In-Context Learning for Code Intelligence. (10%)
Yifei Ge; Weisong Sun; Yihang Lou; Chunrong Fang; Yiran Zhang; Yiming Li; Xiaofang Zhang; Yang Liu; Zhihong Zhao; Zhenyu Chen
  Recent advancements in large language models (LLMs) have revolutionized code
intelligence by improving programming productivity and alleviating challenges
faced by software developers. To further improve the performance of LLMs on
specific code intelligence tasks and reduce training costs, researchers reveal
a new capability of LLMs: in-context learning (ICL). ICL allows LLMs to learn
from a few demonstrations within a specific context, achieving impressive
results without parameter updating. However, the rise of ICL introduces new
security vulnerabilities in the code intelligence field. In this paper, we
explore a novel security scenario based on the ICL paradigm, where attackers
act as third-party ICL agencies and provide users with bad ICL content to
mislead LLMs outputs in code intelligence tasks. Our study demonstrates the
feasibility and risks of such a scenario, revealing how attackers can leverage
malicious demonstrations to construct bad ICL content and induce LLMs to
produce incorrect outputs, posing significant threats to system security. We
propose a novel method to construct bad ICL content called DICE, which is
composed of two stages: Demonstration Selection and Bad ICL Construction,
constructing targeted bad ICL content based on the user query and transferable
across different query inputs. Ultimately, our findings emphasize the critical
importance of securing ICL mechanisms to protect code intelligence systems from
adversarial manipulation.


http://arxiv.org/abs/2410.02440
Optimizing Adaptive Attacks against Watermarks for Language Models. (5%)
Abdulrahman Diaa; Toluwani Aremu; Nils Lukas
  Large Language Models (LLMs) can be misused to spread unwanted content at
scale. Content watermarking deters misuse by hiding messages in content,
enabling its detection using a secret watermarking key. Robustness is a core
security property, stating that evading detection requires (significant)
degradation of the content's quality. Many LLM watermarking methods have been
proposed, but robustness is tested only against non-adaptive attackers who lack
knowledge of the watermarking method and can find only suboptimal attacks. We
formulate watermark robustness as an objective function and use
preference-based optimization to tune adaptive attacks against the specific
watermarking method. Our evaluation shows that (i) adaptive attacks evade
detection against all surveyed watermarks, (ii) training against any watermark
succeeds in evading unseen watermarks, and (iii) optimization-based attacks are
cost-effective. Our findings underscore the need to test robustness against
adaptively tuned attacks. We release our adaptively optimized paraphrasers at
https://github.com/nilslukas/ada-wm-evasion.


http://arxiv.org/abs/2410.02970
F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI. (4%)
Xu Zheng; Farhad Shirani; Zhuomin Chen; Chaohao Lin; Wei Cheng; Wenbo Guo; Dongsheng Luo
  Recent research has developed a number of eXplainable AI (XAI) techniques,
such as gradient-based approaches, input perturbation-base methods, and
black-box explanation methods. While these XAI techniques can extract
meaningful insights from deep learning models, how to properly evaluate them
remains an open problem. The most widely used approach is to perturb or even
remove what the XAI method considers to be the most important features in an
input and observe the changes in the output prediction. This approach, although
straightforward, suffers the Out-of-Distribution (OOD) problem as the perturbed
samples may no longer follow the original data distribution. A recent method
RemOve And Retrain (ROAR) solves the OOD issue by retraining the model with
perturbed samples guided by explanations. However, using the model retrained
based on XAI methods to evaluate these explainers may cause information leakage
and thus lead to unfair comparisons. We propose Fine-tuned Fidelity
(F-Fidelity), a robust evaluation framework for XAI, which utilizes i) an
explanation-agnostic fine-tuning strategy, thus mitigating the information
leakage issue, and ii) a random masking operation that ensures that the removal
step does not generate an OOD input. We also design controlled experiments with
state-of-the-art (SOTA) explainers and their degraded version to verify the
correctness of our framework. We conduct experiments on multiple data
modalities, such as images, time series, and natural language. The results
demonstrate that F-Fidelity significantly improves upon prior evaluation
metrics in recovering the ground-truth ranking of the explainers. Furthermore,
we show both theoretically and empirically that, given a faithful explainer,
F-Fidelity metric can be used to compute the sparsity of influential input
components, i.e., to extract the true explanation size.


http://arxiv.org/abs/2410.02384
Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errors. (3%)
Shuangpeng Han; Mengmi Zhang
  AI models make mistakes when recognizing images-whether in-domain,
out-of-domain, or adversarial. Predicting these errors is critical for
improving system reliability, reducing costly mistakes, and enabling proactive
corrections in real-world applications such as healthcare, finance, and
autonomous systems. However, understanding what mistakes AI models make, why
they occur, and how to predict them remains an open challenge. Here, we conduct
comprehensive empirical evaluations using a "mentor" model-a deep neural
network designed to predict another "mentee" model's errors. Our findings show
that the mentor excels at learning from a mentee's mistakes on adversarial
images with small perturbations and generalizes effectively to predict
in-domain and out-of-domain errors of the mentee. Additionally,
transformer-based mentor models excel at predicting errors across various
mentee architectures. Subsequently, we draw insights from these observations
and develop an "oracle" mentor model, dubbed SuperMentor, that can outperform
baseline mentors in predicting errors across different error types from the
ImageNet-1K dataset. Our framework paves the way for future research on
anticipating and correcting AI model behaviors, ultimately increasing trust in
AI systems. All code, models, and data will be made publicly available.


http://arxiv.org/abs/2410.02298
Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models. (3%)
Guobin Shen; Dongcheng Zhao; Yiting Dong; Xiang He; Yi Zeng
  As large language models (LLMs) become integral to various applications,
ensuring both their safety and utility is paramount. Jailbreak attacks, which
manipulate LLMs into generating harmful content, pose significant challenges to
this balance. Existing defenses, such as prompt engineering and safety
fine-tuning, often introduce computational overhead, increase inference
latency, and lack runtime flexibility. Moreover, overly restrictive safety
measures can degrade model utility by causing refusals of benign queries. In
this paper, we introduce Jailbreak Antidote, a method that enables real-time
adjustment of LLM safety preferences by manipulating a sparse subset of the
model's internal states during inference. By shifting the model's hidden
representations along a safety direction with varying strengths, we achieve
flexible control over the safety-utility balance without additional token
overhead or inference delays. Our analysis reveals that safety-related
information in LLMs is sparsely distributed; adjusting approximately 5% of the
internal state is as effective as modifying the entire state. Extensive
experiments on nine LLMs (ranging from 2 billion to 72 billion parameters),
evaluated against ten jailbreak attack methods and compared with six defense
strategies, validate the effectiveness and efficiency of our approach. By
directly manipulating internal states during reasoning, Jailbreak Antidote
offers a lightweight, scalable solution that enhances LLM safety while
preserving utility, opening new possibilities for real-time safety mechanisms
in widely-deployed AI systems.


http://arxiv.org/abs/2410.02254
MTDNS: Moving Target Defense for Resilient DNS Infrastructure. (2%)
Abdullah Aydeger; Pei Zhou; Sanzida Hoque; Marco Carvalho; Engin Zeydan
  One of the most critical components of the Internet that an attacker could
exploit is the DNS (Domain Name System) protocol and infrastructure.
Researchers have been constantly developing methods to detect and defend
against the attacks against DNS, specifically DNS flooding attacks. However,
most solutions discard packets for defensive approaches, which can cause
legitimate packets to be dropped, making them highly dependable on detection
strategies. In this paper, we propose MTDNS, a resilient MTD-based approach
that employs Moving Target Defense techniques through Software Defined
Networking (SDN) switches to redirect traffic to alternate DNS servers that are
dynamically created and run under the Network Function Virtualization (NFV)
framework. The proposed approach is implemented in a testbed environment by
running our DNS servers as separate Virtual Network Functions, NFV Manager, SDN
switches, and an SDN Controller. The experimental result shows that the MTDNS
approach achieves a much higher success rate in resolving DNS queries and
significantly reduces average latency even if there is a DNS flooding attack.


http://arxiv.org/abs/2410.02506
Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems. (1%)
Guibin Zhang; Yanwei Yue; Zhixun Li; Sukwon Yun; Guancheng Wan; Kun Wang; Dawei Cheng; Jeffrey Xu Yu; Tianlong Chen
  Recent advancements in large language model (LLM)-powered agents have shown
that collective intelligence can significantly outperform individual
capabilities, largely attributed to the meticulously designed inter-agent
communication topologies. Though impressive in performance, existing
multi-agent pipelines inherently introduce substantial token overhead, as well
as increased economic costs, which pose challenges for their large-scale
deployments. In response to this challenge, we propose an economical, simple,
and robust multi-agent communication framework, termed $\texttt{AgentPrune}$,
which can seamlessly integrate into mainstream multi-agent systems and prunes
redundant or even malicious communication messages. Technically,
$\texttt{AgentPrune}$ is the first to identify and formally define the
\textit{communication redundancy} issue present in current LLM-based
multi-agent pipelines, and efficiently performs one-shot pruning on the
spatial-temporal message-passing graph, yielding a token-economic and
high-performing communication topology. Extensive experiments across six
benchmarks demonstrate that $\texttt{AgentPrune}$ \textbf{(I)} achieves
comparable results as state-of-the-art topologies at merely $\$5.6$ cost
compared to their $\$43.7$, \textbf{(II)} integrates seamlessly into existing
multi-agent frameworks with $28.1\%\sim72.8\%\downarrow$ token reduction, and
\textbf{(III)} successfully defend against two types of agent-based adversarial
attacks with $3.5\%\sim10.8\%\uparrow$ performance boost.


http://arxiv.org/abs/2410.02611
IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages? (1%)
Akhilesh Aravapalli; Mounika Marreddy; Subba Reddy Oota; Radhika Mamidi; Manish Gupta
  Transformer-based models have revolutionized the field of natural language
processing. To understand why they perform so well and to assess their
reliability, several studies have focused on questions such as: Which
linguistic properties are encoded by these models, and to what extent? How
robust are these models in encoding linguistic properties when faced with
perturbations in the input text? However, these studies have mainly focused on
BERT and the English language. In this paper, we investigate similar questions
regarding encoding capability and robustness for 8 linguistic properties across
13 different perturbations in 6 Indic languages, using 9 multilingual
Transformer models (7 universal and 2 Indic-specific). To conduct this study,
we introduce a novel multilingual benchmark dataset, IndicSentEval, containing
approximately $\sim$47K sentences. Surprisingly, our probing analysis of
surface, syntactic, and semantic properties reveals that while almost all
multilingual models demonstrate consistent encoding performance for English,
they show mixed results for Indic languages. As expected, Indic-specific
multilingual models capture linguistic properties in Indic languages better
than universal models. Intriguingly, universal models broadly exhibit better
robustness compared to Indic-specific models, particularly under perturbations
such as dropping both nouns and verbs, dropping only verbs, or keeping only
nouns. Overall, this study provides valuable insights into probing and
perturbation-specific strengths and weaknesses of popular multilingual
Transformer-based models for different Indic languages. We make our code and
dataset publicly available [https://tinyurl.com/IndicSentEval}].


http://arxiv.org/abs/2410.02195
BACKTIME: Backdoor Attacks on Multivariate Time Series Forecasting. (1%)
Xiao Lin; Zhining Liu; Dongqi Fu; Ruizhong Qiu; Hanghang Tong
  Multivariate Time Series (MTS) forecasting is a fundamental task with
numerous real-world applications, such as transportation, climate, and
epidemiology. While a myriad of powerful deep learning models have been
developed for this task, few works have explored the robustness of MTS
forecasting models to malicious attacks, which is crucial for their trustworthy
employment in high-stake scenarios. To address this gap, we dive deep into the
backdoor attacks on MTS forecasting models and propose an effective attack
method named BackTime.By subtly injecting a few stealthy triggers into the MTS
data, BackTime can alter the predictions of the forecasting model according to
the attacker's intent. Specifically, BackTime first identifies vulnerable
timestamps in the data for poisoning, and then adaptively synthesizes stealthy
and effective triggers by solving a bi-level optimization problem with a
GNN-based trigger generator. Extensive experiments across multiple datasets and
state-of-the-art MTS forecasting models demonstrate the effectiveness,
versatility, and stealthiness of \method{} attacks. The code is available at
\url{https://github.com/xiaolin-cs/BackTime}.


http://arxiv.org/abs/2410.02890
Universally Optimal Watermarking Schemes for LLMs: from Theory to Practice. (1%)
Haiyun He; Yepeng Liu; Ziqiao Wang; Yongyi Mao; Yuheng Bu
  Large Language Models (LLMs) boosts human efficiency but also poses misuse
risks, with watermarking serving as a reliable method to differentiate
AI-generated content from human-created text. In this work, we propose a novel
theoretical framework for watermarking LLMs. Particularly, we jointly optimize
both the watermarking scheme and detector to maximize detection performance,
while controlling the worst-case Type-I error and distortion in the watermarked
text. Within our framework, we characterize the universally minimum Type-II
error, showing a fundamental trade-off between detection performance and
distortion. More importantly, we identify the optimal type of detectors and
watermarking schemes. Building upon our theoretical analysis, we introduce a
practical, model-agnostic and computationally efficient token-level
watermarking algorithm that invokes a surrogate model and the Gumbel-max trick.
Empirical results on Llama-13B and Mistral-8$\times$7B demonstrate the
effectiveness of our method. Furthermore, we also explore how robustness can be
integrated into our theoretical framework, which provides a foundation for
designing future watermarking systems with improved resilience to adversarial
attacks.


http://arxiv.org/abs/2410.02220
Buckle Up: Robustifying LLMs at Every Customization Stage via Data Curation. (1%)
Xiaoqun Liu; Jiacheng Liang; Luoxi Tang; Chenyu You; Muchao Ye; Zhaohan Xi
  Large language models (LLMs) are extensively adapted for downstream
applications through a process known as "customization," with fine-tuning being
a common method for integrating domain-specific expertise. However, recent
studies have revealed a vulnerability that tuning LLMs with malicious samples
can compromise their robustness and amplify harmful content, an attack known as
"jailbreaking." To mitigate such attack, we propose an effective defensive
framework utilizing data curation to revise commonsense texts and enhance their
safety implication from the perspective of LLMs. The curated texts can mitigate
jailbreaking attacks at every stage of the customization process: before
customization to immunize LLMs against future jailbreak attempts, during
customization to neutralize jailbreaking risks, or after customization to
restore the compromised models. Since the curated data strengthens LLMs through
the standard fine-tuning workflow, we do not introduce additional modules
during LLM inference, thereby preserving the original customization process.
Experimental results demonstrate a substantial reduction in jailbreaking
effects, with up to a 100% success in generating responsible responses.
Notably, our method is effective even with commonsense texts, which are often
more readily available than safety-relevant data. With the every-stage
defensive framework and supporting experimental performance, this work
represents a significant advancement in mitigating jailbreaking risks and
ensuring the secure customization of LLMs.


http://arxiv.org/abs/2410.02760
Erasing Conceptual Knowledge from Language Models. (1%)
Rohit Gandikota; Sheridan Feucht; Samuel Marks; David Bau
  In this work, we propose Erasure of Language Memory (ELM), an approach for
concept-level unlearning built on the principle of matching the distribution
defined by an introspective classifier. Our key insight is that effective
unlearning should leverage the model's ability to evaluate its own knowledge,
using the model itself as a classifier to identify and reduce the likelihood of
generating content related to undesired concepts. ELM applies this framework to
create targeted low-rank updates that reduce generation probabilities for
concept-specific content while preserving the model's broader capabilities. We
demonstrate ELM's efficacy on biosecurity, cybersecurity, and literary domain
erasure tasks. Comparative analysis shows that ELM achieves superior
performance across key metrics, including near-random scores on erased topic
assessments, maintained coherence in text generation, preserved accuracy on
unrelated benchmarks, and robustness under adversarial attacks. Our code, data,
and trained models are available at https://elm.baulab.info


http://arxiv.org/abs/2410.02043
Impact of White-Box Adversarial Attacks on Convolutional Neural Networks. (99%)
Rakesh Podder; Sudipto Ghosh
  Autonomous vehicle navigation and healthcare diagnostics are among the many
fields where the reliability and security of machine learning models for image
data are critical. We conduct a comprehensive investigation into the
susceptibility of Convolutional Neural Networks (CNNs), which are widely used
for image data, to white-box adversarial attacks. We investigate the effects of
various sophisticated attacks -- Fast Gradient Sign Method, Basic Iterative
Method, Jacobian-based Saliency Map Attack, Carlini & Wagner, Projected
Gradient Descent, and DeepFool -- on CNN performance metrics, (e.g., loss,
accuracy), the differential efficacy of adversarial techniques in increasing
error rates, the relationship between perceived image quality metrics (e.g.,
ERGAS, PSNR, SSIM, and SAM) and classification performance, and the comparative
effectiveness of iterative versus single-step attacks. Using the MNIST,
CIFAR-10, CIFAR-100, and Fashio_MNIST datasets, we explore the effect of
different attacks on the CNNs performance metrics by varying the
hyperparameters of CNNs. Our study provides insights into the robustness of
CNNs against adversarial threats, pinpoints vulnerabilities, and underscores
the urgent need for developing robust defense mechanisms to protect CNNs and
ensuring their trustworthy deployment in real-world scenarios.


http://arxiv.org/abs/2410.01393
Signal Adversarial Examples Generation for Signal Detection Network via White-Box Attack. (99%)
Dongyang Li; Linyuan Wang; Guangwei Xiong; Bin Yan; Dekui Ma; Jinxian Peng
  With the development and application of deep learning in signal detection
tasks, the vulnerability of neural networks to adversarial attacks has also
become a security threat to signal detection networks. This paper defines a
signal adversarial examples generation model for signal detection network from
the perspective of adding perturbations to the signal. The model uses the
inequality relationship of L2-norm between time domain and time-frequency
domain to constrain the energy of signal perturbations. Building upon this
model, we propose a method for generating signal adversarial examples utilizing
gradient-based attacks and Short-Time Fourier Transform. The experimental
results show that under the constraint of signal perturbation energy ratio less
than 3%, our adversarial attack resulted in a 28.1% reduction in the mean
Average Precision (mAP), a 24.7% reduction in recall, and a 30.4% reduction in
precision of the signal detection network. Compared to random noise
perturbation of equivalent intensity, our adversarial attack demonstrates a
significant attack effect.


http://arxiv.org/abs/2410.01617
On Using Certified Training towards Empirical Robustness. (99%)
Palma Alessandro De; Serge Durand; Zakaria Chihani; François Terrier; Caterina Urban
  Adversarial training is arguably the most popular way to provide empirical
robustness against specific adversarial examples. While variants based on
multi-step attacks incur significant computational overhead, single-step
variants are vulnerable to a failure mode known as catastrophic overfitting,
which hinders their practical utility for large perturbations. A parallel line
of work, certified training, has focused on producing networks amenable to
formal guarantees of robustness against any possible attack. However, the wide
gap between the best-performing empirical and certified defenses has severely
limited the applicability of the latter. Inspired by recent developments in
certified training, which rely on a combination of adversarial attacks with
network over-approximations, and by the connections between local linearity and
catastrophic overfitting, we present experimental evidence on the practical
utility and limitations of using certified training towards empirical
robustness. We show that, when tuned for the purpose, a recent certified
training algorithm can prevent catastrophic overfitting on single-step attacks,
and that it can bridge the gap to multi-step baselines under appropriate
experimental settings. Finally, we present a conceptually simple regularizer
for network over-approximations that can achieve similar effects while markedly
reducing runtime.


http://arxiv.org/abs/2410.01697
MOREL: Enhancing Adversarial Robustness through Multi-Objective Representation Learning. (99%)
Sedjro Salomon Hotegni; Sebastian Peitz
  Extensive research has shown that deep neural networks (DNNs) are vulnerable
to slight adversarial perturbations$-$small changes to the input data that
appear insignificant but cause the model to produce drastically different
outputs. In addition to augmenting training data with adversarial examples
generated from a specific attack method, most of the current defense strategies
necessitate modifying the original model architecture components to improve
robustness or performing test-time data purification to handle adversarial
attacks. In this work, we demonstrate that strong feature representation
learning during training can significantly enhance the original model's
robustness. We propose MOREL, a multi-objective feature representation learning
approach, encouraging classification models to produce similar features for
inputs within the same class, despite perturbations. Our training method
involves an embedding space where cosine similarity loss and multi-positive
contrastive loss are used to align natural and adversarial features from the
model encoder and ensure tight clustering. Concurrently, the classifier is
motivated to achieve accurate predictions. Through extensive experiments, we
demonstrate that our approach significantly enhances the robustness of DNNs
against white-box and black-box adversarial attacks, outperforming other
methods that similarly require no architectural changes or test-time data
purification. Our code is available at https://github.com/salomonhotegni/MOREL


http://arxiv.org/abs/2410.01574
Adversarial Robustness of AI-Generated Image Detectors in the Real World. (99%)
Sina Mavali; Jonas Ricker; David Pape; Asja Fischer; Lea Schönherr
  The rapid advancement of Generative Artificial Intelligence (GenAI)
capabilities is accompanied by a concerning rise in its misuse. In particular
the generation of credible misinformation in the form of images poses a
significant threat to the public trust in democratic processes. Consequently,
there is an urgent need to develop tools to reliably distinguish between
authentic and AI-generated content. The majority of detection methods are based
on neural networks that are trained to recognize forensic artifacts. In this
work, we demonstrate that current state-of-the-art classifiers are vulnerable
to adversarial examples under real-world conditions. Through extensive
experiments, comprising four detection methods and five attack algorithms, we
show that an attacker can dramatically decrease classification performance,
without internal knowledge of the detector's architecture. Notably, most
attacks remain effective even when images are degraded during the upload to,
e.g., social media platforms. In a case study, we demonstrate that these
robustness challenges are also found in commercial tools by conducting
black-box attacks on HIVE, a proprietary online GenAI media detector. In
addition, we evaluate the robustness of using generated features of a robust
pre-trained model and showed that this increases the robustness, while not
reaching the performance on benign inputs. These results, along with the
increasing potential of GenAI to erode public trust, underscore the need for
more research and new perspectives on methods to prevent its misuse.


http://arxiv.org/abs/2410.01272
"No Matter What You Do": Purifying GNN Models via Backdoor Unlearning. (93%)
Jiale Zhang; Chengcheng Zhu; Bosen Rao; Hao Sui; Xiaobing Sun; Bing Chen; Chunyi Zhou; Shouling Ji
  Recent studies have exposed that GNNs are vulnerable to several adversarial
attacks, among which backdoor attack is one of the toughest. Similar to Deep
Neural Networks (DNNs), backdoor attacks in GNNs lie in the fact that the
attacker modifies a portion of graph data by embedding triggers and enforces
the model to learn the trigger feature during the model training process.
Despite the massive prior backdoor defense works on DNNs, defending against
backdoor attacks in GNNs is largely unexplored, severely hindering the
widespread application of GNNs in real-world tasks. To bridge this gap, we
present GCleaner, the first backdoor mitigation method on GNNs. GCleaner can
mitigate the presence of the backdoor logic within backdoored GNNs by reversing
the backdoor learning procedure, aiming to restore the model performance to a
level similar to that is directly trained on the original clean dataset. To
achieve this objective, we ask: How to recover universal and hard backdoor
triggers in GNNs? How to unlearn the backdoor trigger feature while maintaining
the model performance? We conduct the graph trigger recovery via the
explanation method to identify optimal trigger locations, facilitating the
search of universal and hard backdoor triggers in the feature space of the
backdoored model through maximal similarity. Subsequently, we introduce the
backdoor unlearning mechanism, which combines knowledge distillation and
gradient-based explainable knowledge for fine-grained backdoor erasure.
Extensive experimental evaluations on four benchmark datasets demonstrate that
GCleaner can reduce the backdoor attack success rate to 10% with only 1% of
clean data, and has almost negligible degradation in model performance, which
far outperforms the state-of-the-art (SOTA) defense methods.


http://arxiv.org/abs/2410.01906
Social Media Authentication and Combating Deepfakes using Semi-fragile Invisible Image Watermarking. (82%)
Aakash Varma Nadimpalli; Ajita Rattani
  With the significant advances in deep generative models for image and video
synthesis, Deepfakes and manipulated media have raised severe societal
concerns. Conventional machine learning classifiers for deepfake detection
often fail to cope with evolving deepfake generation technology and are
susceptible to adversarial attacks. Alternatively, invisible image watermarking
is being researched as a proactive defense technique that allows media
authentication by verifying an invisible secret message embedded in the image
pixels. A handful of invisible image watermarking techniques introduced for
media authentication have proven vulnerable to basic image processing
operations and watermark removal attacks. In response, we have proposed a
semi-fragile image watermarking technique that embeds an invisible secret
message into real images for media authentication. Our proposed watermarking
framework is designed to be fragile to facial manipulations or tampering while
being robust to benign image-processing operations and watermark removal
attacks. This is facilitated through a unique architecture of our proposed
technique consisting of critic and adversarial networks that enforce high image
quality and resiliency to watermark removal efforts, respectively, along with
the backbone encoder-decoder and the discriminator networks. Thorough
experimental investigations on SOTA facial Deepfake datasets demonstrate that
our proposed model can embed a $64$-bit secret as an imperceptible image
watermark that can be recovered with a high-bit recovery accuracy when benign
image processing operations are applied while being non-recoverable when unseen
Deepfake manipulations are applied. In addition, our proposed watermarking
technique demonstrates high resilience to several white-box and black-box
watermark removal attacks. Thus, obtaining state-of-the-art performance.


http://arxiv.org/abs/2410.01289
The Unlikely Hero: Nonideality in Analog Photonic Neural Networks as Built-in Defender Against Adversarial Attacks. (76%)
Haotian Lu; Ziang Yin; Partho Bhoumik; Sanmitra Banerjee; Krishnendu Chakrabarty; Jiaqi Gu
  Electronic-photonic computing systems have emerged as a promising platform
for accelerating deep neural network (DNN) workloads. Major efforts have been
focused on countering hardware non-idealities and boosting efficiency with
various hardware/algorithm co-design methods. However, the adversarial
robustness of such photonic analog mixed-signal AI hardware remains unexplored.
Though the hardware variations can be mitigated with robustness-driven
optimization methods, malicious attacks on the hardware show distinct behaviors
from noises, which requires a customized protection method tailored to optical
analog hardware. In this work, we rethink the role of conventionally undesired
non-idealities in photonic analog accelerators and claim their surprising
effects on defending against adversarial weight attacks. Inspired by the
protection effects from DNN quantization and pruning, we propose a synergistic
defense framework tailored for optical analog hardware that proactively
protects sensitive weights via pre-attack unary weight encoding and post-attack
vulnerability-aware weight locking. Efficiency-reliability trade-offs are
formulated as constrained optimization problems and efficiently solved offline
without model re-training costs. Extensive evaluation of various DNN benchmarks
with a multi-core photonic accelerator shows that our framework maintains
near-ideal on-chip inference accuracy under adversarial bit-flip attacks with
merely <3% memory overhead. Our codes are open-sourced at
https://github.com/ScopeX-ASU/Unlikely_Hero.


http://arxiv.org/abs/2410.01438
The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models? (38%)
Ching-Chia Kao; Chia-Mu Yu; Chun-Shien Lu; Chu-Song Chen
  Vision-Language Models (VLMs) have achieved remarkable performance on a
variety of tasks, yet they remain vulnerable to jailbreak attacks that
compromise safety and reliability. In this paper, we provide an
information-theoretic framework for understanding the fundamental trade-off
between the effectiveness of these attacks and their stealthiness. Drawing on
Fano's inequality, we demonstrate how an attacker's success probability is
intrinsically linked to the stealthiness of generated prompts. Building on
this, we propose an efficient algorithm for detecting non-stealthy jailbreak
attacks, offering significant improvements in model robustness. Experimental
results highlight the tension between strong attacks and their detectability,
providing insights into both adversarial strategies and defense mechanisms.


http://arxiv.org/abs/2410.01294
Endless Jailbreaks with Bijection Learning. (16%)
Brian R. Y. Huang; Maximilian Li; Leonard Tang
  Despite extensive safety measures, LLMs are vulnerable to adversarial inputs,
or jailbreaks, which can elicit unsafe behaviors. In this work, we introduce
bijection learning, a powerful attack algorithm which automatically fuzzes LLMs
for safety vulnerabilities using randomly-generated encodings whose complexity
can be tightly controlled. We leverage in-context learning to teach models
bijective encodings, pass encoded queries to the model to bypass built-in
safety mechanisms, and finally decode responses back into English. Our attack
is extremely effective on a wide range of frontier language models. Moreover,
by controlling complexity parameters such as number of key-value mappings in
the encodings, we find a close relationship between the capability level of the
attacked LLM and the average complexity of the most effective bijection
attacks. Our work highlights that new vulnerabilities in frontier models can
emerge with scale: more capable models are more severely jailbroken by
bijection attacks.


http://arxiv.org/abs/2410.02182
BadCM: Invisible Backdoor Attack Against Cross-Modal Learning. (13%)
Zheng Zhang; Xu Yuan; Lei Zhu; Jingkuan Song; Liqiang Nie
  Despite remarkable successes in unimodal learning tasks, backdoor attacks
against cross-modal learning are still underexplored due to the limited
generalization and inferior stealthiness when involving multiple modalities.
Notably, since works in this area mainly inherit ideas from unimodal visual
attacks, they struggle with dealing with diverse cross-modal attack
circumstances and manipulating imperceptible trigger samples, which hinders
their practicability in real-world applications. In this paper, we introduce a
novel bilateral backdoor to fill in the missing pieces of the puzzle in the
cross-modal backdoor and propose a generalized invisible backdoor framework
against cross-modal learning (BadCM). Specifically, a cross-modal mining scheme
is developed to capture the modality-invariant components as target poisoning
areas, where well-designed trigger patterns injected into these regions can be
efficiently recognized by the victim models. This strategy is adapted to
different image-text cross-modal models, making our framework available to
various attack scenarios. Furthermore, for generating poisoned samples of high
stealthiness, we conceive modality-specific generators for visual and
linguistic modalities that facilitate hiding explicit trigger patterns in
modality-invariant regions. To the best of our knowledge, BadCM is the first
invisible backdoor method deliberately designed for diverse cross-modal attacks
within one unified framework. Comprehensive experimental evaluations on two
typical applications, i.e., cross-modal retrieval and VQA, demonstrate the
effectiveness and generalization of our method under multiple kinds of attack
scenarios. Moreover, we show that BadCM can robustly evade existing backdoor
defenses. Our code is available at https://github.com/xandery-geek/BadCM.


http://arxiv.org/abs/2410.02163
Controlled Generation of Natural Adversarial Documents for Stealthy Retrieval Poisoning. (13%)
Collin Zhang; Tingwei Zhang; Vitaly Shmatikov
  Recent work showed that retrieval based on embedding similarity (e.g., for
retrieval-augmented generation) is vulnerable to poisoning: an adversary can
craft malicious documents that are retrieved in response to broad classes of
queries. We demonstrate that previous, HotFlip-based techniques produce
documents that are very easy to detect using perplexity filtering. Even if
generation is constrained to produce low-perplexity text, the resulting
documents are recognized as unnatural by LLMs and can be automatically filtered
from the retrieval corpus.
  We design, implement, and evaluate a new controlled generation technique that
combines an adversarial objective (embedding similarity) with a "naturalness"
objective based on soft scores computed using an open-source, surrogate LLM.
The resulting adversarial documents (1) cannot be automatically detected using
perplexity filtering and/or other LLMs, except at the cost of significant false
positives in the retrieval corpus, yet (2) achieve similar poisoning efficacy
to easily-detectable documents generated using HotFlip, and (3) are
significantly more effective than prior methods for energy-guided generation,
such as COLD.


http://arxiv.org/abs/2410.01606
Automated Red Teaming with GOAT: the Generative Offensive Agent Tester. (11%)
Maya Pavlova; Erik Brinkman; Krithika Iyer; Vitor Albiero; Joanna Bitton; Hailey Nguyen; Joe Li; Cristian Canton Ferrer; Ivan Evtimov; Aaron Grattafiori
  Red teaming assesses how large language models (LLMs) can produce content
that violates norms, policies, and rules set during their safety training.
However, most existing automated methods in the literature are not
representative of the way humans tend to interact with AI models. Common users
of AI models may not have advanced knowledge of adversarial machine learning
methods or access to model internals, and they do not spend a lot of time
crafting a single highly effective adversarial prompt. Instead, they are likely
to make use of techniques commonly shared online and exploit the multiturn
conversational nature of LLMs. While manual testing addresses this gap, it is
an inefficient and often expensive process. To address these limitations, we
introduce the Generative Offensive Agent Tester (GOAT), an automated agentic
red teaming system that simulates plain language adversarial conversations
while leveraging multiple adversarial prompting techniques to identify
vulnerabilities in LLMs. We instantiate GOAT with 7 red teaming attacks by
prompting a general-purpose model in a way that encourages reasoning through
the choices of methods available, the current target model's response, and the
next steps. Our approach is designed to be extensible and efficient, allowing
human testers to focus on exploring new areas of risk while automation covers
the scaled adversarial stress-testing of known risk territory. We present the
design and evaluation of GOAT, demonstrating its effectiveness in identifying
vulnerabilities in state-of-the-art LLMs, with an ASR@10 of 97% against Llama
3.1 and 88% against GPT-4 on the JailbreakBench dataset.


http://arxiv.org/abs/2410.01482
One Wave to Explain Them All: A Unifying Perspective on Post-hoc Explainability. (1%)
Gabriel Kasmi; Amandine Brunetto; Thomas Fel; Jayneel Parekh
  Despite the growing use of deep neural networks in safety-critical
decision-making, their inherent black-box nature hinders transparency and
interpretability. Explainable AI (XAI) methods have thus emerged to understand
a model's internal workings, and notably attribution methods also called
saliency maps. Conventional attribution methods typically identify the
locations -- the where -- of significant regions within an input. However,
because they overlook the inherent structure of the input data, these methods
often fail to interpret what these regions represent in terms of structural
components (e.g., textures in images or transients in sounds). Furthermore,
existing methods are usually tailored to a single data modality, limiting their
generalizability. In this paper, we propose leveraging the wavelet domain as a
robust mathematical foundation for attribution. Our approach, the Wavelet
Attribution Method (WAM) extends the existing gradient-based feature
attributions into the wavelet domain, providing a unified framework for
explaining classifiers across images, audio, and 3D shapes. Empirical
evaluations demonstrate that WAM matches or surpasses state-of-the-art methods
across faithfulness metrics and models in image, audio, and 3D explainability.
Finally, we show how our method explains not only the where -- the important
parts of the input -- but also the what -- the relevant patterns in terms of
structural components.


http://arxiv.org/abs/2410.00878
Empirical Perturbation Analysis of Linear System Solvers from a Data Poisoning Perspective. (54%)
Yixin Liu; Arielle Carr; Lichao Sun
  The perturbation analysis of linear solvers applied to systems arising
broadly in machine learning settings -- for instance, when using linear
regression models -- establishes an important perspective when reframing these
analyses through the lens of a data poisoning attack. By analyzing solvers'
responses to such attacks, this work aims to contribute to the development of
more robust linear solvers and provide insights into poisoning attacks on
linear solvers. In particular, we investigate how the errors in the input data
will affect the fitting error and accuracy of the solution from a linear
system-solving algorithm under perturbations common in adversarial attacks. We
propose data perturbation through two distinct knowledge levels, developing a
poisoning optimization and studying two methods of perturbation: Label-guided
Perturbation (LP) and Unconditioning Perturbation (UP). Existing works mainly
focus on deriving the worst-case perturbation bound from a theoretical
perspective, and the analysis is often limited to specific kinds of linear
system solvers. Under the circumstance that the data is intentionally perturbed
-- as is the case with data poisoning -- we seek to understand how different
kinds of solvers react to these perturbations, identifying those algorithms
most impacted by different types of adversarial attacks.


http://arxiv.org/abs/2410.00451
Adversarial Suffixes May Be Features Too! (45%)
Wei Zhao; Zhe Li; Yige Li; Jun Sun
  Despite significant ongoing efforts in safety alignment, large language
models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks
that can induce harmful behaviors, including those triggered by adversarial
suffixes. Building on prior research, we hypothesize that these adversarial
suffixes are not mere bugs but may represent features that can dominate the
LLM's behavior. To evaluate this hypothesis, we conduct several experiments.
First, we demonstrate that benign features can be effectively made to function
as adversarial suffixes, i.e., we develop a feature extraction method to
extract sample-agnostic features from benign dataset in the form of suffixes
and show that these suffixes may effectively compromise safety alignment.
Second, we show that adversarial suffixes generated from jailbreak attacks may
contain meaningful features, i.e., appending the same suffix to different
prompts results in responses exhibiting specific characteristics. Third, we
show that such benign-yet-safety-compromising features can be easily introduced
through fine-tuning using only benign datasets, i.e., even in the absence of
harmful content. This highlights the critical risk posed by dominating benign
features in the training data and calls for further research to reinforce LLM
safety alignment. Our code and data is available at
\url{https://github.com/suffix-maybe-feature/adver-suffix-maybe-features}.


http://arxiv.org/abs/2409.20139
Characterizing Model Robustness via Natural Input Gradients. (92%)
Adrián Rodríguez-Muñoz; Tongzhou Wang; Antonio Torralba
  Adversarially robust models are locally smooth around each data sample so
that small perturbations cannot drastically change model outputs. In modern
systems, such smoothness is usually obtained via Adversarial Training, which
explicitly enforces models to perform well on perturbed examples. In this work,
we show the surprising effectiveness of instead regularizing the gradient with
respect to model inputs on natural examples only. Penalizing input Gradient
Norm is commonly believed to be a much inferior approach. Our analyses identify
that the performance of Gradient Norm regularization critically depends on the
smoothness of activation functions, and are in fact extremely effective on
modern vision transformers that adopt smooth activations over piecewise linear
ones (eg, ReLU), contrary to prior belief. On ImageNet-1k, Gradient Norm
training achieves > 90% the performance of state-of-the-art PGD-3 Adversarial
Training} (52% vs.~56%), while using only 60% computation cost of the
state-of-the-art without complex adversarial optimization. Our analyses also
highlight the relationship between model robustness and properties of natural
input gradients, such as asymmetric sample and channel statistics.
Surprisingly, we find model robustness can be significantly improved by simply
regularizing its gradients to concentrate on image edges without explicit
conditioning on the gradient norm.


http://arxiv.org/abs/2409.20089
Robust LLM safeguarding via refusal feature adversarial training. (80%)
Lei Yu; Virginie Do; Karen Hambardzumyan; Nicola Cancedda
  Large language models (LLMs) are vulnerable to adversarial attacks that can
elicit harmful responses. Defending against such attacks remains challenging
due to the opacity of jailbreaking mechanisms and the high computational cost
of training LLMs robustly. We demonstrate that adversarial attacks share a
universal mechanism for circumventing LLM safeguards that works by ablating a
dimension in the residual stream embedding space called the refusal feature. We
further show that the operation of refusal feature ablation (RFA) approximates
the worst-case perturbation of offsetting model safety. Based on these
findings, we propose Refusal Feature Adversarial Training (ReFAT), a novel
algorithm that efficiently performs LLM adversarial training by simulating the
effect of input-level attacks via RFA. Experiment results show that ReFAT
significantly improves the robustness of three popular LLMs against a wide
range of adversarial attacks, with considerably less computational overhead
compared to existing adversarial training methods.


http://arxiv.org/abs/2410.00126
Resonance Reduction Against Adversarial Attacks in Dynamic Networks via Eigenspectrum Optimization. (76%)
Alp Sahin; Nicolas Kozachuk; Rick S. Blum; Subhrajit Bhattacharya
  Resonance is a well-known phenomenon that happens in systems with second
order dynamics. In this paper we address the fundamental question of making a
network robust to signal being periodically pumped into it at or near a
resonant frequency by an adversarial agent with the aim of saturating the
network with the signal. Towards this goal, we develop the notion of network
vulnerability, which is measured by the expected resonance amplitude on the
network under a stochastically modeled adversarial attack. Assuming a second
order dynamics model based on the network graph Laplacian matrix and a known
stochastic model for the adversarial attack, we propose two methods for
minimizing the network vulnerability that leverage the principle of
eigenspectrum optimization. We provide extensive numerical results analyzing
the effects of both methods.


http://arxiv.org/abs/2409.20426
Navigating Threats: A Survey of Physical Adversarial Attacks on LiDAR Perception Systems in Autonomous Vehicles. (45%)
Amira Guesmi; Muhammad Shafique
  Autonomous vehicles (AVs) rely heavily on LiDAR (Light Detection and Ranging)
systems for accurate perception and navigation, providing high-resolution 3D
environmental data that is crucial for object detection and classification.
However, LiDAR systems are vulnerable to adversarial attacks, which pose
significant challenges to the safety and robustness of AVs. This survey
presents a thorough review of the current research landscape on physical
adversarial attacks targeting LiDAR-based perception systems, covering both
single-modality and multi-modality contexts. We categorize and analyze various
attack types, including spoofing and physical adversarial object attacks,
detailing their methodologies, impacts, and potential real-world implications.
Through detailed case studies and analyses, we identify critical challenges and
highlight gaps in existing attacks for LiDAR-based systems. Additionally, we
propose future research directions to enhance the security and resilience of
these systems, ultimately contributing to the safer deployment of autonomous
vehicles.


http://arxiv.org/abs/2410.00296
VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data. (8%)
Xuefeng Du; Reshmi Ghosh; Robert Sim; Ahmed Salem; Vitor Carvalho; Emily Lawton; Yixuan Li; Jack W. Stokes
  Vision-language models (VLMs) are essential for contextual understanding of
both visual and textual information. However, their vulnerability to
adversarially manipulated inputs presents significant risks, leading to
compromised outputs and raising concerns about the reliability in
VLM-integrated applications. Detecting these malicious prompts is thus crucial
for maintaining trust in VLM generations. A major challenge in developing a
safeguarding prompt classifier is the lack of a large amount of labeled benign
and malicious data. To address the issue, we introduce VLMGuard, a novel
learning framework that leverages the unlabeled user prompts in the wild for
malicious prompt detection. These unlabeled prompts, which naturally arise when
VLMs are deployed in the open world, consist of both benign and malicious
information. To harness the unlabeled data, we present an automated
maliciousness estimation score for distinguishing between benign and malicious
samples within this unlabeled mixture, thereby enabling the training of a
binary prompt classifier on top. Notably, our framework does not require extra
human annotations, offering strong flexibility and practicality for real-world
applications. Extensive experiment shows VLMGuard achieves superior detection
results, significantly outperforming state-of-the-art methods. Disclaimer: This
paper may contain offensive examples; reader discretion is advised.


http://arxiv.org/abs/2409.19594
MASKDROID: Robust Android Malware Detection with Masked Graph Representations. (99%)
Jingnan Zheng; Jiaohao Liu; An Zhang; Jun Zeng; Ziqi Yang; Zhenkai Liang; Tat-Seng Chua
  Android malware attacks have posed a severe threat to mobile users,
necessitating a significant demand for the automated detection system. Among
the various tools employed in malware detection, graph representations (e.g.,
function call graphs) have played a pivotal role in characterizing the
behaviors of Android apps. However, though achieving impressive performance in
malware detection, current state-of-the-art graph-based malware detectors are
vulnerable to adversarial examples. These adversarial examples are meticulously
crafted by introducing specific perturbations to normal malicious inputs. To
defend against adversarial attacks, existing defensive mechanisms are typically
supplementary additions to detectors and exhibit significant limitations, often
relying on prior knowledge of adversarial examples and failing to defend
against unseen types of attacks effectively. In this paper, we propose
MASKDROID, a powerful detector with a strong discriminative ability to identify
malware and remarkable robustness against adversarial attacks. Specifically, we
introduce a masking mechanism into the Graph Neural Network (GNN) based
framework, forcing MASKDROID to recover the whole input graph using a small
portion (e.g., 20%) of randomly selected nodes.This strategy enables the model
to understand the malicious semantics and learn more stable representations,
enhancing its robustness against adversarial attacks. While capturing stable
malicious semantics in the form of dependencies inside the graph structures, we
further employ a contrastive module to encourage MASKDROID to learn more
compact representations for both the benign and malicious classes to boost its
discriminative power in detecting malware from benign apps and adversarial
examples.


http://arxiv.org/abs/2409.19788
Adversarial Examples for DNA Classification. (98%)
Hyunwoo Yoo
  Pre-trained language models such as DNABERT2 and Nucleotide Transformer,
which are trained on DNA sequences, have shown promising performance in DNA
sequence classification tasks. The classification ability of these models stems
from language models trained on vast amounts of DNA sequence samples, followed
by fine-tuning with relatively smaller classification datasets. However, these
text-based systems are not robust enough and can be vulnerable to adversarial
examples. While adversarial attacks have been widely studied in text
classification, there is limited research in DNA sequence classification. In
this paper, we adapt commonly used attack algorithms in text classification for
DNA sequence classification. We evaluated the impact of various attack methods
on DNA sequence classification at the character, word, and sentence levels. Our
findings indicate that actual DNA language model sequence classifiers are
vulnerable to these attacks.


http://arxiv.org/abs/2409.19619
Discerning the Chaos: Detecting Adversarial Perturbations while Disentangling Intentional from Unintentional Noises. (86%)
Anubhooti Jain; Susim Roy; Kwanit Gupta; Mayank Vatsa; Richa Singh
  Deep learning models, such as those used for face recognition and attribute
prediction, are susceptible to manipulations like adversarial noise and
unintentional noise, including Gaussian and impulse noise. This paper
introduces CIAI, a Class-Independent Adversarial Intent detection network built
on a modified vision transformer with detection layers. CIAI employs a novel
loss function that combines Maximum Mean Discrepancy and Center Loss to detect
both intentional (adversarial attacks) and unintentional noise, regardless of
the image class. It is trained in a multi-step fashion. We also introduce the
aspect of intent during detection that can act as an added layer of security.
We further showcase the performance of our proposed detector on CelebA,
CelebA-HQ, LFW, AgeDB, and CIFAR-10 datasets. Our detector is able to detect
both intentional (like FGSM, PGD, and DeepFool) and unintentional (like
Gaussian and Salt & Pepper noises) perturbations.


http://arxiv.org/abs/2409.19671
Nonideality-aware training makes memristive networks more robust to adversarial attacks. (38%)
Dovydas Joksas; Luis Muñoz-González; Emil Lupu; Adnan Mehonic
  Neural networks are now deployed in a wide number of areas from object
classification to natural language systems. Implementations using analog
devices like memristors promise better power efficiency, potentially bringing
these applications to a greater number of environments. However, such systems
suffer from more frequent device faults and overall, their exposure to
adversarial attacks has not been studied extensively. In this work, we
investigate how nonideality-aware training - a common technique to deal with
physical nonidealities - affects adversarial robustness. We find that
adversarial robustness is significantly improved, even with limited knowledge
of what nonidealities will be encountered during test time.


http://arxiv.org/abs/2409.19601
Infighting in the Dark: Multi-Labels Backdoor Attack in Federated Learning. (33%)
Ye Li; Yanchao Zhao; Chengcheng Zhu; Jiale Zhang
  Federated Learning (FL), a privacy-preserving decentralized machine learning
framework, has been shown to be vulnerable to backdoor attacks. Current
research primarily focuses on the Single-Label Backdoor Attack (SBA), wherein
adversaries share a consistent target. However, a critical fact is overlooked:
adversaries may be non-cooperative, have distinct targets, and operate
independently, which exhibits a more practical scenario called Multi-Label
Backdoor Attack (MBA). Unfortunately, prior works are ineffective in MBA
scenario since non-cooperative attackers exclude each other. In this work, we
conduct an in-depth investigation to uncover the inherent constraints of the
exclusion: similar backdoor mappings are constructed for different targets,
resulting in conflicts among backdoor functions. To address this limitation, we
propose Mirage, the first non-cooperative MBA strategy in FL that allows
attackers to inject effective and persistent backdoors into the global model
without collusion by constructing in-distribution (ID) backdoor mapping.
Specifically, we introduce an adversarial adaptation method to bridge the
backdoor features and the target distribution in an ID manner. Additionally, we
further leverage a constrained optimization method to ensure the ID mapping
survives in the global training dynamics. Extensive evaluations demonstrate
that Mirage outperforms various state-of-the-art attacks and bypasses existing
defenses, achieving an average ASR greater than 97\% and maintaining over 90\%
after 900 rounds. This work aims to alert researchers to this potential threat
and inspire the design of effective defense mechanisms. Code has been made
open-source.


http://arxiv.org/abs/2409.19766
Towards Robust Extractive Question Answering Models: Rethinking the Training Methodology. (10%)
Son Quoc Tran; Matt Kretchmar
  This paper proposes a novel training method to improve the robustness of
Extractive Question Answering (EQA) models. Previous research has shown that
existing models, when trained on EQA datasets that include unanswerable
questions, demonstrate a significant lack of robustness against distribution
shifts and adversarial attacks. Despite this, the inclusion of unanswerable
questions in EQA training datasets is essential for ensuring real-world
reliability. Our proposed training method includes a novel loss function for
the EQA problem and challenges an implicit assumption present in numerous EQA
datasets. Models trained with our method maintain in-domain performance while
achieving a notable improvement on out-of-domain datasets. This results in an
overall F1 score improvement of 5.7 across all testing sets. Furthermore, our
models exhibit significantly enhanced robustness against two types of
adversarial attacks, with a performance decrease of only about a third compared
to the default models.


http://arxiv.org/abs/2409.19746
Learning Robust Policies via Interpretable Hamilton-Jacobi Reachability-Guided Disturbances. (5%)
Hanyang Hu; Xilun Zhang; Xubo Lyu; Mo Chen
  Deep Reinforcement Learning (RL) has shown remarkable success in robotics
with complex and heterogeneous dynamics. However, its vulnerability to unknown
disturbances and adversarial attacks remains a significant challenge. In this
paper, we propose a robust policy training framework that integrates
model-based control principles with adversarial RL training to improve
robustness without the need for external black-box adversaries. Our approach
introduces a novel Hamilton-Jacobi reachability-guided disturbance for
adversarial RL training, where we use interpretable worst-case or
near-worst-case disturbances as adversaries against the robust policy. We
evaluated its effectiveness across three distinct tasks: a reach-avoid game in
both simulation and real-world settings, and a highly dynamic quadrotor
stabilization task in simulation. We validate that our learned critic network
is consistent with the ground-truth HJ value function, while the policy network
shows comparable performance with other learning-based methods.


http://arxiv.org/abs/2409.19627
IDEAW: Robust Neural Audio Watermarking with Invertible Dual-Embedding. (1%)
Pengcheng Li; Xulong Zhang; Jing Xiao; Jianzong Wang
  The audio watermarking technique embeds messages into audio and accurately
extracts messages from the watermarked audio. Traditional methods develop
algorithms based on expert experience to embed watermarks into the time-domain
or transform-domain of signals. With the development of deep neural networks,
deep learning-based neural audio watermarking has emerged. Compared to
traditional algorithms, neural audio watermarking achieves better robustness by
considering various attacks during training. However, current neural
watermarking methods suffer from low capacity and unsatisfactory
imperceptibility. Additionally, the issue of watermark locating, which is
extremely important and even more pronounced in neural audio watermarking, has
not been adequately studied. In this paper, we design a dual-embedding
watermarking model for efficient locating. We also consider the impact of the
attack layer on the invertible neural network in robustness training, improving
the model to enhance both its reasonableness and stability. Experiments show
that the proposed model, IDEAW, can withstand various attacks with higher
capacity and more efficient locating ability compared to existing methods.


http://arxiv.org/abs/2409.19808
Can Models Learn Skill Composition from Examples? (1%)
Haoyu Zhao; Simran Kaur; Dingli Yu; Anirudh Goyal; Sanjeev Arora
  As large language models (LLMs) become increasingly advanced, their ability
to exhibit compositional generalization -- the capacity to combine learned
skills in novel ways not encountered during training -- has garnered
significant attention. This type of generalization, particularly in scenarios
beyond training data, is also of great interest in the study of AI safety and
alignment. A recent study introduced the SKILL-MIX evaluation, where models are
tasked with composing a short paragraph demonstrating the use of a specified
$k$-tuple of language skills. While small models struggled with composing even
with $k=3$, larger models like GPT-4 performed reasonably well with $k=5$ and
$6$.
  In this paper, we employ a setup akin to SKILL-MIX to evaluate the capacity
of smaller models to learn compositional generalization from examples.
Utilizing a diverse set of language skills -- including rhetorical, literary,
reasoning, theory of mind, and common sense -- GPT-4 was used to generate text
samples that exhibit random subsets of $k$ skills. Subsequent fine-tuning of 7B
and 13B parameter models on these combined skill texts, for increasing values
of $k$, revealed the following findings: (1) Training on combinations of $k=2$
and $3$ skills results in noticeable improvements in the ability to compose
texts with $k=4$ and $5$ skills, despite models never having seen such examples
during training. (2) When skill categories are split into training and held-out
groups, models significantly improve at composing texts with held-out skills
during testing despite having only seen training skills during fine-tuning,
illustrating the efficacy of the training approach even with previously unseen
skills. This study also suggests that incorporating skill-rich (potentially
synthetic) text into training can substantially enhance the compositional
capabilities of models.


http://arxiv.org/abs/2409.19526
Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats. (74%)
Kuanrong Liu; Siyuan Liang; Jiawei Liang; Pengwen Dai; Xiaochun Cao
  Multimodal contrastive learning uses various data modalities to create
high-quality features, but its reliance on extensive data sources on the
Internet makes it vulnerable to backdoor attacks. These attacks insert
malicious behaviors during training, which are activated by specific triggers
during inference, posing significant security risks. Despite existing
countermeasures through fine-tuning that reduce the malicious impacts of such
attacks, these defenses frequently necessitate extensive training time and
degrade clean accuracy. In this study, we propose an efficient defense
mechanism against backdoor threats using a concept known as machine unlearning.
This entails strategically creating a small set of poisoned samples to aid the
model's rapid unlearning of backdoor vulnerabilities, known as Unlearn Backdoor
Threats (UBT). We specifically use overfit training to improve backdoor
shortcuts and accurately detect suspicious samples in the potential poisoning
data set. Then, we select fewer unlearned samples from suspicious samples for
rapid forgetting in order to eliminate the backdoor effect and thus improve
backdoor defense efficiency. In the backdoor unlearning process, we present a
novel token-based portion unlearning training regime. This technique focuses on
the model's compromised elements, dissociating backdoor correlations while
maintaining the model's overall integrity. Extensive experimental results show
that our method effectively defends against various backdoor attack methods in
the CLIP model. Compared to SoTA backdoor defense methods, UBT achieves the
lowest attack success rate while maintaining a high clean accuracy of the model
(attack success rate decreases by 19% compared to SOTA, while clean accuracy
increases by 2.57%).


http://arxiv.org/abs/2409.19521
GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attacks. (13%)
Rongchang Li; Minjie Chen; Chang Hu; Han Chen; Wenpeng Xing; Meng Han
  Large Language Models (LLMs) like GPT-4, LLaMA, and Qwen have demonstrated
remarkable success across a wide range of applications. However, these models
remain inherently vulnerable to prompt injection attacks, which can bypass
existing safety mechanisms, highlighting the urgent need for more robust attack
detection methods and comprehensive evaluation benchmarks. To address these
challenges, we introduce GenTel-Safe, a unified framework that includes a novel
prompt injection attack detection method, GenTel-Shield, along with a
comprehensive evaluation benchmark, GenTel-Bench, which compromises 84812
prompt injection attacks, spanning 3 major categories and 28 security
scenarios. To prove the effectiveness of GenTel-Shield, we evaluate it together
with vanilla safety guardrails against the GenTel-Bench dataset. Empirically,
GenTel-Shield can achieve state-of-the-art attack detection success rates,
which reveals the critical weakness of existing safeguarding techniques against
harmful prompts. For reproducibility, we have made the code and benchmarking
dataset available on the project page at
https://gentellab.github.io/gentel-safe.github.io/.


http://arxiv.org/abs/2409.19302
Leveraging MTD to Mitigate Poisoning Attacks in Decentralized FL with Non-IID Data. (11%)
Chao Feng; Alberto Huertas Celdrán; Zien Zeng; Zi Ye; der Assen Jan von; Gerome Bovet; Burkhard Stiller
  Decentralized Federated Learning (DFL), a paradigm for managing big data in a
privacy-preserved manner, is still vulnerable to poisoning attacks where
malicious clients tamper with data or models. Current defense methods often
assume Independently and Identically Distributed (IID) data, which is
unrealistic in real-world applications. In non-IID contexts, existing defensive
strategies face challenges in distinguishing between models that have been
compromised and those that have been trained on heterogeneous data
distributions, leading to diminished efficacy. In response, this paper proposes
a framework that employs the Moving Target Defense (MTD) approach to bolster
the robustness of DFL models. By continuously modifying the attack surface of
the DFL system, this framework aims to mitigate poisoning attacks effectively.
The proposed MTD framework includes both proactive and reactive modes,
utilizing a reputation system that combines metrics of model similarity and
loss, alongside various defensive techniques. Comprehensive experimental
evaluations indicate that the MTD-based mechanism significantly mitigates a
range of poisoning attack types across multiple datasets with different
topologies.


http://arxiv.org/abs/2410.00055
Survey of Security and Data Attacks on Machine Unlearning In Financial and E-Commerce. (2%)
Carl E. J. Brodzinski
  This paper surveys the landscape of security and data attacks on machine
unlearning, with a focus on financial and e-commerce applications. We discuss
key privacy threats such as Membership Inference Attacks and Data
Reconstruction Attacks, where adversaries attempt to infer or reconstruct data
that should have been removed. In addition, we explore security attacks
including Machine Unlearning Data Poisoning, Unlearning Request Attacks, and
Machine Unlearning Jailbreak Attacks, which target the underlying mechanisms of
unlearning to manipulate or corrupt the model. To mitigate these risks, various
defense strategies are examined, including differential privacy, robust
cryptographic guarantees, and Zero-Knowledge Proofs (ZKPs), offering verifiable
and tamper-proof unlearning mechanisms. These approaches are essential for
safeguarding data integrity and privacy in high-stakes financial and e-commerce
contexts, where compromised models can lead to fraud, data leaks, and
reputational damage. This survey highlights the need for continued research and
innovation in secure machine unlearning, as well as the importance of
developing strong defenses against evolving attack vectors.


http://arxiv.org/abs/2409.19301
Privacy Attack in Federated Learning is Not Easy: An Experimental Study. (1%)
Hangyu Zhu; Liyuan Huang; Zhenping Xie
  Federated learning (FL) is an emerging distributed machine learning paradigm
proposed for privacy preservation. Unlike traditional centralized learning
approaches, FL enables multiple users to collaboratively train a shared global
model without disclosing their own data, thereby significantly reducing the
potential risk of privacy leakage. However, recent studies have indicated that
FL cannot entirely guarantee privacy protection, and attackers may still be
able to extract users' private data through the communicated model gradients.
Although numerous privacy attack FL algorithms have been developed, most are
designed to reconstruct private data from a single step of calculated
gradients. It remains uncertain whether these methods are effective in
realistic federated environments or if they have other limitations. In this
paper, we aim to help researchers better understand and evaluate the
effectiveness of privacy attacks on FL. We analyze and discuss recent research
papers on this topic and conduct experiments in a real FL environment to
compare the performance of various attack methods. Our experimental results
reveal that none of the existing state-of-the-art privacy attack algorithms can
effectively breach private client data in realistic FL settings, even in the
absence of defense strategies. This suggests that privacy attacks in FL are
more challenging than initially anticipated.


http://arxiv.org/abs/2409.18736
Adversarial Challenges in Network Intrusion Detection Systems: Research Insights and Future Prospects. (96%)
Sabrine Ennaji; Gaspari Fabio De; Dorjan Hitaj; Alicia K/Bidi; Luigi V. Mancini
  Machine learning has brought significant advances in cybersecurity,
particularly in the area of intrusion detection systems. This improvements can
be mostly attributed to the ability of machine learning algorithms to identify
complex relations between features in the data and to generalize well to unseen
samples. Deep neural networks in particular contributed to this progress by
enabling the analysis of large amounts of training data, significantly
enhancing detection performance. However, machine learning models are
vulnerable to adversarial attacks: manipulations of input data designed to
mislead the models into making incorrect predictions. While much attention has
been given to adversarial threats in unstructured data such as text and images,
their effectiveness in structured data such as network traffic has not been as
thoroughly explored.
  This survey seeks to fill this gap by providing an critical review of machine
learning-based Network Intrusion Detection Systems (NIDS) and a thorough
analysis of their vulnerability to adversarial attacks. We critically review
existing NIDS research, highlighting key trends, strengths, and limitations,
and we identify gaps in understanding that require further exploration. We
further discuss emerging challenges and offer insights for developing more
robust and resilient NIDS models. In summary, this paper aims to enhance
understanding of adversarial attacks and defenses in NIDS and guide future
research in improving the robustness of machine learning models in
cybersecurity applications.


http://arxiv.org/abs/2409.19096
Enhancing Robustness of Graph Neural Networks through p-Laplacian. (12%)
Anuj Kumar Sirohi; Subhanu Halder; Kabir Kumar; Sandeep Kumar
  With the increase of data in day-to-day life, businesses and different
stakeholders need to analyze the data for better predictions. Traditionally,
relational data has been a source of various insights, but with the increase in
computational power and the need to understand deeper relationships between
entities, the need to design new techniques has arisen. For this graph data
analysis has become an extraordinary tool for understanding the data, which
reveals more realistic and flexible modelling of complex relationships.
Recently, Graph Neural Networks (GNNs) have shown great promise in various
applications, such as social network analysis, recommendation systems, drug
discovery, and more. However, many adversarial attacks can happen over the
data, whether during training (poisoning attack) or during testing (evasion
attack), which can adversely manipulate the desired outcome from the GNN model.
Therefore, it is crucial to make the GNNs robust to such attacks. The existing
robustness methods are computationally demanding and perform poorly when the
intensity of attack increases. This paper presents a computationally efficient
framework, namely, pLapGNN, based on weighted p-Laplacian for making GNNs
robust. Empirical evaluation on real datasets establishes the efficacy and
efficiency of the proposed method.


http://arxiv.org/abs/2409.18553
Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators. (1%)
Seyedarmin Azizi; Mohammad Erfan Sadeghi; Mehdi Kamal; Massoud Pedram
  In this paper, we propose a framework to enhance the robustness of the neural
models by mitigating the effects of process-induced and aging-related
variations of analog computing components on the accuracy of the analog neural
networks. We model these variations as the noise affecting the precision of the
activations and introduce a denoising block inserted between selected layers of
a pre-trained model. We demonstrate that training the denoising block
significantly increases the model's robustness against various noise levels. To
minimize the overhead associated with adding these blocks, we present an
exploration algorithm to identify optimal insertion points for the denoising
blocks. Additionally, we propose a specialized architecture to efficiently
execute the denoising blocks, which can be integrated into mixed-signal
accelerators. We evaluate the effectiveness of our approach using Deep Neural
Network (DNN) models trained on the ImageNet and CIFAR-10 datasets. The results
show that on average, by accepting 2.03% parameter count overhead, the accuracy
drop due to the variations reduces from 31.7% to 1.15%.


http://arxiv.org/abs/2409.18907
In-depth Analysis of Privacy Threats in Federated Learning for Medical Data. (1%)
Badhan Chandra Das; M. Hadi Amini; Yanzhao Wu
  Federated learning is emerging as a promising machine learning technique in
the medical field for analyzing medical images, as it is considered an
effective method to safeguard sensitive patient data and comply with privacy
regulations. However, recent studies have revealed that the default settings of
federated learning may inadvertently expose private training data to privacy
attacks. Thus, the intensity of such privacy risks and potential mitigation
strategies in the medical domain remain unclear. In this paper, we make three
original contributions to privacy risk analysis and mitigation in federated
learning for medical data. First, we propose a holistic framework, MedPFL, for
analyzing privacy risks in processing medical data in the federated learning
environment and developing effective mitigation strategies for protecting
privacy. Second, through our empirical analysis, we demonstrate the severe
privacy risks in federated learning to process medical images, where
adversaries can accurately reconstruct private medical images by performing
privacy attacks. Third, we illustrate that the prevalent defense mechanism of
adding random noises may not always be effective in protecting medical images
against privacy attacks in federated learning, which poses unique and pressing
challenges related to protecting the privacy of medical data. Furthermore, the
paper discusses several unique research questions related to the privacy
protection of medical data in the federated learning environment. We conduct
extensive experiments on several benchmark medical image datasets to analyze
and mitigate the privacy risks associated with federated learning for medical
data.


http://arxiv.org/abs/2409.17568
Showing Many Labels in Multi-label Classification Models: An Empirical Study of Adversarial Examples. (98%)
Yujiang Liu; Wenjian Luo; Zhijian Chen; Muhammad Luqman Naseem
  With the rapid development of Deep Neural Networks (DNNs), they have been
applied in numerous fields. However, research indicates that DNNs are
susceptible to adversarial examples, and this is equally true in the
multi-label domain. To further investigate multi-label adversarial examples, we
introduce a novel type of attacks, termed "Showing Many Labels". The objective
of this attack is to maximize the number of labels included in the classifier's
prediction results. In our experiments, we select nine attack algorithms and
evaluate their performance under "Showing Many Labels". Eight of the attack
algorithms were adapted from the multi-class environment to the multi-label
environment, while the remaining one was specifically designed for the
multi-label environment. We choose ML-LIW and ML-GCN as target models and train
them on four popular multi-label datasets: VOC2007, VOC2012, NUS-WIDE, and
COCO. We record the success rate of each algorithm when it shows the expected
number of labels in eight different scenarios. Experimental results indicate
that under the "Showing Many Labels", iterative attacks perform significantly
better than one-step attacks. Moreover, it is possible to show all labels in
the dataset.


http://arxiv.org/abs/2409.17977
Cross-Modality Attack Boosted by Gradient-Evolutionary Multiform Optimization. (98%)
Yunpeng Gong; Qingyuan Zeng; Dejun Xu; Zhenzhong Wang; Min Jiang
  In recent years, despite significant advancements in adversarial attack
research, the security challenges in cross-modal scenarios, such as the
transferability of adversarial attacks between infrared, thermal, and RGB
images, have been overlooked. These heterogeneous image modalities collected by
different hardware devices are widely prevalent in practical applications, and
the substantial differences between modalities pose significant challenges to
attack transferability. In this work, we explore a novel cross-modal
adversarial attack strategy, termed multiform attack. We propose a dual-layer
optimization framework based on gradient-evolution, facilitating efficient
perturbation transfer between modalities. In the first layer of optimization,
the framework utilizes image gradients to learn universal perturbations within
each modality and employs evolutionary algorithms to search for shared
perturbations with transferability across different modalities through
secondary optimization. Through extensive testing on multiple heterogeneous
datasets, we demonstrate the superiority and robustness of Multiform Attack
compared to existing techniques. This work not only enhances the
transferability of cross-modal adversarial attacks but also provides a new
perspective for understanding security vulnerabilities in cross-modal systems.


http://arxiv.org/abs/2409.18248
Discovering New Shadow Patterns for Black-Box Attacks on Lane Detection of Autonomous Vehicles. (97%)
Pedram MohajerAnsari; Alkim Domeke; Voor Jan de; Arkajyoti Mitra; Grace Johnson; Amir Salarpour; Habeeb Olufowobi; Mohammad Hamad; Mert D. Pesé
  Ensuring autonomous vehicle (AV) security remains a critical concern. An area
of paramount importance is the study of physical-world adversarial examples
(AEs) aimed at exploiting vulnerabilities in perception systems. However, most
of the prevailing research on AEs has neglected considerations of stealthiness
and legality, resulting in scenarios where human drivers would promptly
intervene or attackers would be swiftly detected and punished. These
limitations hinder the applicability of such examples in real-life settings. In
this paper, we introduce a novel approach to generate AEs using what we term
negative shadows: deceptive patterns of light on the road created by
strategically blocking sunlight, which then cast artificial lane-like patterns.
These shadows are inconspicuous to a driver while deceiving AV perception
systems, particularly those reliant on lane detection algorithms. By
prioritizing the stealthy nature of attacks to minimize driver interventions
and ensuring their legality from an attacker's standpoint, a more plausible
range of scenarios is established. In multiple scenarios, including at low
speeds, our method shows a high safety violation rate. Using a 20-meter
negative shadow, it can direct a vehicle off-road with a 100% violation rate at
speeds over 10 mph. Other attack scenarios, such as causing collisions, can be
performed with at least 30 meters of negative shadow, achieving a 60-100%
success rate. The attack also maintains an average stealthiness of 83.6% as
measured through a human subject experiment, ensuring its efficacy in covert
settings.


http://arxiv.org/abs/2409.17589
Improving Fast Adversarial Training via Self-Knowledge Guidance. (82%)
Chengze Jiang; Junkai Wang; Minjing Dong; Jie Gui; Xinli Shi; Yuan Cao; Yuan Yan Tang; James Tin-Yau Kwok
  Adversarial training has achieved remarkable advancements in defending
against adversarial attacks. Among them, fast adversarial training (FAT) is
gaining attention for its ability to achieve competitive robustness with fewer
computing resources. Existing FAT methods typically employ a uniform strategy
that optimizes all training data equally without considering the influence of
different examples, which leads to an imbalanced optimization. However, this
imbalance remains unexplored in the field of FAT. In this paper, we conduct a
comprehensive study of the imbalance issue in FAT and observe an obvious class
disparity regarding their performances. This disparity could be embodied from a
perspective of alignment between clean and robust accuracy. Based on the
analysis, we mainly attribute the observed misalignment and disparity to the
imbalanced optimization in FAT, which motivates us to optimize different
training data adaptively to enhance robustness. Specifically, we take disparity
and misalignment into consideration. First, we introduce self-knowledge guided
regularization, which assigns differentiated regularization weights to each
class based on its training state, alleviating class disparity. Additionally,
we propose self-knowledge guided label relaxation, which adjusts label
relaxation according to the training accuracy, alleviating the misalignment and
improving robustness. By combining these methods, we formulate the
Self-Knowledge Guided FAT (SKG-FAT), leveraging naturally generated knowledge
during training to enhance the adversarial robustness without compromising
training efficiency. Extensive experiments on four standard datasets
demonstrate that the SKG-FAT improves the robustness and preserves competitive
clean accuracy, outperforming the state-of-the-art methods.


http://arxiv.org/abs/2409.17774
Faithfulness and the Notion of Adversarial Sensitivity in NLP Explanations. (69%)
Supriya Manna; Niladri Sett
  Faithfulness is arguably the most critical metric to assess the reliability
of explainable AI. In NLP, current methods for faithfulness evaluation are
fraught with discrepancies and biases, often failing to capture the true
reasoning of models. We introduce Adversarial Sensitivity as a novel approach
to faithfulness evaluation, focusing on the explainer's response when the model
is under adversarial attack. Our method accounts for the faithfulness of
explainers by capturing sensitivity to adversarial input changes. This work
addresses significant limitations in existing evaluation techniques, and
furthermore, quantifies faithfulness from a crucial yet underexplored paradigm.


http://arxiv.org/abs/2409.17601
CleanerCLIP: Fine-grained Counterfactual Semantic Augmentation for Backdoor Defense in Contrastive Learning. (69%)
Yuan Xun; Siyuan Liang; Xiaojun Jia; Xinwei Liu; Xiaochun Cao
  Pre-trained large models for multimodal contrastive learning, such as CLIP,
have been widely recognized in the industry as highly susceptible to
data-poisoned backdoor attacks. This poses significant risks to downstream
model training. In response to such potential threats, finetuning offers a
simpler and more efficient defense choice compared to retraining large models
with augmented data. In the supervised learning domain, fine-tuning defense
strategies can achieve excellent defense performance. However, in the
unsupervised and semi-supervised domain, we find that when CLIP faces some
complex attack techniques, the existing fine-tuning defense strategy,
CleanCLIP, has some limitations on defense performance. The synonym
substitution of its text-augmentation is insufficient to enhance the text
feature space. To compensate for this weakness, we improve it by proposing a
fine-grained \textbf{T}ext \textbf{A}lignment \textbf{C}leaner (TA-Cleaner) to
cut off feature connections of backdoor triggers. We randomly select a few
samples for positive and negative subtext generation at each epoch of
CleanCLIP, and align the subtexts to the images to strengthen the text
self-supervision. We evaluate the effectiveness of our TA-Cleaner against six
attack algorithms and conduct comprehensive zero-shot classification tests on
ImageNet1K. Our experimental results demonstrate that TA-Cleaner achieves
state-of-the-art defensiveness among finetuning-based defense techniques. Even
when faced with the novel attack technique BadCLIP, our TA-Cleaner outperforms
CleanCLIP by reducing the ASR of Top-1 and Top-10 by 52.02\% and 63.88\%,
respectively.


http://arxiv.org/abs/2409.17874
DarkSAM: Fooling Segment Anything Model to Segment Nothing. (68%)
Ziqi Zhou; Yufei Song; Minghui Li; Shengshan Hu; Xianlong Wang; Leo Yu Zhang; Dezhong Yao; Hai Jin
  Segment Anything Model (SAM) has recently gained much attention for its
outstanding generalization to unseen data and tasks. Despite its promising
prospect, the vulnerabilities of SAM, especially to universal adversarial
perturbation (UAP) have not been thoroughly investigated yet. In this paper, we
propose DarkSAM, the first prompt-free universal attack framework against SAM,
including a semantic decoupling-based spatial attack and a texture
distortion-based frequency attack. We first divide the output of SAM into
foreground and background. Then, we design a shadow target strategy to obtain
the semantic blueprint of the image as the attack target. DarkSAM is dedicated
to fooling SAM by extracting and destroying crucial object features from images
in both spatial and frequency domains. In the spatial domain, we disrupt the
semantics of both the foreground and background in the image to confuse SAM. In
the frequency domain, we further enhance the attack effectiveness by distorting
the high-frequency components (i.e., texture information) of the image.
Consequently, with a single UAP, DarkSAM renders SAM incapable of segmenting
objects across diverse images with varying prompts. Experimental results on
four datasets for SAM and its two variant models demonstrate the powerful
attack capability and transferability of DarkSAM.


http://arxiv.org/abs/2409.17941
Perturb, Attend, Detect and Localize (PADL): Robust Proactive Image Defense. (56%)
Filippo Bartolucci; Iacopo Masi; Giuseppe Lisanti
  Image manipulation detection and localization have received considerable
attention from the research community given the blooming of Generative Models
(GMs). Detection methods that follow a passive approach may overfit to specific
GMs, limiting their application in real-world scenarios, due to the growing
diversity of generative models. Recently, approaches based on a proactive
framework have shown the possibility of dealing with this limitation. However,
these methods suffer from two main limitations, which raises concerns about
potential vulnerabilities: i) the manipulation detector is not robust to noise
and hence can be easily fooled; ii) the fact that they rely on fixed
perturbations for image protection offers a predictable exploit for malicious
attackers, enabling them to reverse-engineer and evade detection. To overcome
this issue we propose PADL, a new solution able to generate image-specific
perturbations using a symmetric scheme of encoding and decoding based on
cross-attention, which drastically reduces the possibility of reverse
engineering, even when evaluated with adaptive attack [31]. Additionally, PADL
is able to pinpoint manipulated areas, facilitating the identification of
specific regions that have undergone alterations, and has more generalization
power than prior art on held-out generative models. Indeed, although being
trained only on an attribute manipulation GAN model [15], our method
generalizes to a range of unseen models with diverse architectural designs,
such as StarGANv2, BlendGAN, DiffAE, StableDiffusion and StableDiffusionXL.
Additionally, we introduce a novel evaluation protocol, which offers a fair
evaluation of localisation performance in function of detection accuracy and
better captures real-world scenarios.


http://arxiv.org/abs/2409.18244
Development of an Edge Resilient ML Ensemble to Tolerate ICS Adversarial Attacks. (54%)
Likai Yao; Qinxuan Shi; Zhanglong Yang; Sicong Shao; Salim Hariri
  Deploying machine learning (ML) in dynamic data-driven applications systems
(DDDAS) can improve the security of industrial control systems (ICS). However,
ML-based DDDAS are vulnerable to adversarial attacks because adversaries can
alter the input data slightly so that the ML models predict a different result.
In this paper, our goal is to build a resilient edge machine learning (reML)
architecture that is designed to withstand adversarial attacks by performing
Data Air Gap Transformation (DAGT) to anonymize data feature spaces using deep
neural networks and randomize the ML models used for predictions. The reML is
based on the Resilient DDDAS paradigm, Moving Target Defense (MTD) theory, and
TinyML and is applied to combat adversarial attacks on ICS. Furthermore, the
proposed approach is power-efficient and privacy-preserving and, therefore, can
be deployed on power-constrained devices to enhance ICS security. This approach
enables resilient ML inference at the edge by shifting the computation from the
computing-intensive platforms to the resource-constrained edge devices. The
incorporation of TinyML with TensorFlow Lite ensures efficient resource
utilization and, consequently, makes reML suitable for deployment in various
industrial control environments. Furthermore, the dynamic nature of reML,
facilitated by the resilient DDDAS development environment, allows for
continuous adaptation and improvement in response to emerging threats. Lastly,
we evaluate our approach on an ICS dataset and demonstrate that reML provides a
viable and effective solution for resilient ML inference at the edge devices.


http://arxiv.org/abs/2409.17946
Backdoor Attacks for LLMs with Weak-To-Strong Knowledge Distillation. (15%)
Shuai Zhao; Leilei Gan; Zhongliang Guo; Xiaobao Wu; Luwei Xiao; Xiaoyu Xu; Cong-Duy Nguyen; Luu Anh Tuan
  Despite being widely applied due to their exceptional capabilities, Large
Language Models (LLMs) have been proven to be vulnerable to backdoor attacks.
These attacks introduce targeted vulnerabilities into LLMs by poisoning
training samples and full-parameter fine-tuning. However, this kind of backdoor
attack is limited since they require significant computational resources,
especially as the size of LLMs increases. Besides, parameter-efficient
fine-tuning (PEFT) offers an alternative but the restricted parameter updating
may impede the alignment of triggers with target labels. In this study, we
first verify that backdoor attacks with PEFT may encounter challenges in
achieving feasible performance. To address these issues and improve the
effectiveness of backdoor attacks with PEFT, we propose a novel backdoor attack
algorithm from weak to strong based on feature alignment-enhanced knowledge
distillation (W2SAttack). Specifically, we poison small-scale language models
through full-parameter fine-tuning to serve as the teacher model. The teacher
model then covertly transfers the backdoor to the large-scale student model
through feature alignment-enhanced knowledge distillation, which employs PEFT.
Theoretical analysis reveals that W2SAttack has the potential to augment the
effectiveness of backdoor attacks. We demonstrate the superior performance of
W2SAttack on classification tasks across four language models, four backdoor
attack algorithms, and two different architectures of teacher models.
Experimental results indicate success rates close to 100% for backdoor attacks
targeting PEFT.


http://arxiv.org/abs/2409.18169
Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey. (15%)
Tiansheng Huang; Sihao Hu; Fatih Ilhan; Selim Furkan Tekin; Ling Liu
  Recent research demonstrates that the nascent fine-tuning-as-a-service
business model exposes serious safety concerns -- fine-tuning over a few
harmful data uploaded by the users can compromise the safety alignment of the
model. The attack, known as harmful fine-tuning, has raised a broad research
interest among the community. However, as the attack is still new, \textbf{we
observe from our miserable submission experience that there are general
misunderstandings within the research community.} We in this paper aim to clear
some common concerns for the attack setting, and formally establish the
research problem. Specifically, we first present the threat model of the
problem, and introduce the harmful fine-tuning attack and its variants. Then we
systematically survey the existing literature on attacks/defenses/mechanical
analysis of the problem. Finally, we outline future research directions that
might contribute to the development of the field. Additionally, we present a
list of questions of interest, which might be useful to refer to when reviewers
in the peer review process question the realism of the
experiment/attack/defense setting. A curated list of relevant papers is
maintained and made accessible at:
\url{https://github.com/git-disl/awesome_LLM-harmful-fine-tuning-papers.}


http://arxiv.org/abs/2409.17682
Dark Miner: Defend against unsafe generation for text-to-image diffusion models. (5%)
Zheling Meng; Bo Peng; Xiaochuan Jin; Yue Jiang; Jing Dong; Wei Wang; Tieniu Tan
  Text-to-image diffusion models have been demonstrated with unsafe generation
due to unfiltered large-scale training data, such as violent, sexual, and
shocking images, necessitating the erasure of unsafe concepts. Most existing
methods focus on modifying the generation probabilities conditioned on the
texts containing unsafe descriptions. However, they fail to guarantee safe
generation for unseen texts in the training phase, especially for the prompts
from adversarial attacks. In this paper, we re-analyze the erasure task and
point out that existing methods cannot guarantee the minimization of the total
probabilities of unsafe generation. To tackle this problem, we propose Dark
Miner. It entails a recurring three-stage process that comprises mining,
verifying, and circumventing. It greedily mines embeddings with maximum
generation probabilities of unsafe concepts and reduces unsafe generation more
effectively. In the experiments, we evaluate its performance on two
inappropriate concepts, two objects, and two styles. Compared with 6 previous
state-of-the-art methods, our method achieves better erasure and defense
results in most cases, especially under 4 state-of-the-art attacks, while
preserving the model's native generation capability. Our code will be available
on GitHub.


http://arxiv.org/abs/2409.18025
An Adversarial Perspective on Machine Unlearning for AI Safety. (2%)
Jakub Łucki; Boyi Wei; Yangsibo Huang; Peter Henderson; Florian Tramèr; Javier Rando
  Large language models are finetuned to refuse questions about hazardous
knowledge, but these protections can often be bypassed. Unlearning methods aim
at completely removing hazardous capabilities from models and make them
inaccessible to adversaries. This work challenges the fundamental differences
between unlearning and traditional safety post-training from an adversarial
perspective. We demonstrate that existing jailbreak methods, previously
reported as ineffective against unlearning, can be successful when applied
carefully. Furthermore, we develop a variety of adaptive methods that recover
most supposedly unlearned capabilities. For instance, we show that finetuning
on 10 unrelated examples or removing specific directions in the activation
space can recover most hazardous capabilities for models edited with RMU, a
state-of-the-art unlearning method. Our findings challenge the robustness of
current unlearning approaches and question their advantages over safety
training.


http://arxiv.org/abs/2409.18219
Revolutionizing Payload Inspection: A Self-Supervised Journey to Precision with Few Shots. (2%)
Kyle Stein; Arash Mahyari; Guillermo III Francia; Eman El-Sheikh
  As networks continue to expand and become more interconnected, the need for
novel malware detection methods becomes more pronounced. Traditional security
measures are increasingly inadequate against the sophistication of modern cyber
attacks. Deep Packet Inspection (DPI) has been pivotal in enhancing network
security, offering an in-depth analysis of network traffic that surpasses
conventional monitoring techniques. DPI not only examines the metadata of
network packets, but also dives into the actual content being carried within
the packet payloads, providing a comprehensive view of the data flowing through
networks. The integration of advanced deep learning techniques with DPI has
introduced modern methodologies into malware detection. However, the challenge
with the state-of-the-art supervised learning approaches is that they prevent
the generalization to unseen attacks embedded in the payloads, prohibiting them
from accurately detecting new attacks and transferring knowledge learned from
previous attacks to the new attacks with small labeled sample sizes. This paper
leverages the recent advancements in self-supervised learning and few-shot
learning. Our proposed self-supervised approach trains a transformer to learn
the embedding of the payloads from a vast amount of unlabeled datasets by
masking portions of payloads, leading to a learnt representation that well
generalizes to various downstream tasks. Once the representation is extracted
from payloads, they are used to train a malware detection algorithm. The
representation obtained from the transformer is then used to adapt the malware
detector to novel types of attacks using few-shot learning approaches. Our
experimental results across several datasets show the great success and
generalization of the proposed approach to novel scenarios.


http://arxiv.org/abs/2409.17476
Improving the Shortest Plank: Vulnerability-Aware Adversarial Training for Robust Recommender System. (93%)
Kaike Zhang; Qi Cao; Yunfan Wu; Fei Sun; Huawei Shen; Xueqi Cheng
  Recommender systems play a pivotal role in mitigating information overload in
various fields. Nonetheless, the inherent openness of these systems introduces
vulnerabilities, allowing attackers to insert fake users into the system's
training data to skew the exposure of certain items, known as poisoning
attacks. Adversarial training has emerged as a notable defense mechanism
against such poisoning attacks within recommender systems. Existing adversarial
training methods apply perturbations of the same magnitude across all users to
enhance system robustness against attacks. Yet, in reality, we find that
attacks often affect only a subset of users who are vulnerable. These
perturbations of indiscriminate magnitude make it difficult to balance
effective protection for vulnerable users without degrading recommendation
quality for those who are not affected. To address this issue, our research
delves into understanding user vulnerability. Considering that poisoning
attacks pollute the training data, we note that the higher degree to which a
recommender system fits users' training data correlates with an increased
likelihood of users incorporating attack information, indicating their
vulnerability. Leveraging these insights, we introduce the Vulnerability-aware
Adversarial Training (VAT), designed to defend against poisoning attacks in
recommender systems. VAT employs a novel vulnerability-aware function to
estimate users' vulnerability based on the degree to which the system fits
them. Guided by this estimation, VAT applies perturbations of adaptive
magnitude to each user, not only reducing the success ratio of attacks but also
preserving, and potentially enhancing, the quality of recommendations.
Comprehensive experiments confirm VAT's superior defensive capabilities across
different recommendation models and against various types of attacks.


http://arxiv.org/abs/2409.17311
A Hybrid Quantum-Classical AI-Based Detection Strategy for Generative Adversarial Network-Based Deepfake Attacks on an Autonomous Vehicle Traffic Sign Classification System. (82%)
M Sabbir Salek; Shaozhi Li; Mashrur Chowdhury
  The perception module in autonomous vehicles (AVs) relies heavily on deep
learning-based models to detect and identify various objects in their
surrounding environment. An AV traffic sign classification system is integral
to this module, which helps AVs recognize roadway traffic signs. However,
adversarial attacks, in which an attacker modifies or alters the image captured
for traffic sign recognition, could lead an AV to misrecognize the traffic
signs and cause hazardous consequences. Deepfake presents itself as a promising
technology to be used for such adversarial attacks, in which a deepfake traffic
sign would replace a real-world traffic sign image before the image is fed to
the AV traffic sign classification system. In this study, the authors present
how a generative adversarial network-based deepfake attack can be crafted to
fool the AV traffic sign classification systems. The authors developed a
deepfake traffic sign image detection strategy leveraging hybrid
quantum-classical neural networks (NNs). This hybrid approach utilizes
amplitude encoding to represent the features of an input traffic sign image
using quantum states, which substantially reduces the memory requirement
compared to its classical counterparts. The authors evaluated this hybrid
deepfake detection approach along with several baseline classical convolutional
NNs on real-world and deepfake traffic sign images. The results indicate that
the hybrid quantum-classical NNs for deepfake detection could achieve similar
or higher performance than the baseline classical convolutional NNs in most
cases while requiring less than one-third of the memory required by the
shallowest classical convolutional NN considered in this study.


http://arxiv.org/abs/2409.17458
RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreaking. (75%)
Yifan Jiang; Kriti Aggarwal; Tanmay Laud; Kashif Munir; Jay Pujara; Subhabrata Mukherjee
  The rapid progress of Large Language Models (LLMs) has opened up new
opportunities across various domains and applications; yet it also presents
challenges related to potential misuse. To mitigate such risks, red teaming has
been employed as a proactive security measure to probe language models for
harmful outputs via jailbreak attacks. However, current jailbreak attack
approaches are single-turn with explicit malicious queries that do not fully
capture the complexity of real-world interactions. In reality, users can engage
in multi-turn interactions with LLM-based chat assistants, allowing them to
conceal their true intentions in a more covert manner. To bridge this gap, we,
first, propose a new jailbreak approach, RED QUEEN ATTACK. This method
constructs a multi-turn scenario, concealing the malicious intent under the
guise of preventing harm. We craft 40 scenarios that vary in turns and select
14 harmful categories to generate 56k multi-turn attack data points. We conduct
comprehensive experiments on the RED QUEEN ATTACK with four representative LLM
families of different sizes. Our experiments reveal that all LLMs are
vulnerable to RED QUEEN ATTACK, reaching 87.62% attack success rate on GPT-4o
and 75.4% on Llama3-70B. Further analysis reveals that larger models are more
susceptible to the RED QUEEN ATTACK, with multi-turn structures and concealment
strategies contributing to its success. To prioritize safety, we introduce a
straightforward mitigation strategy called RED QUEEN GUARD, which aligns LLMs
to effectively counter adversarial attacks. This approach reduces the attack
success rate to below 1% while maintaining the model's performance across
standard benchmarks. Full implementation and dataset are publicly accessible at
https://github.com/kriti-hippo/red_queen.


http://arxiv.org/abs/2409.17403
Transient Adversarial 3D Projection Attacks on Object Detection in Autonomous Driving. (67%)
Ce Zhou; Qiben Yan; Sijia Liu
  Object detection is a crucial task in autonomous driving. While existing
research has proposed various attacks on object detection, such as those using
adversarial patches or stickers, the exploration of projection attacks on 3D
surfaces remains largely unexplored. Compared to adversarial patches or
stickers, which have fixed adversarial patterns, projection attacks allow for
transient modifications to these patterns, enabling a more flexible attack. In
this paper, we introduce an adversarial 3D projection attack specifically
targeting object detection in autonomous driving scenarios. We frame the attack
formulation as an optimization problem, utilizing a combination of color
mapping and geometric transformation models. Our results demonstrate the
effectiveness of the proposed attack in deceiving YOLOv3 and Mask R-CNN in
physical settings. Evaluations conducted in an indoor environment show an
attack success rate of up to 100% under low ambient light conditions,
highlighting the potential damage of our attack in real-world driving
scenarios.


http://arxiv.org/abs/2409.16639
Examining the Rat in the Tunnel: Interpretable Multi-Label Classification of Tor-based Malware. (45%)
Ishan Karunanayake; Mashael AlSabah; Nadeem Ahmed; Sanjay Jha
  Despite being the most popular privacy-enhancing network, Tor is increasingly
adopted by cybercriminals to obfuscate malicious traffic, hindering the
identification of malware-related communications between compromised devices
and Command and Control (C&C) servers. This malicious traffic can induce
congestion and reduce Tor's performance, while encouraging network
administrators to block Tor traffic. Recent research, however, demonstrates the
potential for accurately classifying captured Tor traffic as malicious or
benign. While existing efforts have addressed malware class identification,
their performance remains limited, with micro-average precision and recall
values around 70%. Accurately classifying specific malware classes is crucial
for effective attack prevention and mitigation. Furthermore, understanding the
unique patterns and attack vectors employed by different malware classes helps
the development of robust and adaptable defence mechanisms.
  We utilise a multi-label classification technique based on Message-Passing
Neural Networks, demonstrating its superiority over previous approaches such as
Binary Relevance, Classifier Chains, and Label Powerset, by achieving
micro-average precision (MAP) and recall (MAR) exceeding 90%. Compared to
previous work, we significantly improve performance by 19.98%, 10.15%, and
59.21% in MAP, MAR, and Hamming Loss, respectively. Next, we employ Explainable
Artificial Intelligence (XAI) techniques to interpret the decision-making
process within these models. Finally, we assess the robustness of all
techniques by crafting adversarial perturbations capable of manipulating
classifier predictions and generating false positives and negatives.


http://arxiv.org/abs/2409.16673
SWE2: SubWord Enriched and Significant Word Emphasized Framework for Hate Speech Detection. (38%)
Guanyi Mou; Pengyi Ye; Kyumin Lee
  Hate speech detection on online social networks has become one of the
emerging hot topics in recent years. With the broad spread and fast propagation
speed across online social networks, hate speech makes significant impacts on
society by increasing prejudice and hurting people. Therefore, there are
aroused attention and concern from both industry and academia. In this paper,
we address the hate speech problem and propose a novel hate speech detection
framework called SWE2, which only relies on the content of messages and
automatically identifies hate speech. In particular, our framework exploits
both word-level semantic information and sub-word knowledge. It is intuitively
persuasive and also practically performs well under a situation with/without
character-level adversarial attack. Experimental results show that our proposed
model achieves 0.975 accuracy and 0.953 macro F1, outperforming 7
state-of-the-art baselines under no adversarial attack. Our model robustly and
significantly performed well under extreme adversarial attack (manipulation of
50% messages), achieving 0.967 accuracy and 0.934 macro F1.


http://arxiv.org/abs/2409.17279
SHEATH: Defending Horizontal Collaboration for Distributed CNNs against Adversarial Noise. (22%)
Muneeba Asif; Mohammad Kumail Kazmi; Mohammad Ashiqur Rahman; Syed Rafay Hasan; Soamar Homsi
  As edge computing and the Internet of Things (IoT) expand, horizontal
collaboration (HC) emerges as a distributed data processing solution for
resource-constrained devices. In particular, a convolutional neural network
(CNN) model can be deployed on multiple IoT devices, allowing distributed
inference execution for image recognition while ensuring model and data
privacy. Yet, this distributed architecture remains vulnerable to adversaries
who want to make subtle alterations that impact the model, even if they lack
access to the entire model. Such vulnerabilities can have severe implications
for various sectors, including healthcare, military, and autonomous systems.
However, security solutions for these vulnerabilities have not been explored.
This paper presents a novel framework for Secure Horizontal Edge with
Adversarial Threat Handling (SHEATH) to detect adversarial noise and eliminate
its effect on CNN inference by recovering the original feature maps.
Specifically, SHEATH aims to address vulnerabilities without requiring complete
knowledge of the CNN model in HC edge architectures based on sequential
partitioning. It ensures data and model integrity, offering security against
adversarial attacks in diverse HC environments. Our evaluations demonstrate
SHEATH's adaptability and effectiveness across diverse CNN configurations.


http://arxiv.org/abs/2409.16618
Claim-Guided Textual Backdoor Attack for Practical Applications. (10%)
Minkyoo Song; Hanna Kim; Jaehan Kim; Youngjin Jin; Seungwon Shin
  Recent advances in natural language processing and the increased use of large
language models have exposed new security vulnerabilities, such as backdoor
attacks. Previous backdoor attacks require input manipulation after model
distribution to activate the backdoor, posing limitations in real-world
applicability. Addressing this gap, we introduce a novel Claim-Guided Backdoor
Attack (CGBA), which eliminates the need for such manipulations by utilizing
inherent textual claims as triggers. CGBA leverages claim extraction,
clustering, and targeted training to trick models to misbehave on targeted
claims without affecting their performance on clean data. CGBA demonstrates its
effectiveness and stealthiness across various datasets and models,
significantly enhancing the feasibility of practical backdoor attacks. Our code
and data will be available at https://github.com/PaperCGBA/CGBA.


http://arxiv.org/abs/2409.17443
Cat-and-Mouse Satellite Dynamics: Divergent Adversarial Reinforcement Learning for Contested Multi-Agent Space Operations. (1%)
Cameron Mehlman; Joseph Abramov; Gregory Falco
  As space becomes increasingly crowded and contested, robust autonomous
capabilities for multi-agent environments are gaining critical importance.
Current autonomous systems in space primarily rely on optimization-based path
planning or long-range orbital maneuvers, which have not yet proven effective
in adversarial scenarios where one satellite is actively pursuing another. We
introduce Divergent Adversarial Reinforcement Learning (DARL), a two-stage
Multi-Agent Reinforcement Learning (MARL) approach designed to train autonomous
evasion strategies for satellites engaged with multiple adversarial spacecraft.
Our method enhances exploration during training by promoting diverse
adversarial strategies, leading to more robust and adaptable evader models. We
validate DARL through a cat-and-mouse satellite scenario, modeled as a
partially observable multi-agent capture the flag game where two adversarial
`cat' spacecraft pursue a single `mouse' evader. DARL's performance is compared
against several benchmarks, including an optimization-based satellite path
planner, demonstrating its ability to produce highly robust models for
adversarial multi-agent space environments.


http://arxiv.org/abs/2409.15968
Adversarial Backdoor Defense in CLIP. (99%)
Junhao Kuang; Siyuan Liang; Jiawei Liang; Kuanrong Liu; Xiaochun Cao
  Multimodal contrastive pretraining, exemplified by models like CLIP, has been
found to be vulnerable to backdoor attacks. While current backdoor defense
methods primarily employ conventional data augmentation to create augmented
samples aimed at feature alignment, these methods fail to capture the distinct
features of backdoor samples, resulting in suboptimal defense performance.
Observations reveal that adversarial examples and backdoor samples exhibit
similarities in the feature space within the compromised models. Building on
this insight, we propose Adversarial Backdoor Defense (ABD), a novel data
augmentation strategy that aligns features with meticulously crafted
adversarial examples. This approach effectively disrupts the backdoor
association. Our experiments demonstrate that ABD provides robust defense
against both traditional uni-modal and multimodal backdoor attacks targeting
CLIP. Compared to the current state-of-the-art defense method, CleanCLIP, ABD
reduces the attack success rate by 8.66% for BadNet, 10.52% for Blended, and
53.64% for BadCLIP, while maintaining a minimal average decrease of just 1.73%
in clean accuracy.


http://arxiv.org/abs/2409.16399
Revisiting Acoustic Features for Robust ASR. (84%)
Muhammad A. Shah; Bhiksha Raj
  Automatic Speech Recognition (ASR) systems must be robust to the myriad types
of noises present in real-world environments including environmental noise,
room impulse response, special effects as well as attacks by malicious actors
(adversarial attacks). Recent works seek to improve accuracy and robustness by
developing novel Deep Neural Networks (DNNs) and curating diverse training
datasets for them, while using relatively simple acoustic features. While this
approach improves robustness to the types of noise present in the training
data, it confers limited robustness against unseen noises and negligible
robustness to adversarial attacks. In this paper, we revisit the approach of
earlier works that developed acoustic features inspired by biological auditory
perception that could be used to perform accurate and robust ASR. In contrast,
Specifically, we evaluate the ASR accuracy and robustness of several
biologically inspired acoustic features. In addition to several features from
prior works, such as gammatone filterbank features (GammSpec), we also propose
two new acoustic features called frequency masked spectrogram (FreqMask) and
difference of gammatones spectrogram (DoGSpec) to simulate the
neuro-psychological phenomena of frequency masking and lateral suppression.
Experiments on diverse models and datasets show that (1) DoGSpec achieves
significantly better robustness than the highly popular log mel spectrogram
(LogMelSpec) with minimal accuracy degradation, and (2) GammSpec achieves
better accuracy and robustness to non-adversarial noises from the Speech Robust
Bench benchmark, but it is outperformed by DoGSpec against adversarial attacks.


http://arxiv.org/abs/2409.16056
Adversarial Watermarking for Face Recognition. (80%)
Yuguang Yao; Anil Jain; Sijia Liu
  Watermarking is an essential technique for embedding an identifier (i.e.,
watermark message) within digital images to assert ownership and monitor
unauthorized alterations. In face recognition systems, watermarking plays a
pivotal role in ensuring data integrity and security. However, an adversary
could potentially interfere with the watermarking process, significantly
impairing recognition performance. We explore the interaction between
watermarking and adversarial attacks on face recognition models. Our findings
reveal that while watermarking or input-level perturbation alone may have a
negligible effect on recognition accuracy, the combined effect of watermarking
and perturbation can result in an adversarial watermarking attack,
significantly degrading recognition performance. Specifically, we introduce a
novel threat model, the adversarial watermarking attack, which remains stealthy
in the absence of watermarking, allowing images to be correctly recognized
initially. However, once watermarking is applied, the attack is activated,
causing recognition failures. Our study reveals a previously unrecognized
vulnerability: adversarial perturbations can exploit the watermark message to
evade face recognition systems. Evaluated on the CASIA-WebFace dataset, our
proposed adversarial watermarking attack reduces face matching accuracy by
67.2% with an $\ell_\infty$ norm-measured perturbation strength of ${2}/{255}$
and by 95.9% with a strength of ${4}/{255}$.


http://arxiv.org/abs/2409.16491
Proactive Schemes: A Survey of Adversarial Attacks for Social Good. (54%)
Vishal Asnani; Xi Yin; Xiaoming Liu
  Adversarial attacks in computer vision exploit the vulnerabilities of machine
learning models by introducing subtle perturbations to input data, often
leading to incorrect predictions or classifications. These attacks have evolved
in sophistication with the advent of deep learning, presenting significant
challenges in critical applications, which can be harmful for society. However,
there is also a rich line of research from a transformative perspective that
leverages adversarial techniques for social good. Specifically, we examine the
rise of proactive schemes-methods that encrypt input data using additional
signals termed templates, to enhance the performance of deep learning models.
By embedding these imperceptible templates into digital media, proactive
schemes are applied across various applications, from simple image enhancements
to complicated deep learning frameworks to aid performance, as compared to the
passive schemes, which don't change the input data distribution for their
framework. The survey delves into the methodologies behind these proactive
schemes, the encryption and learning processes, and their application to modern
computer vision and natural language processing applications. Additionally, it
discusses the challenges, potential vulnerabilities, and future directions for
proactive schemes, ultimately highlighting their potential to foster the
responsible and secure advancement of deep learning technologies.


http://arxiv.org/abs/2409.15868
Privacy Evaluation Benchmarks for NLP Models. (45%)
Wei Huang; Yinggui Wang; Cen Chen
  By inducing privacy attacks on NLP models, attackers can obtain sensitive
information such as training data and model parameters, etc. Although
researchers have studied, in-depth, several kinds of attacks in NLP models,
they are non-systematic analyses. It lacks a comprehensive understanding of the
impact caused by the attacks. For example, we must consider which scenarios can
apply to which attacks, what the common factors are that affect the performance
of different attacks, the nature of the relationships between different
attacks, and the influence of various datasets and models on the effectiveness
of the attacks, etc. Therefore, we need a benchmark to holistically assess the
privacy risks faced by NLP models. In this paper, we present a privacy attack
and defense evaluation benchmark in the field of NLP, which includes the
conventional/small models and large language models (LLMs). This benchmark
supports a variety of models, datasets, and protocols, along with standardized
modules for comprehensive evaluation of attacks and defense strategies. Based
on the above framework, we present a study on the association between auxiliary
data from different domains and the strength of privacy attacks. And we provide
an improved attack method in this scenario with the help of Knowledge
Distillation (KD). Furthermore, we propose a chained framework for privacy
attacks. Allowing a practitioner to chain multiple attacks to achieve a
higher-level attack objective. Based on this, we provide some defense and
enhanced attack strategies. The code for reproducing the results can be found
at https://github.com/user2311717757/nlp_doctor.


http://arxiv.org/abs/2409.16057
Towards Robust Object Detection: Identifying and Removing Backdoors via Module Inconsistency Analysis. (33%)
Xianda Zhang; Siyuan Liang
  Object detection models, widely used in security-critical applications, are
vulnerable to backdoor attacks that cause targeted misclassifications when
triggered by specific patterns. Existing backdoor defense techniques, primarily
designed for simpler models like image classifiers, often fail to effectively
detect and remove backdoors in object detectors. We propose a backdoor defense
framework tailored to object detection models, based on the observation that
backdoor attacks cause significant inconsistencies between local modules'
behaviors, such as the Region Proposal Network (RPN) and classification head.
By quantifying and analyzing these inconsistencies, we develop an algorithm to
detect backdoors. We find that the inconsistent module is usually the main
source of backdoor behavior, leading to a removal method that localizes the
affected module, resets its parameters, and fine-tunes the model on a small
clean dataset. Extensive experiments with state-of-the-art two-stage object
detectors show our method achieves a 90% improvement in backdoor removal rate
over fine-tuning baselines, while limiting clean data accuracy loss to less
than 4%. To the best of our knowledge, this work presents the first approach
that addresses both the detection and removal of backdoors in two-stage object
detection models, advancing the field of securing these complex systems against
backdoor attacks.


http://arxiv.org/abs/2409.15990
PACE: Poisoning Attacks on Learned Cardinality Estimation. (4%)
Jintao Tsinghua University Zhang; Chao Tsinghua University Zhang; Guoliang Tsinghua University Li; Chengliang Beijing Institute of Technology Chai
  Cardinality estimation (CE) plays a crucial role in database optimizer. We
have witnessed the emergence of numerous learned CE models recently which can
outperform traditional methods such as histograms and samplings. However,
learned models also bring many security risks. For example, a query-driven
learned CE model learns a query-to-cardinality mapping based on the historical
workload. Such a learned model could be attacked by poisoning queries, which
are crafted by malicious attackers and woven into the historical workload,
leading to performance degradation of CE. In this paper, we explore the
potential security risks in learned CE and study a new problem of poisoning
attacks on learned CE in a black-box setting. Experiments show that PACE
reduces the accuracy of the learned CE models by 178 times, leading to a 10
times decrease in the end-to-end performance of the target database.


http://arxiv.org/abs/2409.14940
Improving Adversarial Robustness for 3D Point Cloud Recognition at Test-Time through Purified Self-Training. (96%)
Jinpeng Lin; Xulei Yang; Tianrui Li; Xun Xu
  Recognizing 3D point cloud plays a pivotal role in many real-world
applications. However, deploying 3D point cloud deep learning model is
vulnerable to adversarial attacks. Despite many efforts into developing robust
model by adversarial training, they may become less effective against emerging
attacks. This limitation motivates the development of adversarial purification
which employs generative model to mitigate the impact of adversarial attacks.
In this work, we highlight the remaining challenges from two perspectives.
First, the purification based method requires retraining the classifier on
purified samples which introduces additional computation overhead. Moreover, in
a more realistic scenario, testing samples arrives in a streaming fashion and
adversarial samples are not isolated from clean samples. These challenges
motivates us to explore dynamically update model upon observing testing
samples. We proposed a test-time purified self-training strategy to achieve
this objective. Adaptive thresholding and feature distribution alignment are
introduced to improve the robustness of self-training. Extensive results on
different adversarial attacks suggest the proposed method is complementary to
purification based method in handling continually changing adversarial attacks
on the testing data stream.


http://arxiv.org/abs/2409.15190
Interpretability-Guided Test-Time Adversarial Defense. (87%)
Akshay Kulkarni; Tsui-Wei Weng
  We propose a novel and low-cost test-time adversarial defense by devising
interpretability-guided neuron importance ranking methods to identify neurons
important to the output classes. Our method is a training-free approach that
can significantly improve the robustness-accuracy tradeoff while incurring
minimal computational overhead. While being among the most efficient test-time
defenses (4x faster), our method is also robust to a wide range of black-box,
white-box, and adaptive attacks that break previous test-time defenses. We
demonstrate the efficacy of our method for CIFAR10, CIFAR100, and ImageNet-1k
on the standard RobustBench benchmark (with average gains of 2.6%, 4.9%, and
2.8% respectively). We also show improvements (average 1.5%) over the
state-of-the-art test-time defenses even under strong adaptive attacks.


http://arxiv.org/abs/2409.14866
PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs. (87%)
Xueluan Gong; Mingzhe Li; Yilin Zhang; Fengyuan Ran; Chen Chen; Yanjiao Chen; Qian Wang; Kwok-Yan Lam
  Large Language Models (LLMs) have excelled in various tasks but are still
vulnerable to jailbreaking attacks, where attackers create jailbreak prompts to
mislead the model to produce harmful or offensive content. Current jailbreak
methods either rely heavily on manually crafted templates, which pose
challenges in scalability and adaptability, or struggle to generate
semantically coherent prompts, making them easy to detect. Additionally, most
existing approaches involve lengthy prompts, leading to higher query costs.In
this paper, to remedy these challenges, we introduce a novel jailbreaking
attack framework called PAPILLON, which is an automated, black-box jailbreaking
attack framework that adapts the black-box fuzz testing approach with a series
of customized designs. Instead of relying on manually crafted
templates,PAPILLON starts with an empty seed pool, removing the need to search
for any related jailbreaking templates. We also develop three novel
question-dependent mutation strategies using an LLM helper to generate prompts
that maintain semantic coherence while significantly reducing their length.
Additionally, we implement a two-level judge module to accurately detect
genuine successful jailbreaks. We evaluated PAPILLON on 7 representative LLMs
and compared it with 5 state-of-the-art jailbreaking attack strategies. For
proprietary LLM APIs, such as GPT-3.5 turbo, GPT-4, and Gemini-Pro, PAPILLONs
achieves attack success rates of over 90%, 80%, and 74%, respectively,
exceeding existing baselines by more than 60\%. Additionally, PAPILLON can
maintain high semantic coherence while significantly reducing the length of
jailbreak prompts. When targeting GPT-4, PAPILLON can achieve over 78% attack
success rate even with 100 tokens. Moreover, PAPILLON demonstrates
transferability and is robust to state-of-the-art defenses.


http://arxiv.org/abs/2409.15670
Data Poisoning-based Backdoor Attack Framework against Supervised Learning Rules of Spiking Neural Networks. (68%)
Lingxin Jin; Meiyu Lin; Wei Jiang; Jinyu Zhan
  Spiking Neural Networks (SNNs), the third generation neural networks, are
known for their low energy consumption and high robustness. SNNs are developing
rapidly and can compete with Artificial Neural Networks (ANNs) in many fields.
To ensure that the widespread use of SNNs does not cause serious security
incidents, much research has been conducted to explore the robustness of SNNs
under adversarial sample attacks. However, many other unassessed security
threats exist, such as highly stealthy backdoor attacks. Therefore, to fill the
research gap in this and further explore the security vulnerabilities of SNNs,
this paper explores the robustness performance of SNNs trained by supervised
learning rules under backdoor attacks. Specifically, the work herein includes:
i) We propose a generic backdoor attack framework that can be launched against
the training process of existing supervised learning rules and covers all
learnable dataset types of SNNs. ii) We analyze the robustness differences
between different learning rules and between SNN and ANN, which suggests that
SNN no longer has inherent robustness under backdoor attacks. iii) We reveal
the vulnerability of conversion-dependent learning rules caused by backdoor
migration and further analyze the migration ability during the conversion
process, finding that the backdoor migration rate can even exceed 99%. iv)
Finally, we discuss potential countermeasures against this kind of backdoor
attack and its technical challenges and point out several promising research
directions.


http://arxiv.org/abs/2409.15398
Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI. (47%)
Ambrish Rawat; Stefan Schoepf; Giulio Zizzo; Giandomenico Cornacchia; Muhammad Zaid Hameed; Kieran Fraser; Erik Miehling; Beat Buesser; Elizabeth M. Daly; Mark Purcell; Prasanna Sattigeri; Pin-Yu Chen; Kush R. Varshney
  As generative AI, particularly large language models (LLMs), become
increasingly integrated into production applications, new attack surfaces and
vulnerabilities emerge and put a focus on adversarial threats in natural
language and multi-modal systems. Red-teaming has gained importance in
proactively identifying weaknesses in these systems, while blue-teaming works
to protect against such adversarial attacks. Despite growing academic interest
in adversarial risks for generative AI, there is limited guidance tailored for
practitioners to assess and mitigate these challenges in real-world
environments. To address this, our contributions include: (1) a practical
examination of red- and blue-teaming strategies for securing generative AI, (2)
identification of key challenges and open questions in defense development and
evaluation, and (3) the Attack Atlas, an intuitive framework that brings a
practical approach to analyzing single-turn input attacks, placing it at the
forefront for practitioners. This work aims to bridge the gap between academic
insights and practical security measures for the protection of generative AI
systems.


http://arxiv.org/abs/2409.14729
PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs. (33%)
Jiahao Yu; Yangguang Shao; Hanwen Miao; Junzheng Shi
  Large Language Models (LLMs) have gained widespread use in various
applications due to their powerful capability to generate human-like text.
However, prompt injection attacks, which involve overwriting a model's original
instructions with malicious prompts to manipulate the generated text, have
raised significant concerns about the security and reliability of LLMs.
Ensuring that LLMs are robust against such attacks is crucial for their
deployment in real-world applications, particularly in critical tasks.
  In this paper, we propose PROMPTFUZZ, a novel testing framework that
leverages fuzzing techniques to systematically assess the robustness of LLMs
against prompt injection attacks. Inspired by software fuzzing, PROMPTFUZZ
selects promising seed prompts and generates a diverse set of prompt injections
to evaluate the target LLM's resilience. PROMPTFUZZ operates in two stages: the
prepare phase, which involves selecting promising initial seeds and collecting
few-shot examples, and the focus phase, which uses the collected examples to
generate diverse, high-quality prompt injections. Using PROMPTFUZZ, we can
uncover more vulnerabilities in LLMs, even those with strong defense prompts.
  By deploying the generated attack prompts from PROMPTFUZZ in a real-world
competition, we achieved the 7th ranking out of over 4000 participants (top
0.14%) within 2 hours. Additionally, we construct a dataset to fine-tune LLMs
for enhanced robustness against prompt injection attacks. While the fine-tuned
model shows improved robustness, PROMPTFUZZ continues to identify
vulnerabilities, highlighting the importance of robust testing for LLMs. Our
work emphasizes the critical need for effective testing tools and provides a
practical framework for evaluating and improving the robustness of LLMs against
prompt injection attacks.


http://arxiv.org/abs/2409.15119
Log-normal Mutations and their Use in Detecting Surreptitious Fake Images. (13%)
Ismail Labiad; Thomas Bäck; Pierre Fernandez; Laurent Najman; Tom Sander; Furong Ye; Mariia Zameshina; Olivier Teytaud
  In many cases, adversarial attacks are based on specialized algorithms
specifically dedicated to attacking automatic image classifiers. These
algorithms perform well, thanks to an excellent ad hoc distribution of initial
attacks. However, these attacks are easily detected due to their specific
initial distribution. We therefore consider other black-box attacks, inspired
from generic black-box optimization tools, and in particular the log-normal
algorithm.
  We apply the log-normal method to the attack of fake detectors, and get
successful attacks: importantly, these attacks are not detected by detectors
specialized on classical adversarial attacks. Then, combining these attacks and
deep detection, we create improved fake detectors.


http://arxiv.org/abs/2410.07191
Curb Your Attention: Causal Attention Gating for Robust Trajectory Prediction in Autonomous Driving. (12%)
Ehsan Ahmadi; Ray Mercurius; Soheil Alizadeh; Kasra Rezaee; Amir Rasouli
  Trajectory prediction models in autonomous driving are vulnerable to
perturbations from non-causal agents whose actions should not affect the
ego-agent's behavior. Such perturbations can lead to incorrect predictions of
other agents' trajectories, potentially compromising the safety and efficiency
of the ego-vehicle's decision-making process. Motivated by this challenge, we
propose $\textit{Causal tRajecTory predICtion}$ $\textbf{(CRiTIC)}$, a novel
model that utilizes a $\textit{Causal Discovery Network}$ to identify
inter-agent causal relations over a window of past time steps. To incorporate
discovered causal relationships, we propose a novel $\textit{Causal Attention
Gating}$ mechanism to selectively filter information in the proposed
Transformer-based architecture. We conduct extensive experiments on two
autonomous driving benchmark datasets to evaluate the robustness of our model
against non-causal perturbations and its generalization capacity. Our results
indicate that the robustness of predictions can be improved by up to
$\textbf{54%}$ without a significant detriment to prediction accuracy. Lastly,
we demonstrate the superior domain generalizability of the proposed model,
which achieves up to $\textbf{29%}$ improvement in cross-domain performance.
These results underscore the potential of our model to enhance both robustness
and generalization capacity for trajectory prediction in diverse autonomous
driving domains. Further details can be found on our project page:
https://ehsan-ami.github.io/critic.


http://arxiv.org/abs/2409.15695
Toward Mixture-of-Experts Enabled Trustworthy Semantic Communication for 6G Networks. (5%)
Jiayi He; Xiaofeng Luo; Jiawen Kang; Hongyang Du; Zehui Xiong; Ci Chen; Dusit Niyato; Xuemin Shen
  Semantic Communication (SemCom) plays a pivotal role in 6G networks, offering
a viable solution for future efficient communication. Deep Learning (DL)-based
semantic codecs further enhance this efficiency. However, the vulnerability of
DL models to security threats, such as adversarial attacks, poses significant
challenges for practical applications of SemCom systems. These vulnerabilities
enable attackers to tamper with messages and eavesdrop on private information,
especially in wireless communication scenarios. Although existing defenses
attempt to address specific threats, they often fail to simultaneously handle
multiple heterogeneous attacks. To overcome this limitation, we introduce a
novel Mixture-of-Experts (MoE)-based SemCom system. This system comprises a
gating network and multiple experts, each specializing in different security
challenges. The gating network adaptively selects suitable experts to counter
heterogeneous attacks based on user-defined security requirements. Multiple
experts collaborate to accomplish semantic communication tasks while meeting
the security requirements of users. A case study in vehicular networks
demonstrates the efficacy of the MoE-based SemCom system. Simulation results
show that the proposed MoE-based SemCom system effectively mitigates concurrent
heterogeneous attacks, with minimal impact on downstream task accuracy.


http://arxiv.org/abs/2409.14712
Room Impulse Responses help attackers to evade Deep Fake Detection. (1%)
Hieu-Thi Luong; Duc-Tuan Truong; Kong Aik Lee; Eng Siong Chng
  The ASVspoof 2021 benchmark, a widely-used evaluation framework for
anti-spoofing, consists of two subsets: Logical Access (LA) and Deepfake (DF),
featuring samples with varied coding characteristics and compression artifacts.
Notably, the current state-of-the-art (SOTA) system boasts impressive
performance, achieving an Equal Error Rate (EER) of 0.87% on the LA subset and
2.58% on the DF. However, benchmark accuracy is no guarantee of robustness in
real-world scenarios. This paper investigates the effectiveness of utilizing
room impulse responses (RIRs) to enhance fake speech and increase their
likelihood of evading fake speech detection systems. Our findings reveal that
this simple approach significantly improves the evasion rate, doubling the SOTA
system's EER. To counter this type of attack, We augmented training data with a
large-scale synthetic/simulated RIR dataset. The results demonstrate
significant improvement on both reverberated fake speech and original samples,
reducing DF task EER to 2.13%.


http://arxiv.org/abs/2409.15041
AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark. (1%)
Michal Nazarczuk; Thomas Tanay; Sibi Catley-Chandar; Richard Shaw; Radu Timofte; Eduardo Pérez-Pellitero
  Recent developments in differentiable and neural rendering have made
impressive breakthroughs in a variety of 2D and 3D tasks, e.g. novel view
synthesis, 3D reconstruction. Typically, differentiable rendering relies on a
dense viewpoint coverage of the scene, such that the geometry can be
disambiguated from appearance observations alone. Several challenges arise when
only a few input views are available, often referred to as sparse or few-shot
neural rendering. As this is an underconstrained problem, most existing
approaches introduce the use of regularisation, together with a diversity of
learnt and hand-crafted priors. A recurring problem in sparse rendering
literature is the lack of an homogeneous, up-to-date, dataset and evaluation
protocol. While high-resolution datasets are standard in dense reconstruction
literature, sparse rendering methods often evaluate with low-resolution images.
Additionally, data splits are inconsistent across different manuscripts, and
testing ground-truth images are often publicly available, which may lead to
over-fitting. In this work, we propose the Sparse Rendering (SpaRe) dataset and
benchmark. We introduce a new dataset that follows the setup of the DTU MVS
dataset. The dataset is composed of 97 new scenes based on synthetic,
high-quality assets. Each scene has up to 64 camera views and 7 lighting
configurations, rendered at 1600x1200 resolution. We release a training split
of 82 scenes to foster generalizable approaches, and provide an online
evaluation platform for the validation and test sets, whose ground-truth images
remain hidden. We propose two different sparse configurations (3 and 9 input
images respectively). This provides a powerful and convenient tool for
reproducible evaluation, and enable researchers easy access to a public
leaderboard with the state-of-the-art performance scores. Available at:
https://sparebenchmark.github.io/


http://arxiv.org/abs/2409.15126
UTrace: Poisoning Forensics for Private Collaborative Learning. (1%)
Evan Rose; Hidde Lycklama; Harsh Chaudhari; Anwar Hithnawi; Alina Oprea
  Privacy-preserving machine learning (PPML) enables multiple data owners to
contribute their data privately to a set of servers that run a secure
multi-party computation (MPC) protocol to train a joint ML model. In these
protocols, the input data remains private throughout the training process, and
only the resulting model is made available. While this approach benefits
privacy, it also exacerbates the risks of data poisoning, where compromised
data owners induce undesirable model behavior by contributing malicious
datasets. Existing MPC mechanisms can mitigate certain poisoning attacks, but
these measures are not exhaustive. To complement existing poisoning defenses,
we introduce UTrace: a framework for User-level Traceback of poisoning attacks
in PPML. Utrace computes user responsibility scores using gradient similarity
metrics aggregated across the most relevant samples in an owner's dataset.
UTrace is effective at low poisoning rates and is resilient to poisoning
attacks distributed across multiple data owners, unlike existing
unlearning-based methods. We introduce methods for checkpointing gradients with
low storage overhead, enabling traceback in the absence of data owners at
deployment time. We also design several optimizations that reduce traceback
time and communication in MPC. We provide a comprehensive evaluation of UTrace
across four datasets from three data modalities (vision, text, and malware) and
show its effectiveness against 10 poisoning attacks.


http://arxiv.org/abs/2409.14805
SDBA: A Stealthy and Long-Lasting Durable Backdoor Attack in Federated Learning. (1%)
Minyeong Choe; Cheolhee Park; Changho Seo; Hyunil Kim
  Federated Learning is a promising approach for training machine learning
models while preserving data privacy, but its distributed nature makes it
vulnerable to backdoor attacks, particularly in NLP tasks while related
research remains limited. This paper introduces SDBA, a novel backdoor attack
mechanism designed for NLP tasks in FL environments. Our systematic analysis
across LSTM and GPT-2 models identifies the most vulnerable layers for backdoor
injection and achieves both stealth and long-lasting durability through
layer-wise gradient masking and top-k% gradient masking within these layers.
Experiments on next token prediction and sentiment analysis tasks show that
SDBA outperforms existing backdoors in durability and effectively bypasses
representative defense mechanisms, with notable performance in LLM such as
GPT-2. These results underscore the need for robust defense strategies in
NLP-based FL systems.


http://arxiv.org/abs/2409.14488
Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attacks. (10%)
Ruoyu Song; Muslum Ozgur Ozmen; Hyungsub Kim; Antonio Bianchi; Z. Berkay Celik
  There is a growing interest in integrating Large Language Models (LLMs) with
autonomous driving (AD) systems. However, AD systems are vulnerable to attacks
against their object detection and tracking (ODT) functions. Unfortunately, our
evaluation of four recent LLM agents against ODT attacks shows that the attacks
are 63.26% successful in causing them to crash or violate traffic rules due to
(1) misleading memory modules that provide past experiences for decision
making, (2) limitations of prompts in identifying inconsistencies, and (3)
reliance on ground truth perception data.
  In this paper, we introduce Hudson, a driving reasoning agent that extends
prior LLM-based driving systems to enable safer decision making during
perception attacks while maintaining effectiveness under benign conditions.
Hudson achieves this by first instrumenting the AD software to collect
real-time perception results and contextual information from the driving scene.
This data is then formalized into a domain-specific language (DSL). To guide
the LLM in detecting and making safe control decisions during ODT attacks,
Hudson translates the DSL into natural language, along with a list of custom
attack detection instructions. Following query execution, Hudson analyzes the
LLM's control decision to understand its causal reasoning process.
  We evaluate the effectiveness of Hudson using a proprietary LLM (GPT-4) and
two open-source LLMs (Llama and Gemma) in various adversarial driving
scenarios. GPT-4, Llama, and Gemma achieve, on average, an attack detection
accuracy of 83. 3%, 63. 6%, and 73. 6%. Consequently, they make safe control
decisions in 86.4%, 73.9%, and 80% of the attacks. Our results, following the
growing interest in integrating LLMs into AD systems, highlight the strengths
of LLMs and their potential to detect and mitigate ODT attacks.


http://arxiv.org/abs/2409.14572
Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions. (1%)
Hongchen Wang; Kangming Li; Scott Ramsay; Yao Fehlis; Edward Kim; Jason Hattrick-Simpers
  Large Language Models (LLMs) have the potential to revolutionize scientific
research, yet their robustness and reliability in domain-specific applications
remain insufficiently explored. This study conducts a comprehensive evaluation
and robustness analysis of LLMs within the field of materials science, focusing
on domain-specific question answering and materials property prediction. Three
distinct datasets are used in this study: 1) a set of multiple-choice questions
from undergraduate-level materials science courses, 2) a dataset including
various steel compositions and yield strengths, and 3) a band gap dataset,
containing textual descriptions of material crystal structures and band gap
values. The performance of LLMs is assessed using various prompting strategies,
including zero-shot chain-of-thought, expert prompting, and few-shot in-context
learning. The robustness of these models is tested against various forms of
'noise', ranging from realistic disturbances to intentionally adversarial
manipulations, to evaluate their resilience and reliability under real-world
conditions. Additionally, the study uncovers unique phenomena of LLMs during
predictive tasks, such as mode collapse behavior when the proximity of prompt
examples is altered and performance enhancement from train/test mismatch. The
findings aim to provide informed skepticism for the broad use of LLMs in
materials science and to inspire advancements that enhance their robustness and
reliability for practical applications.


http://arxiv.org/abs/2409.14240
Cloud Adversarial Example Generation for Remote Sensing Image Classification. (99%)
Fei Ma; Yuqiang Feng; Fan Zhang; Yongsheng Zhou
  Most existing adversarial attack methods for remote sensing images merely add
adversarial perturbations or patches, resulting in unnatural modifications.
Clouds are common atmospheric effects in remote sensing images. Generating
clouds on these images can produce adversarial examples better aligning with
human perception. In this paper, we propose a Perlin noise based cloud
generation attack method. Common Perlin noise based cloud generation is a
random, non-optimizable process, which cannot be directly used to attack the
target models. We design a Perlin Gradient Generator Network (PGGN), which
takes a gradient parameter vector as input and outputs the grids of Perlin
noise gradient vectors at different scales. After a series of computations
based on the gradient vectors, cloud masks at corresponding scales can be
produced. These cloud masks are then weighted and summed depending on a mixing
coefficient vector and a scaling factor to produce the final cloud masks. The
gradient vector, coefficient vector and scaling factor are collectively
represented as a cloud parameter vector, transforming the cloud generation into
a black-box optimization problem. The Differential Evolution (DE) algorithm is
employed to solve for the optimal solution of the cloud parameter vector,
achieving a query-based black-box attack. Detailed experiments confirm that
this method has strong attack capabilities and achieves high query efficiency.
Additionally, we analyze the transferability of the generated adversarial
examples and their robustness in adversarial defense scenarios.


http://arxiv.org/abs/2409.15381
Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation. (98%)
G M Shahariar; Jia Chen; Jiachen Li; Yue Dong
  Recent studies show that text-to-image (T2I) models are vulnerable to
adversarial attacks, especially with noun perturbations in text prompts. In
this study, we investigate the impact of adversarial attacks on different POS
tags within text prompts on the images generated by T2I models. We create a
high-quality dataset for realistic POS tag token swapping and perform
gradient-based attacks to find adversarial suffixes that mislead T2I models
into generating images with altered tokens. Our empirical results show that the
attack success rate (ASR) varies significantly among different POS tag
categories, with nouns, proper nouns, and adjectives being the easiest to
attack. We explore the mechanism behind the steering effect of adversarial
suffixes, finding that the number of critical tokens and content fusion vary
among POS tags, while features like suffix transferability are consistent
across categories. We have made our implementation publicly available at -
https://github.com/shahariar-shibli/Adversarial-Attack-on-POS-Tags.


http://arxiv.org/abs/2409.14161
When Witnesses Defend: A Witness Graph Topological Layer for Adversarial Graph Learning. (69%)
Naheed Anjum Arafat; Debabrota Basu; Yulia Gel; Yuzhou Chen
  Capitalizing on the intuitive premise that shape characteristics are more
robust to perturbations, we bridge adversarial graph learning with the emerging
tools from computational topology, namely, persistent homology representations
of graphs. We introduce the concept of witness complex to adversarial analysis
on graphs, which allows us to focus only on the salient shape characteristics
of graphs, yielded by the subset of the most essential nodes (i.e., landmarks),
with minimal loss of topological information on the whole graph. The remaining
nodes are then used as witnesses, governing which higher-order graph
substructures are incorporated into the learning process. Armed with the
witness mechanism, we design Witness Graph Topological Layer (WGTL), which
systematically integrates both local and global topological graph feature
representations, the impact of which is, in turn, automatically controlled by
the robust regularized topological loss. Given the attacker's budget, we derive
the important stability guarantees of both local and global topology encodings
and the associated robust topological loss. We illustrate the versatility and
efficiency of WGTL by its integration with five GNNs and three existing
non-topological defense mechanisms. Our extensive experiments across six
datasets demonstrate that WGTL boosts the robustness of GNNs across a range of
perturbations and against a range of adversarial attacks, leading to relative
gains of up to 18%.


http://arxiv.org/abs/2409.14177
PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approach. (62%)
Zhihao Lin; Wei Ma; Mingyi Zhou; Yanjie Zhao; Haoyu Wang; Yang Liu; Jun Wang; Li Li
  In recent years, Large Language Models (LLMs) have gained widespread use,
raising concerns about their security. Traditional jailbreak attacks, which
often rely on the model internal information or have limitations when exploring
the unsafe behavior of the victim model, limiting their reducing their general
applicability. In this paper, we introduce PathSeeker, a novel black-box
jailbreak method, which is inspired by the game of rats escaping a maze. We
think that each LLM has its unique "security maze", and attackers attempt to
find the exit learning from the received feedback and their accumulated
experience to compromise the target LLM's security defences. Our approach
leverages multi-agent reinforcement learning, where smaller models collaborate
to guide the main LLM in performing mutation operations to achieve the attack
objectives. By progressively modifying inputs based on the model's feedback,
our system induces richer, harmful responses. During our manual attempts to
perform jailbreak attacks, we found that the vocabulary of the response of the
target model gradually became richer and eventually produced harmful responses.
Based on the observation, we also introduce a reward mechanism that exploits
the expansion of vocabulary richness in LLM responses to weaken security
constraints. Our method outperforms five state-of-the-art attack techniques
when tested across 13 commercial and open-source LLMs, achieving high attack
success rates, especially in strongly aligned commercial models like
GPT-4o-mini, Claude-3.5, and GLM-4-air with strong safety alignment. This study
aims to improve the understanding of LLM security vulnerabilities and we hope
that this sturdy can contribute to the development of more robust defenses.


http://arxiv.org/abs/2409.14285
ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination. (10%)
Navid Ayoobi; Lily Knab; Wen Cheng; David Pantoja; Hamidreza Alikhani; Sylvain Flamant; Jin Kim; Arjun Mukherjee
  While large language models (LLMs) exhibit significant utility across various
domains, they simultaneously are susceptible to exploitation for unethical
purposes, including academic misconduct and dissemination of misinformation.
Consequently, AI-generated text detection systems have emerged as a
countermeasure. However, these detection mechanisms demonstrate vulnerability
to evasion techniques and lack robustness against textual manipulations. This
paper introduces back-translation as a novel technique for evading detection,
underscoring the need to enhance the robustness of current detection systems.
The proposed method involves translating AI-generated text through multiple
languages before back-translating to English. We present a model that combines
these back-translated texts to produce a manipulated version of the original
AI-generated text. Our findings demonstrate that the manipulated text retains
the original semantics while significantly reducing the true positive rate
(TPR) of existing detection methods. We evaluate this technique on nine AI
detectors, including six open-source and three proprietary systems, revealing
their susceptibility to back-translation manipulation. In response to the
identified shortcomings of existing AI text detectors, we present a
countermeasure to improve the robustness against this form of manipulation. Our
results indicate that the TPR of the proposed method declines by only 1.85%
after back-translation manipulation. Furthermore, we build a large dataset of
720k texts using eight different LLMs. Our dataset contains both human-authored
and LLM-generated texts in various domains and writing styles to assess the
performance of our method and existing detectors. This dataset is publicly
shared for the benefit of the research community.


http://arxiv.org/abs/2409.14260
Perfect Gradient Inversion in Federated Learning: A New Paradigm from the Hidden Subset Sum Problem. (8%)
Qiongxiu Li; Lixia Luo; Agnese Gini; Changlong Ji; Zhanhao Hu; Xiao Li; Chengfang Fang; Jie Shi; Xiaolin Hu
  Federated Learning (FL) has emerged as a popular paradigm for collaborative
learning among multiple parties. It is considered privacy-friendly because
local data remains on personal devices, and only intermediate parameters --
such as gradients or model updates -- are shared. Although gradient inversion
is widely viewed as a common attack method in FL, analytical research on
reconstructing input training samples from shared gradients remains limited and
is typically confined to constrained settings like small batch sizes. In this
paper, we aim to overcome these limitations by addressing the problem from a
cryptographic perspective. We mathematically formulate the input reconstruction
problem using the gradient information shared in FL as the Hidden Subset Sum
Problem (HSSP), an extension of the well-known NP-complete Subset Sum Problem
(SSP). Leveraging this formulation allows us to achieve perfect input
reconstruction, thereby mitigating issues such as dependence on label diversity
and underperformance with large batch sizes that hinder existing empirical
gradient inversion attacks. Moreover, our analysis provides insights into why
empirical input reconstruction attacks degrade with larger batch sizes. By
modeling the problem as HSSP, we demonstrate that the batch size \( B \)
significantly affects attack complexity, with time complexity reaching \(
\mathcal{O}(B^9) \). We further show that applying secure data aggregation
techniques -- such as homomorphic encryption and secure multiparty computation
-- provides a strong defense by increasing the time complexity to \(
\mathcal{O}(N^9 B^9) \), where \( N \) is the number of local clients in FL. To
the best of our knowledge, this is the first work to rigorously analyze privacy
issues in FL by modeling them as HSSP, providing a concrete analytical
foundation for further exploration and development of defense strategies.


http://arxiv.org/abs/2409.14200
Data-centric NLP Backdoor Defense from the Lens of Memorization. (4%)
Zhenting Wang; Zhizhi Wang; Mingyu Jin; Mengnan Du; Juan Zhai; Shiqing Ma
  Backdoor attack is a severe threat to the trustworthiness of DNN-based
language models. In this paper, we first extend the definition of memorization
of language models from sample-wise to more fine-grained sentence element-wise
(e.g., word, phrase, structure, and style), and then point out that language
model backdoors are a type of element-wise memorization. Through further
analysis, we find that the strength of such memorization is positively
correlated to the frequency of duplicated elements in the training dataset. In
conclusion, duplicated sentence elements are necessary for successful backdoor
attacks. Based on this, we propose a data-centric defense. We first detect
trigger candidates in training data by finding memorizable elements, i.e.,
duplicated elements, and then confirm real triggers by testing if the
candidates can activate backdoor behaviors (i.e., malicious elements). Results
show that our method outperforms state-of-the-art defenses in defending against
different types of NLP backdoors.


http://arxiv.org/abs/2409.13559
Efficient Visualization of Neural Networks with Generative Models and Adversarial Perturbations. (99%)
Athanasios Karagounis
  This paper presents a novel approach for deep visualization via a generative
network, offering an improvement over existing methods. Our model simplifies
the architecture by reducing the number of networks used, requiring only a
generator and a discriminator, as opposed to the multiple networks
traditionally involved. Additionally, our model requires less prior training
knowledge and uses a non-adversarial training process, where the discriminator
acts as a guide rather than a competitor to the generator. The core
contribution of this work is its ability to generate detailed visualization
images that align with specific class labels. Our model incorporates a unique
skip-connection-inspired block design, which enhances label-directed image
generation by propagating class information across multiple layers.
Furthermore, we explore how these generated visualizations can be utilized as
adversarial examples, effectively fooling classification networks with minimal
perceptible modifications to the original images. Experimental results
demonstrate that our method outperforms traditional adversarial example
generation techniques in both targeted and non-targeted attacks, achieving up
to a 94.5% fooling rate with minimal perturbation. This work bridges the gap
between visualization methods and adversarial examples, proposing that fooling
rate could serve as a quantitative measure for evaluating visualization
quality. The insights from this study provide a new perspective on the
interpretability of neural networks and their vulnerabilities to adversarial
attacks.


http://arxiv.org/abs/2409.13828
ViTGuard: Attention-aware Detection against Adversarial Examples for Vision Transformer. (99%)
Shihua Sun; Kenechukwu Nwodo; Shridatt Sugrim; Angelos Stavrou; Haining Wang
  The use of transformers for vision tasks has challenged the traditional
dominant role of convolutional neural networks (CNN) in computer vision (CV).
For image classification tasks, Vision Transformer (ViT) effectively
establishes spatial relationships between patches within images, directing
attention to important areas for accurate predictions. However, similar to
CNNs, ViTs are vulnerable to adversarial attacks, which mislead the image
classifier into making incorrect decisions on images with carefully designed
perturbations. Moreover, adversarial patch attacks, which introduce arbitrary
perturbations within a small area, pose a more serious threat to ViTs. Even
worse, traditional detection methods, originally designed for CNN models, are
impractical or suffer significant performance degradation when applied to ViTs,
and they generally overlook patch attacks.
  In this paper, we propose ViTGuard as a general detection method for
defending ViT models against adversarial attacks, including typical attacks
where perturbations spread over the entire input and patch attacks. ViTGuard
uses a Masked Autoencoder (MAE) model to recover randomly masked patches from
the unmasked regions, providing a flexible image reconstruction strategy. Then,
threshold-based detectors leverage distinctive ViT features, including
attention maps and classification (CLS) token representations, to distinguish
between normal and adversarial samples. The MAE model does not involve any
adversarial samples during training, ensuring the effectiveness of our
detectors against unseen attacks. ViTGuard is compared with seven existing
detection methods under nine attacks across three datasets. The evaluation
results show the superiority of ViTGuard over existing detectors. Finally,
considering the potential detection evasion, we further demonstrate ViTGuard's
robustness against adaptive attacks for evasion.


http://arxiv.org/abs/2409.13546
Certified Adversarial Robustness via Partition-based Randomized Smoothing. (81%)
Hossein Goli; Farzan Farnia
  A reliable application of deep neural network classifiers requires robustness
certificates against adversarial perturbations. Gaussian smoothing is a widely
analyzed approach to certifying robustness against norm-bounded perturbations,
where the certified prediction radius depends on the variance of the Gaussian
noise and the confidence level of the neural net's prediction under the
additive Gaussian noise. However, in application to high-dimensional image
datasets, the certified radius of the plain Gaussian smoothing could be
relatively small, since Gaussian noise with high variances can significantly
harm the visibility of an image. In this work, we propose the Pixel
Partitioning-based Randomized Smoothing (PPRS) methodology to boost the neural
net's confidence score and thus the robustness radius of the certified
prediction. We demonstrate that the proposed PPRS algorithm improves the
visibility of the images under additive Gaussian noise. We discuss the
numerical results of applying PPRS to standard computer vision datasets and
neural network architectures. Our empirical findings indicate a considerable
improvement in the certified accuracy and stability of the prediction model to
the additive Gaussian noise in randomized smoothing.


http://arxiv.org/abs/2409.13349
ID-Guard: A Universal Framework for Combating Facial Manipulation via Breaking Identification. (80%)
Zuomin Qu; Wei Lu; Xiangyang Luo; Qian Wang; Xiaochun Cao
  The misuse of deep learning-based facial manipulation poses a significant
threat to civil rights. To prevent this fraud at its source, proactive defense
has been proposed to disrupt the manipulation process by adding invisible
adversarial perturbations into images, making the forged output unconvincing to
observers. However, the non-specific disruption against the output may lead to
the retention of identifiable facial features, potentially resulting in the
stigmatization of the individual. This paper proposes a universal framework for
combating facial manipulation, termed ID-Guard. Specifically, this framework
operates with a single forward pass of an encoder-decoder network to produce a
cross-model transferable adversarial perturbation. A novel Identity Destruction
Module (IDM) is introduced to degrade identifiable features in forged faces. We
optimize the perturbation generation by framing the disruption of different
facial manipulations as a multi-task learning problem, and a dynamic weight
strategy is devised to enhance cross-model performance. Experimental results
demonstrate that the proposed ID-Guard exhibits strong efficacy in defending
against various facial manipulation models, effectively degrading identifiable
regions in manipulated images. It also enables disrupted images to evade facial
inpainting and image recognition systems. Additionally, ID-Guard can seamlessly
function as a plug-and-play component, integrating with other tasks such as
adversarial training.


http://arxiv.org/abs/2409.13232
Relationship between Uncertainty in DNNs and Adversarial Attacks. (70%)
Mabel Ogonna; Abigail Adeniran; Adewale Adeyemo
  Deep Neural Networks (DNNs) have achieved state of the art results and even
outperformed human accuracy in many challenging tasks, leading to DNNs adoption
in a variety of fields including natural language processing, pattern
recognition, prediction, and control optimization. However, DNNs are
accompanied by uncertainty about their results, causing them to predict an
outcome that is either incorrect or outside of a certain level of confidence.
These uncertainties stem from model or data constraints, which could be
exacerbated by adversarial attacks. Adversarial attacks aim to provide
perturbed input to DNNs, causing the DNN to make incorrect predictions or
increase model uncertainty. In this review, we explore the relationship between
DNN uncertainty and adversarial attacks, emphasizing how adversarial attacks
might raise DNN uncertainty.


http://arxiv.org/abs/2409.13945
PureDiffusion: Using Backdoor to Counter Backdoor in Generative Diffusion Models. (61%)
Vu Tuan Truong; Long Bao Le
  Diffusion models (DMs) are advanced deep learning models that achieved
state-of-the-art capability on a wide range of generative tasks. However,
recent studies have shown their vulnerability regarding backdoor attacks, in
which backdoored DMs consistently generate a designated result (e.g., a harmful
image) called backdoor target when the models' input contains a backdoor
trigger. Although various backdoor techniques have been investigated to attack
DMs, defense methods against these threats are still limited and underexplored,
especially in inverting the backdoor trigger. In this paper, we introduce
PureDiffusion, a novel backdoor defense framework that can efficiently detect
backdoor attacks by inverting backdoor triggers embedded in DMs. Our extensive
experiments on various trigger-target pairs show that PureDiffusion outperforms
existing defense methods with a large gap in terms of fidelity (i.e., how much
the inverted trigger resembles the original trigger) and backdoor success rate
(i.e., the rate that the inverted trigger leads to the corresponding backdoor
target). Notably, in certain cases, backdoor triggers inverted by PureDiffusion
even achieve higher attack success rate than the original triggers.


http://arxiv.org/abs/2409.13793
On the Feasibility of Fully AI-automated Vishing Attacks. (1%)
João Figueiredo; Afonso Carvalho; Daniel Castro; Daniel Gonçalves; Nuno Santos
  A vishing attack is a form of social engineering where attackers use phone
calls to deceive individuals into disclosing sensitive information, such as
personal data, financial information, or security credentials. Attackers
exploit the perceived urgency and authenticity of voice communication to
manipulate victims, often posing as legitimate entities like banks or tech
support. Vishing is a particularly serious threat as it bypasses security
controls designed to protect information. In this work, we study the potential
for vishing attacks to escalate with the advent of AI. In theory, AI-powered
software bots may have the ability to automate these attacks by initiating
conversations with potential victims via phone calls and deceiving them into
disclosing sensitive information. To validate this thesis, we introduce ViKing,
an AI-powered vishing system developed using publicly available AI technology.
It relies on a Large Language Model (LLM) as its core cognitive processor to
steer conversations with victims, complemented by a pipeline of speech-to-text
and text-to-speech modules that facilitate audio-text conversion in phone
calls. Through a controlled social experiment involving 240 participants, we
discovered that ViKing has successfully persuaded many participants to reveal
sensitive information, even those who had been explicitly warned about the risk
of vishing campaigns. Interactions with ViKing's bots were generally considered
realistic. From these findings, we conclude that tools like ViKing may already
be accessible to potential malicious actors, while also serving as an
invaluable resource for cyber awareness programs.


http://arxiv.org/abs/2409.12642
Deep generative models as an adversarial attack strategy for tabular machine learning. (99%)
Salijona Dyrmishi; Mihaela Cătălina Stoian; Eleonora Giunchiglia; Maxime Cordy
  Deep Generative Models (DGMs) have found application in computer vision for
generating adversarial examples to test the robustness of machine learning (ML)
systems. Extending these adversarial techniques to tabular ML presents unique
challenges due to the distinct nature of tabular data and the necessity to
preserve domain constraints in adversarial examples. In this paper, we adapt
four popular tabular DGMs into adversarial DGMs (AdvDGMs) and evaluate their
effectiveness in generating realistic adversarial examples that conform to
domain constraints.


http://arxiv.org/abs/2409.12472
TEAM: Temporal Adversarial Examples Attack Model against Network Intrusion Detection System Applied to RNN. (99%)
Ziyi Liu; Dengpan Ye; Long Tang; Yunming Zhang; Jiacheng Deng
  With the development of artificial intelligence, neural networks play a key
role in network intrusion detection systems (NIDS). Despite the tremendous
advantages, neural networks are susceptible to adversarial attacks. To improve
the reliability of NIDS, many research has been conducted and plenty of
solutions have been proposed. However, the existing solutions rarely consider
the adversarial attacks against recurrent neural networks (RNN) with time
steps, which would greatly affect the application of NIDS in real world.
Therefore, we first propose a novel RNN adversarial attack model based on
feature reconstruction called \textbf{T}emporal adversarial \textbf{E}xamples
\textbf{A}ttack \textbf{M}odel \textbf{(TEAM)}, which applied to time series
data and reveals the potential connection between adversarial and time steps in
RNN. That is, the past adversarial examples within the same time steps can
trigger further attacks on current or future original examples. Moreover, TEAM
leverages Time Dilation (TD) to effectively mitigates the effect of temporal
among adversarial examples within the same time steps. Experimental results
show that in most attack categories, TEAM improves the misjudgment rate of NIDS
on both black and white boxes, making the misjudgment rate reach more than
96.68%. Meanwhile, the maximum increase in the misjudgment rate of the NIDS for
subsequent original samples exceeds 95.57%.


http://arxiv.org/abs/2409.13163
Hidden Activations Are Not Enough: A General Approach to Neural Network Predictions. (98%)
Samuel Leblanc; Aiky Rasolomanana; Marco Armenta
  We introduce a novel mathematical framework for analyzing neural networks
using tools from quiver representation theory. This framework enables us to
quantify the similarity between a new data sample and the training data, as
perceived by the neural network. By leveraging the induced quiver
representation of a data sample, we capture more information than traditional
hidden layer outputs. This quiver representation abstracts away the complexity
of the computations of the forward pass into a single matrix, allowing us to
employ simple geometric and statistical arguments in a matrix space to study
neural network predictions. Our mathematical results are architecture-agnostic
and task-agnostic, making them broadly applicable. As proof of concept
experiments, we apply our results for the MNIST and FashionMNIST datasets on
the problem of detecting adversarial examples on different MLP architectures
and several adversarial attack methods. Our experiments can be reproduced with
our
\href{https://github.com/MarcoArmenta/Hidden-Activations-are-not-Enough}{publicly
available repository}.


http://arxiv.org/abs/2409.12914
Defending against Reverse Preference Attacks is Difficult. (83%)
Domenic Rosati; Giles Edkins; Harsh Raj; David Atanasov; Subhabrata Majumdar; Janarthanan Rajendran; Frank Rudzicz; Hassan Sajjad
  While there has been progress towards aligning Large Language Models (LLMs)
with human values and ensuring safe behaviour at inference time, safety-aligned
LLMs are known to be vulnerable to training-time attacks such as supervised
fine-tuning (SFT) on harmful datasets. In this paper, we ask if LLMs are
vulnerable to adversarial reinforcement learning. Motivated by this goal, we
propose Reverse Preference Attacks (RPA), a class of attacks to make LLMs learn
harmful behavior using adversarial reward during reinforcement learning from
human feedback (RLHF). RPAs expose a critical safety gap of safety-aligned LLMs
in RL settings: they easily explore the harmful text generation policies to
optimize adversarial reward. To protect against RPAs, we explore a host of
mitigation strategies. Leveraging Constrained Markov-Decision Processes, we
adapt a number of mechanisms to defend against harmful fine-tuning attacks into
the RL setting. Our experiments show that ``online" defenses that are based on
the idea of minimizing the negative log likelihood of refusals -- with the
defender having control of the loss function -- can effectively protect LLMs
against RPAs. However, trying to defend model weights using ``offline" defenses
that operate under the assumption that the defender has no control over the
loss function are less effective in the face of RPAs. These findings show that
attacks done using RL can be used to successfully undo safety alignment in
open-weight LLMs and use them for malicious purposes.


http://arxiv.org/abs/2409.12946
Revisiting Semi-supervised Adversarial Robustness via Noise-aware Online Robust Distillation. (45%)
Tsung-Han Wu; Hung-Ting Su; Shang-Tse Chen; Winston H. Hsu
  The robust self-training (RST) framework has emerged as a prominent approach
for semi-supervised adversarial training. To explore the possibility of
tackling more complicated tasks with even lower labeling budgets, unlike prior
approaches that rely on robust pretrained models, we present SNORD - a simple
yet effective framework that introduces contemporary semi-supervised learning
techniques into the realm of adversarial training. By enhancing pseudo labels
and managing noisy training data more effectively, SNORD showcases impressive,
state-of-the-art performance across diverse datasets and labeling budgets, all
without the need for pretrained models. Compared to full adversarial
supervision, SNORD achieves a 90% relative robust accuracy under epsilon =
8/255 AutoAttack, requiring less than 0.1%, 2%, and 10% labels for CIFAR-10,
CIFAR-100, and TinyImageNet-200, respectively. Additional experiments confirm
the efficacy of each component and demonstrate the adaptability of integrating
SNORD with existing adversarial pretraining strategies to further bolster
robustness.


http://arxiv.org/abs/2409.12997
VCAT: Vulnerability-aware and Curiosity-driven Adversarial Training for Enhancing Autonomous Vehicle Robustness. (26%)
Xuan Cai; Zhiyong Cui; Xuesong Bai; Ruimin Ke; Zhenshu Ma; Haiyang Yu; Yilong Ren
  Autonomous vehicles (AVs) face significant threats to their safe operation in
complex traffic environments. Adversarial training has emerged as an effective
method of enabling AVs to preemptively fortify their robustness against
malicious attacks. Train an attacker using an adversarial policy, allowing the
AV to learn robust driving through interaction with this attacker. However,
adversarial policies in existing methodologies often get stuck in a loop of
overexploiting established vulnerabilities, resulting in poor improvement for
AVs. To overcome the limitations, we introduce a pioneering framework termed
Vulnerability-aware and Curiosity-driven Adversarial Training (VCAT).
Specifically, during the traffic vehicle attacker training phase, a surrogate
network is employed to fit the value function of the AV victim, providing dense
information about the victim's inherent vulnerabilities. Subsequently, random
network distillation is used to characterize the novelty of the environment,
constructing an intrinsic reward to guide the attacker in exploring unexplored
territories. In the victim defense training phase, the AV is trained in
critical scenarios in which the pretrained attacker is positioned around the
victim to generate attack behaviors. Experimental results revealed that the
training methodology provided by VCAT significantly improved the robust control
capabilities of learning-based AVs, outperforming both conventional training
modalities and alternative reinforcement learning counterparts, with a marked
reduction in crash rates. The code is available at
https://github.com/caixxuan/VCAT.


http://arxiv.org/abs/2409.13004
Data Poisoning and Leakage Analysis in Federated Learning. (11%)
Wenqi Wei; Tiansheng Huang; Zachary Yahn; Anoop Singhal; Margaret Loper; Ling Liu
  Data poisoning and leakage risks impede the massive deployment of federated
learning in the real world. This chapter reveals the truths and pitfalls of
understanding two dominating threats: {\em training data privacy intrusion} and
{\em training data poisoning}. We first investigate training data privacy
threat and present our observations on when and how training data may be leaked
during the course of federated training. One promising defense strategy is to
perturb the raw gradient update by adding some controlled randomized noise
prior to sharing during each round of federated learning. We discuss the
importance of determining the proper amount of randomized noise and the proper
location to add such noise for effective mitigation of gradient leakage threats
against training data privacy. Then we will review and compare different
training data poisoning threats and analyze why and when such data poisoning
induced model Trojan attacks may lead to detrimental damage on the performance
of the global model. We will categorize and compare representative poisoning
attacks and the effectiveness of their mitigation techniques, delivering an
in-depth understanding of the negative impact of data poisoning. Finally, we
demonstrate the potential of dynamic model perturbation in simultaneously
ensuring privacy protection, poisoning resilience, and model performance. The
chapter concludes with a discussion on additional risk factors in federated
learning, including the negative impact of skewness, data and algorithmic
biases, as well as misinformation in training data. Powered by empirical
evidence, our analytical study offers some transformative insights into
effective privacy protection and security assurance strategies in
attack-resilient federated learning.


http://arxiv.org/abs/2409.13174
Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Models. (2%)
Hao Cheng; Erjia Xiao; Chengyuan Yu; Zhao Yao; Jiahang Cao; Qiang Zhang; Jiaxu Wang; Mengshu Sun; Kaidi Xu; Jindong Gu; Renjing Xu
  Recently, driven by advancements in Multimodal Large Language Models (MLLMs),
Vision Language Action Models (VLAMs) are being proposed to achieve better
performance in open-vocabulary scenarios for robotic manipulation tasks. Since
manipulation tasks involve direct interaction with the physical world, ensuring
robustness and safety during the execution of this task is always a very
critical issue. In this paper, by synthesizing current safety research on MLLMs
and the specific application scenarios of the manipulation task in the physical
world, we comprehensively evaluate VLAMs in the face of potential physical
threats. Specifically, we propose the Physical Vulnerability Evaluating
Pipeline (PVEP) that can incorporate as many visual modal physical threats as
possible for evaluating the physical robustness of VLAMs. The physical threats
in PVEP specifically include Out-of-Distribution, Typography-based Visual
Prompts, and Adversarial Patch Attacks. By comparing the performance
fluctuations of VLAMs before and after being attacked, we provide generalizable
Analyses of how VLAMs respond to different physical security threats. Our
project page is in this link:
https://chaducheng.github.io/Manipulat-Facing-Threats/.


http://arxiv.org/abs/2409.12553
Hidden in Plain Sound: Environmental Backdoor Poisoning Attacks on Whisper, and Mitigations. (2%)
Jonatan Bartolini; Todor Stoyanov; Alberto Giaretta
  Thanks to the popularisation of transformer-based models, speech recognition
(SR) is gaining traction in various application fields, such as industrial and
robotics environments populated with mission-critical devices. While
transformer-based SR can provide various benefits for simplifying human-machine
interfacing, the research on the cybersecurity aspects of these models is
lacklustre. In particular, concerning backdoor poisoning attacks. In this
paper, we propose a new poisoning approach that maps different environmental
trigger sounds to target phrases of different lengths, during the fine-tuning
phase. We test our approach on Whisper, one of the most popular
transformer-based SR model, showing that it is highly vulnerable to our attack,
under several testing conditions. To mitigate the attack proposed in this
paper, we investigate the use of Silero VAD, a state-of-the-art voice activity
detection (VAD) model, as a defence mechanism. Our experiments show that it is
possible to use VAD models to filter out malicious triggers and mitigate our
attacks, with a varying degree of success, depending on the type of trigger
sound and testing conditions.


http://arxiv.org/abs/2409.12379
Enhancing 3D Robotic Vision Robustness by Minimizing Adversarial Mutual Information through a Curriculum Training Approach. (99%)
Nastaran Darabi; Dinithi Jayasuriya; Devashri Naik; Theja Tulabandhula; Amit Ranjan Trivedi
  Adversarial attacks exploit vulnerabilities in a model's decision boundaries
through small, carefully crafted perturbations that lead to significant
mispredictions. In 3D vision, the high dimensionality and sparsity of data
greatly expand the attack surface, making 3D vision particularly vulnerable for
safety-critical robotics. To enhance 3D vision's adversarial robustness, we
propose a training objective that simultaneously minimizes prediction loss and
mutual information (MI) under adversarial perturbations to contain the upper
bound of misprediction errors. This approach simplifies handling adversarial
examples compared to conventional methods, which require explicit searching and
training on adversarial samples. However, minimizing prediction loss conflicts
with minimizing MI, leading to reduced robustness and catastrophic forgetting.
To address this, we integrate curriculum advisors in the training setup that
gradually introduce adversarial objectives to balance training and prevent
models from being overwhelmed by difficult cases early in the process. The
advisors also enhance robustness by encouraging training on diverse MI examples
through entropy regularizers. We evaluated our method on ModelNet40 and KITTI
using PointNet, DGCNN, SECOND, and PointTransformers, achieving 2-5% accuracy
gains on ModelNet40 and a 5-10% mAP improvement in object detection. Our code
is publicly available at https://github.com/nstrndrbi/Mine-N-Learn.


http://arxiv.org/abs/2409.12394
ITPatch: An Invisible and Triggered Physical Adversarial Patch against Traffic Sign Recognition. (99%)
Shuai Yuan; Hongwei Li; Xingshuo Han; Guowen Xu; Wenbo Jiang; Tao Ni; Qingchuan Zhao; Yuguang Fang
  Physical adversarial patches have emerged as a key adversarial attack to
cause misclassification of traffic sign recognition (TSR) systems in the real
world. However, existing adversarial patches have poor stealthiness and attack
all vehicles indiscriminately once deployed. In this paper, we introduce an
invisible and triggered physical adversarial patch (ITPatch) with a novel
attack vector, i.e., fluorescent ink, to advance the state-of-the-art. It
applies carefully designed fluorescent perturbations to a target sign, an
attacker can later trigger a fluorescent effect using invisible ultraviolet
light, causing the TSR system to misclassify the sign and potentially resulting
in traffic accidents. We conducted a comprehensive evaluation to investigate
the effectiveness of ITPatch, which shows a success rate of 98.31% in low-light
conditions. Furthermore, our attack successfully bypasses five popular defenses
and achieves a success rate of 96.72%.


http://arxiv.org/abs/2409.11754
NPAT Null-Space Projected Adversarial Training Towards Zero Deterioration. (96%)
Hanyi Hu; Qiao Han; Kui Chen; Yao Yang
  To mitigate the susceptibility of neural networks to adversarial attacks,
adversarial training has emerged as a prevalent and effective defense strategy.
Intrinsically, this countermeasure incurs a trade-off, as it sacrifices the
model's accuracy in processing normal samples. To reconcile the trade-off, we
pioneer the incorporation of null-space projection into adversarial training
and propose two innovative Null-space Projection based Adversarial
Training(NPAT) algorithms tackling sample generation and gradient optimization,
named Null-space Projected Data Augmentation (NPDA) and Null-space Projected
Gradient Descent (NPGD), to search for an overarching optimal solutions, which
enhance robustness with almost zero deterioration in generalization
performance. Adversarial samples and perturbations are constrained within the
null-space of the decision boundary utilizing a closed-form null-space
projector, effectively mitigating threat of attack stemming from unreliable
features. Subsequently, we conducted experiments on the CIFAR10 and SVHN
datasets and reveal that our methodology can seamlessly combine with
adversarial training methods and obtain comparable robustness while keeping
generalization close to a high-accuracy model.


http://arxiv.org/abs/2409.11690
ID-Free Not Risk-Free: LLM-Powered Agents Unveil Risks in ID-Free Recommender Systems. (76%)
Zongwei Wang; Min Gao; Junliang Yu; Xinyi Gao; Quoc Viet Hung Nguyen; Shazia Sadiq; Hongzhi Yin
  Recent advances in ID-free recommender systems have attracted significant
attention for effectively addressing the cold start problem. However, their
vulnerability to malicious attacks remains largely unexplored. In this paper,
we unveil a critical yet overlooked risk: LLM-powered agents can be
strategically deployed to attack ID-free recommenders, stealthily promoting
low-quality items in black-box settings. This attack exploits a novel
rewriting-based deception strategy, where malicious agents synthesize deceptive
textual descriptions by simulating the characteristics of popular items. To
achieve this, the attack mechanism integrates two primary components: (1) a
popularity extraction component that captures essential characteristics of
popular items and (2) a multi-agent collaboration mechanism that enables
iterative refinement of promotional textual descriptions through independent
thinking and team discussion. To counter this risk, we further introduce a
detection method to identify suspicious text generated by our discovered
attack. By unveiling this risk, our work aims to underscore the urgent need to
enhance the security of ID-free recommender systems.


http://arxiv.org/abs/2409.12072
PAD-FT: A Lightweight Defense for Backdoor Attacks via Data Purification and Fine-Tuning. (68%)
Yukai Xu; Yujie Gu; Kouichi Sakurai
  Backdoor attacks pose a significant threat to deep neural networks,
particularly as recent advancements have led to increasingly subtle
implantation, making the defense more challenging. Existing defense mechanisms
typically rely on an additional clean dataset as a standard reference and
involve retraining an auxiliary model or fine-tuning the entire victim model.
However, these approaches are often computationally expensive and not always
feasible in practical applications. In this paper, we propose a novel and
lightweight defense mechanism, termed PAD-FT, that does not require an
additional clean dataset and fine-tunes only a very small part of the model to
disinfect the victim model. To achieve this, our approach first introduces a
simple data purification process to identify and select the most-likely clean
data from the poisoned training dataset. The self-purified clean dataset is
then used for activation clipping and fine-tuning only the last classification
layer of the victim model. By integrating data purification, activation
clipping, and classifier fine-tuning, our mechanism PAD-FT demonstrates
superior effectiveness across multiple backdoor attack methods and datasets, as
confirmed through extensive experimental evaluation.


http://arxiv.org/abs/2409.13770
A constrained optimization approach to improve robustness of neural networks. (54%)
Shudian Zhao; Jan Kronqvist
  In this paper, we present a novel nonlinear programming-based approach to
fine-tune pre-trained neural networks to improve robustness against adversarial
attacks while maintaining high accuracy on clean data. Our method introduces
adversary-correction constraints to ensure correct classification of
adversarial data and minimizes changes to the model parameters. We propose an
efficient cutting-plane-based algorithm to iteratively solve the large-scale
nonconvex optimization problem by approximating the feasible region through
polyhedral cuts and balancing between robustness and accuracy. Computational
experiments on standard datasets such as MNIST and CIFAR10 demonstrate that the
proposed approach significantly improves robustness, even with a very small set
of adversarial data, while maintaining minimal impact on accuracy.


http://arxiv.org/abs/2409.12314
Understanding Implosion in Text-to-Image Generative Models. (2%)
Wenxin Ding; Cathy Y. Li; Shawn Shan; Ben Y. Zhao; Haitao Zheng
  Recent works show that text-to-image generative models are surprisingly
vulnerable to a variety of poisoning attacks. Empirical results find that these
models can be corrupted by altering associations between individual text
prompts and associated visual features. Furthermore, a number of concurrent
poisoning attacks can induce "model implosion," where the model becomes unable
to produce meaningful images for unpoisoned prompts. These intriguing findings
highlight the absence of an intuitive framework to understand poisoning attacks
on these models. In this work, we establish the first analytical framework on
robustness of image generative models to poisoning attacks, by modeling and
analyzing the behavior of the cross-attention mechanism in latent diffusion
models. We model cross-attention training as an abstract problem of "supervised
graph alignment" and formally quantify the impact of training data by the
hardness of alignment, measured by an Alignment Difficulty (AD) metric. The
higher the AD, the harder the alignment. We prove that AD increases with the
number of individual prompts (or concepts) poisoned. As AD grows, the alignment
task becomes increasingly difficult, yielding highly distorted outcomes that
frequently map meaningful text prompts to undefined or meaningless visual
representations. As a result, the generative model implodes and outputs random,
incoherent images at large. We validate our analytical framework through
extensive experiments, and we confirm and explain the unexpected (and
unexplained) effect of model implosion while producing new, unforeseen
insights. Our work provides a useful tool for studying poisoning attacks
against diffusion models and their defenses.


http://arxiv.org/abs/2409.11454
Golden Ratio Search: A Low-Power Adversarial Attack for Deep Learning based Modulation Classification. (98%)
Deepsayan Sadhukhan; Nitin Priyadarshini Shankar; Sheetal Kalyani
  We propose a minimal power white box adversarial attack for Deep Learning
based Automatic Modulation Classification (AMC). The proposed attack uses the
Golden Ratio Search (GRS) method to find powerful attacks with minimal power.
We evaluate the efficacy of the proposed method by comparing it with existing
adversarial attack approaches. Additionally, we test the robustness of the
proposed attack against various state-of-the-art architectures, including
defense mechanisms such as adve